onion URLs, Ahmia has created one of the largest indexes of the deep web. Additionally, Ahmia allows onion service operators to register their own URLs, enabling them to be found. onion URLs from the Tor network, then feeds these pages to their index provided that they don’t contain a robots.txt file saying not to index them (4). There are many “deep web search engines”, but I’ll focus on two: Ahmia, and the Uncensored Hidden Wiki.Īhmia was developed by Juha Nurmi as part of the Tor Project, and it is one of the closest things to a deep web search engine (3).
If the crawlers index the page anyways, then legal action can be taken against the creator of the crawlers, and the search engine can end up on a bot reporting site like (2). Finally, if the creator of a page doesn’t want it to be indexed by popular search engines, they can include a suitable robots.txt file, which tells the crawlers not to index the page. Also, if a page contains illegal content, Google will likely not want that content appearing in search results, so they won’t index it. Additionally, this page might require some sort of authentication such as filling out a search form and clicking submit, or having a certain certificate. For one, Google’s crawlers might never come across this page simply because no other previously crawled page links to it. Google’s search engine could be unable to find this page because of several reasons. Now, consider a single page in the deep web. (1) These crawlers start from a list of known web addresses, visit those pages, then follow the links contained on those pages, and continue following links found on the new pages, collecting information about each page as they go. Google’s search engine functions by using “crawlers”. Note: the deep web shouldn’t be confused with the “dark web”, which pertains strictly to pages containing illegal content such as child pornography, terrorist forums, and illegal auctions/transactions.
It is estimated that search engines like Google index only 4% of the entire world wide web, meaning that the deep web is nearly 25 times larger than the internet you and I have used our whole lives. If search engines like Google, Yahoo, and Bing are unable to index the deep web, then how do deep web search engines work? We’ll try to answer this question by first what I mean by “deep web”, then explaining why Google can’t crawl the deep web, and finally looking at how some popular search engines like Ahmia and the Uncensored Hidden Wiki “search” the deep web.Īccording to Google’s online dictionary, the deep web is “ the part of the World Wide Web that is not discoverable by means of standard search engines, including password-protected or dynamic pages and encrypted networks”. Google Can’t Search the Deep Web, So How Do Deep Web Search Engines Work?
0 kommentar(er)
