Scholars Central

Research articlePublished 28 Apr 2017

Auto-Explore the Web – Web Crawler

Authors: Soumick Chatterjee, Asoke Nath

Ask about this article

Abstract

Select any words in the title or abstract, or press this and choose a sentence.

World Wide Web is an ever-growing public library with hundreds of millions of books without any central management system. Finding a piece of information without a proper directory is like finding a middle in a haystack. Various search engines solve this problem by indexing an amount of the complete content that is available in the internet. For accomplishing this job, search engines use an automated program, known as a web crawler. The most vital job of the web is information retrieval, that too with proper efficiency. Web Crawler helps to accomplish that, by helping search indexing or by helping in making archives. Web Crawler automatically visits all the available links which is further indexed. But, usage of web crawler is not limited to only search engines, but they can also be used for web scrapping, spam filtering, identifying unauthorized use of copyrighted content, identifying illegal and harmful web activities etc. Web Crawler faces various challenges while crawling deep web content, multimedia content etc. Various crawling techniques and various web crawlers are available and discussed in this paper.

Ask about this article

Answers come only from this article's title, abstract and keywords, each sentence pointing to where it comes from. Your question is sent to OpenAI to write the answer. Scholars Central does not store it.

Try:

Similar by topic

More from this journal

The newest in International Journal of Innovative Research in Computer and Communication Engineering.

All its articles

Conferences in this field

Upcoming conferences in this field.

Follow

New articles from what you follow go into your email digest, once you agree to alert emails.

  • JournalInternational Journal of Innovative Research in Computer and Communication Engineering
  • SubjectComputer Science & IT
  • SubjectEngineering
  • SubjectSocial Sciences & Humanities
  • BylineArticles whose byline prints “Soumick Chatterjee”
  • BylineArticles whose byline prints “Asoke Nath”

For authors and organisers

Publish, present and keep up with your field

Find the right journal for your paper and submit your manuscript.

Photo: Pexels