Research article · 31 Oct 2016
Research articlePublished 28 Apr 2017
Auto-Explore the Web – Web Crawler
Abstract
Select any words in the title or abstract, or press this and choose a sentence.
World Wide Web is an ever-growing public library with hundreds of millions of books without any central management system. Finding a piece of information without a proper directory is like finding a middle in a haystack. Various search engines solve this problem by indexing an amount of the complete content that is available in the internet. For accomplishing this job, search engines use an automated program, known as a web crawler. The most vital job of the web is information retrieval, that too with proper efficiency. Web Crawler helps to accomplish that, by helping search indexing or by helping in making archives. Web Crawler automatically visits all the available links which is further indexed. But, usage of web crawler is not limited to only search engines, but they can also be used for web scrapping, spam filtering, identifying unauthorized use of copyrighted content, identifying illegal and harmful web activities etc. Web Crawler faces various challenges while crawling deep web content, multimedia content etc. Various crawling techniques and various web crawlers are available and discussed in this paper.
Ask about this article
Answers come only from this article's title, abstract and keywords, each sentence pointing to where it comes from. Your question is sent to OpenAI to write the answer. Scholars Central does not store it.
Similar by topic
Research article · Dec 2013
AN APPROACH TO BUILD A WEB CRAWLER USING CLUSTERING BASED K-MEANS ALGORITHM
Research article · 25 Jul 2015
Data Access Service through Web Service by Auto-Updation Mechanism
Research article · 11 Jul 2015
Data Access Service through Web Service by Auto-Updation Mechanism
Research article · 2014
A literature survey on web crawlers
More from this journal
The newest in International Journal of Innovative Research in Computer and Communication Engineering.
Commentary · 2023
Computer Vision and Pattern Recognition
Commentary · 2023
Computer Network and Network Optimization
Perspective · 2023
Cloud Computing and Big Data Analysis
Conferences in this field
Upcoming conferences in this field.
Follow
New articles from what you follow go into your email digest, once you agree to alert emails.
- JournalInternational Journal of Innovative Research in Computer and Communication Engineering
- SubjectComputer Science & IT
- SubjectEngineering
- SubjectSocial Sciences & Humanities
- BylineArticles whose byline prints “Soumick Chatterjee”
- BylineArticles whose byline prints “Asoke Nath”
