What is a search engine?
Search engine is a general term for services that search for content on the Internet, such as Google and Yahoo. In the early days of the Internet, the “directory type” system, in which information from sites was collected manually and classified into categories, was the mainstream, but this has now evolved into a type known as the “robot type.”
The robotic search engine consists of three programs.
-
A program called a robot or crawler that collects website information (
HTML
, images, etc.) and creates a database. - A program that analyzes and evaluates (ranks) the information collected by crawlers.
- A program that displays search results (search ranking) based on the evaluation of keywords entered on search sites.
A search engine is a system in which the above programs work together to present search results to users.
What is a crawler?
A crawler is a robot (
bot
) that crawls around a site and collects the elements that search engines use to determine search rankings. It gets its name from the word “crawl”, which means to crawl.
There are different types of crawlers for each search engine. For example, Google uses Googlebot, Bing uses Bingbot, and China’s Baidu uses a crawler called Baiduspider. There are also crawlers for
SEO
tools and crawlers for AI assistants (AppleBot).
Among them, Google has the highest market share in the world as a search engine, with a market share of over 80% in Japan. Yahoo Japan also uses Google’s search engine, and most of the various measures against crawlers are aimed at Googlebot.
Crawlers automatically move (crawl) by following links from websites that have already been compiled into a database. At the destination, the page will be analyzed (parsed), the following files will be collected, converted into data that can be easily handled by the search
algorithm
, and the results will be registered in the database.
Sites that are never visited by crawlers and the above files are not collected will naturally not be displayed in search engines. In such cases, no matter how much SEO you do, it will be in vain. The first step in SEO is to create a site that is easily searchable by crawlers.
Reference article
