Google explains how its search engine crawls the web
The latest video comes after Google launched the educational “How Search Works” series last week to shed light on the search and indexing processes
Google has published a fresh installment of its educational video series “How Search Works,” explaining how its search engine discovers and accesses web pages through crawling.
Key takeaways
- Googlebot starts by following links from known webpages to uncover new URLs, a process called URL discovery;
- It avoids overloading sites by crawling each one at a unique, customized speed based on server response times and content quality;
- Googlebot renders pages using a current version of the Chrome browser to execute any JavaScript and correctly display dynamic content loaded by scripts. It also only crawls publicly available pages, not those behind logins.
Get the full story at Search Engine Journal