Google offered a tutorial of sorts for webmasters trying to figure out how to optimize their sites for web crawling and indexing by Google’s search engine bot.
Web crawling, as Gary Illyes, a member of Google’s crawling and indexing teams explained in a blog this week, is the entry point for sites into Google’s search results.
“Efficient crawling of a website helps with its indexing in Google Search,” he noted.
Googlebot, the technology that Google uses to crawl websites for new and updated content to add to its search index, relies on a sophisticated set of algorithms to determine which sites to crawl, how frequently and how many pages to look at during each visit.
The number of URLs on a website that Googlebot can and wants to crawl, or the crawl budget for a website, depends on a couple of factors. One of them is what Google calls the crawl limit and the other is crawl demand.
Googlebot sets so-called crawl rate limits for each site to ensure that its crawling activity does not degrade site performance. The crawl rate limits depend on factors like how quickly a website responds to Googlebot requests.
For example, if a site slows down or responds with server errors, Googlebot will crawl the site less, Illyes said. Webmasters also have the ability to limit the extent to which Googlebot crawls their site.
Crawl demand refers to overall Googlebot activity on a site and is determined by the relative popularity of the site and the freshness of its content. Major site-wide events like site moves, can also trigger an increase in crawl demand so content can be reindexed under the new URLs, Illyes explained.
For most website owners and webmasters, crawl budgets are not something they need to worry about, Illyes noted. For example, if Googlebot tends to crawl new pages on their website the same day they are published, crawl budget is not something they need to be focusing on, he said. Similarly, owners of sites with fewer than a few thousand URLs have little to worry about Googlebot not crawling their sites efficiently.
“Prioritizing what to crawl, when, and how much resource the server hosting the site can allocate to crawling is more important for bigger sites, or those that auto-generate pages based on URL parameters, for example,” Illyes said.
Google’s analysis has shown that such sites can benefit from ensuring they don’t have too many low-value add URLs on their sites, he noted.
For example, sites that offer faceted navigation—like allowing users to filter by price range or color—can often create many combinations of URLs with duplicative content.
When search engines crawl identical content through varied URLs, there can be several negative consequences, according to Google. Having multiple URLs pointing to the same content can dilute link popularity and result in the wrong links being presented in search results.
Similarly, websites that do a poor job of reporting non-existent pages and other page errors can degrade Googlebot’s crawl coverage and can sometimes result in the best content being overlooked.
Others factors that can impact Googlebot’s crawl coverage are hacked pages, low quality and spam content and so-called infinite spaces of very large numbers of links that point to little or no new content for indexing purpose, Illyes said.
“Wasting server resources on pages like these will drain crawl activity from pages that do actually have value, which may cause a significant delay in discovering great content on a site,” he noted.