The world’s three largest search engines agreed this week to support the same protocol for submitting Web pages to their crawlers.
Google created the protocol, known as sitemaps, in June 2005 as an experiment to help webmasters increase their visibility in Google’s search index. Microsoft and Yahoo agreed to support the protocol yesterday. As part of the announcement, the existing sitemaps protocol from Google gets a version upgrade to Sitemaps 0.9 and moves to sitemaps.org, where it will be maintained as an independent standard with an Attribution/Share Alike creative commons license.
Fourth place search engine Ask.com is not taking part in the standards process. Ask.com does not currently accept site maps submissions from webmasters.
By agreeing to support one set of standards, the search engines are making it easier for publishers to maintain one XML file that describes the content of their site. Given that webmasters are most likely to create site maps first for the largest search engine and can be reticent to maintain different copies for each engine, Microsoft and Yahoo were effectively forced to support the standard or risk falling further behind in terms of pages indexed. The search engines have long argued over which indexes more pages.
From the press release:
“A Sitemap is an XML file that can be made available on a website and acts as a marker for search engines to crawl certain pages. It is an easy way for webmasters to make their sites more search engine friendly. It does this by conveniently allowing webmasters to list all of their URLs along with optional metadata, such as the last time the page changed, to improve how search engines crawl and index their websites.Sitemaps enhance the current model of Web crawling by allowing webmasters to list all their Web pages to improve comprehensiveness, notify search engines of changes or new pages to help freshness, and identify unchanged pages to prevent unnecessary crawling and save bandwidth. Webmasters can now universally submit their content in a uniform manner. Any webmaster can submit their Sitemap to any search engine which has adopted the protocol.The Sitemaps protocol used by Google has been widely adopted by many Web properties, including sites from the Wikimedia Foundation and the New York Times Company. Any company that manages dynamic content and a lot of web pages can benefit from Sitemaps. For example, if a company that utilizes a content management system (CMS) to deliver custom web content – (i.e., pricing, availability and promotional offers) – to thousands of URLs places a Sitemap file on its web servers, search engine crawlers will be able discover what pages are present and which have recently changed and to crawl them accordingly. By using Sitemaps, new links can reach search engine users more rapidly by informing search engine “spiders” and helping them to crawl more pages and discover new content faster. This can also drive online traffic and make search engine marketing more effective by delivering better results to users.For companies looking to improve user experience while keeping costs low, Sitemaps also helps make more efficient use of bandwidth. Sitemaps can help search engines find a company’s newest content more efficiently and avoid the need to revisit unchanged pages. Sitemaps can list what is new on a site and quickly guide crawlers to that new content.“
