Close
  • Latest News
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Logo
Logo
  • Latest News
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Home IT Management
    • IT Management

    No Bots Allowed!

    By
    eWEEK EDITORS
    -
    April 16, 2001
    Share
    Facebook
    Twitter
    Linkedin

      One look at Napster, and its easy to see how new Internet technologies can stir up major conflict. But a looming controversy on the Net might center on a piece of Internet plumbing thats now 7 years old — ancient in Web terms.

      The Robot Exclusion Standard is arcane enough that probably only hard-core site developers and search engine specialists ever even think about it. But just because its been overlooked doesnt mean its insignificant. The standard figured into at least one important legal dispute on the Net last year, and chances are good it will soon surface again.

      As its name suggests, the Robot Exclusion Standard was created to govern robots — computer programs that surf the Web without human supervision. Among other purposes, robots (or “bots”) are used to compile the vast Web databases that make search engines possible.

      Yet not everyone appreciates robots efficiency. After all, using a robot, someone can extract content from a site without viewing a single ad. A competitor can make an instant copy of your Web sites data assets. Robots can also harm a site operationally, when they surf a site more quickly than a server can serve pages. So in 1994, when the Web was still very much an uncharted frontier, a group of developers cobbled together the Robot Exclusion Standard, which lets Web site owners tell robots where to go and where not to go on their sites.

      While the standard created a workable compromise between bot developers and Web sites, it has not eliminated the potential for conflict when their interests collide.

      Last year, eBay took auction aggregator Bidders Edge to court for sending robots to crawl the eBay site. A federal judge granted an injunction against Bidders Edge, relying partly on eBays use of the Robot Exclusion Standard.

      Bidders Edge has shut down its Web service, and the companies have settled their dispute.

      How could Bidders Edge robots crawl eBays site in the first place if eBay had taken steps to keep them out? The answer lies in how the Robot Exclusion Standard works. When it first enters a site, a visiting Web robot will ask the site for its “robots.txt” file. The file contains commands that tell robots which directories theyre not permitted to visit. (You can view CNN.coms file at www.cnn.com/robots.txt.)

      However, the standard relies entirely on the courtesy of the visiting robot. Its completely optional. Nothing prevents robots from simply ignoring the directives in a robots.txt file — and many robots do just that. In that sense, a robots.txt file is less like a locked door than a “no entry” sign hanging in an open doorway.

      Thats how the creators of the standard intended it, says Martijn Koster, who helped develop the standard and maintains it to this day. Koster, a software engineer at [email protected], says the developers main concern was preventing underpowered Web servers from being overloaded by out-of-control bots.

      “Its about server administrators providing information to a robot — thats where it ends,” he says. The standard was never intended to be used to simply bar robots from accessing specific content, or to guarantee that visiting robots would comply with the robots.txt directives.

      Hands Off

      Last years eBay decision, however, suggests that where the standard left off, the courts may be willing to step in by granting legal force to the Robot Exclusion Standard.

      “We likened it to a no trespassing sign, and the court agreed,” says Jay Monahan, eBays legal counsel for intellectual property issues. EBay cited several other factors in its case, but Monahan says he saw the Robot Exclusion Standard as an important part of the companys argument.

      Koster has mixed feelings about sites employing the Robot Exclusion Standard to discriminate between bots and human users. Server operators should be able to turn to the Robot Exclusion Standard to curtail abuse, yet Koster also thinks using a robots.txt file merely to prevent bots from getting at publicly accessible content threatens the openness of the Internet.

      “I dont think thats in the spirit of free information exchange,” Koster says. Some robots may have legitimate reasons to ignore robot exclusion directives. For example, he says, a company might use robots to hunt for copyright infringing content.

      A bot should be allowed to view any publicly accessible pages as long as its not harming a Web site, says Wolfgang Tolle, chief technology officer at Cyveillance. Cyveillance offers digital asset scouring services exactly of the sort Koster describes, although Tolle says Cyveillances robots comply with the Robot Exclusion Standard.

      Another factor that may complicate legal arguments is that the Robot Exclusion Standard isnt a “standard” in the strictest sense, since it isnt sanctioned by any authority.

      Koster says he doesnt intend to push the Robot Exclusion Standard through a standards group, such as the World Wide Web Consortium, which might make it a required part of Web software. As an informal convention, it may have less weight in a courtroom — and to those who want the Internet to remain open, Koster says, that might not be such a bad thing.

      eWEEK EDITORS
      eWeek editors publish top thought leaders and leading experts in emerging technology across a wide variety of Enterprise B2B sectors. Our focus is providing actionable information for today’s technology decision makers.

      MOST POPULAR ARTICLES

      Cybersecurity

      Visa’s Michael Jabbara on Cybersecurity and Digital...

      James Maguire - May 17, 2022 0
      I spoke with Michael Jabbara, VP and Global Head of Fraud Services at Visa, about the cybersecurity technology used to ensure the safe transfer...
      Read more
      Big Data and Analytics

      Alteryx’s Suresh Vittal on the Democratization of...

      James Maguire - May 31, 2022 0
      I spoke with Suresh Vittal, Chief Product Officer at Alteryx, about the industry mega-shift toward making data analytics tools accessible to a company’s complete...
      Read more
      Big Data and Analytics

      GoodData CEO Roman Stanek on Business Intelligence...

      James Maguire - May 4, 2022 0
      I spoke with Roman Stanek, CEO of GoodData, about business intelligence, data as a service, and the frustration that many executives have with data...
      Read more
      Applications

      Cisco’s Thimaya Subaiya on Customer Experience in...

      James Maguire - May 10, 2022 0
      I spoke with Thimaya Subaiya, SVP and GM of Global Customer Experience at Cisco, about the factors that create good customer experience – and...
      Read more
      Cloud

      Yotascale CEO Asim Razzaq on Controlling Multicloud...

      James Maguire - May 5, 2022 0
      Asim Razzaq, CEO of Yotascale, provides guidance on understanding—and containing—the complex cost structure of multicloud computing. Among the topics we covered:  As you survey the...
      Read more
      Logo

      eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site’s focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Advertisers

      Advertise with TechnologyAdvice on eWeek and our other IT-focused platforms.

      Advertise with Us

      Menu

      • About eWeek
      • Subscribe to our Newsletter
      • Latest News

      Our Brands

      • Privacy Policy
      • Terms
      • About
      • Contact
      • Advertise
      • Sitemap
      • California – Do Not Sell My Information

      Property of TechnologyAdvice.
      © 2021 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.

      ×