Close
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Logo
Logo
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Home Latest News
    • Storage

    New Tool Diverts Log Data

    By
    John S. McCright
    -
    October 28, 2002
    Share
    Facebook
    Twitter
    Linkedin

      One ant on the tablecloth would not spoil a picnic, but a thousand ants would.

      Similarly, a single log file can be easily stored and searched. But businesses in todays Internet age are practically flooded with such an abundance of log file data that their IT systems can be overwhelmed and the data can lose its usefulness.

      Thats where startup Addamark Technologies Inc. steps in. Early next year, the San Francisco-based developer will roll out its LMS (Log Management System) software, which promises to help companies make sense of the mountains of log data accumulating on their systems through patented compression and query technologies.

      Logs list requests for files in a system. Ubiquitous in enterprises, they are generated by applications, Web servers, firewalls and switches.

      IT managers can use log data to track problems or hackers, test that applications are working properly, and generally get a view into how systems are working.

      Marketing people ideally can use logs to see how customers are using a Web site and to identify potential customers. Companies in some industries, such as financial services, are required by regulators to keep log data.

      But the massive amounts of log data being generated and the speed at which it accumulates pose a challenge to understanding it.

      In addition, with so many different sources creating log data, IT departments that want to correlate it can be thwarted because the logs may be stored in formats that cant be queried in a unified way.

      AtomShockwave Corp. uses an early version of Addamarks LMS to track activity on its Shockwave. com and AtomFilms.com Web sites, which together can get up to 1 million unique visitors and produce as much as 30GB of log data a day.

      “LMS compresses our logs and generates a range of statistical reports that allow us to track user activity on the sites,” said Scott Roesch, vice president and general manager of online services at AtomShockwave, also in San Francisco. “Addamarks major innovation has to do with the way they handle compression of log files, which allows the files to be stored efficiently and processed very quickly.”

      The twist that LMS adds to traditional compression is that it time-stamps each log before compressing it to a size that is 15 to 40 times smaller than it would be in a relational database.

      When a user wants to query that log, the system doesnt uncompress the entire log file, as conventional storage technology would, but uses the time stamp to find just the data needed for that query and uncompresses that part of the log, according to Addamark CEO Mark Searle.

      “As more aspects of business are conducted electronically, log files grow immensely; simultaneously, as log files grow, the data is becoming more important,” Searle said. “People have resigned themselves to doing things that are suboptimal, like taking samples of the [log] data” instead of querying large blocks of it.

      The problem with looking at a sample of log data is that a company tracking a security breach cant prove that something didnt happen. Nor can it drill back into the data to do a forensic analysis to trace the steps of a hacker, Searle said.

      Typically, the latest log files are kept in live databases before they are compressed and archived. Once theyre archived, though, their value diminishes because they cant be easily queried.

      Nielsen/NetRatings Inc. keeps about three months worth of raw log data—or about 600GB—live in an Oracle Corp. database, but Senior Director of Operations Kelley Wood wants to keep more online. Nielsen/NetRatings has numerous panelists around the world who send it data as they surf the Web.

      This data is captured and stored on the Milpitas, Calif., companys servers in the form of logs in a proprietary format and design, where it is used to create the reports the company sells.

      “The biggest problem in this process is huge data volumes and just being able to handle it,” Wood said. “We do manage to move the log files through our system and get them stored onto tapes for archive purposes, but keeping them online and available for query has always been an issue.”

      Wood has begun testing LMS to see if it allows him to keep more log files active in his Oracle database and thus increase the depth of the reports Nielsen/ NetRatings analysts produce.

      “Initial numbers show that I can have nine to 12 months of data in LMS, and thats on a very inexpensive cluster of machines,” Wood said. “If I throw some hardware at this, I will be able to have a lot of data stored for a small investment.”

      To aid the mining of log data, LMS includes a tool that uses parallel extended SQL to query compressed data. It includes pre-formatted reports and can create new ones.

      “The query tools look very good,” Wood said. “We have a few canned reports that they have written for us to do some basics. Because the query needs for this data will be 100 percent ad hoc, well be developing the queries ourselves. The SQL interface should be fine. If we decide to do more and go further, our engineers will be able to expand the interface and/or work with Addamark to do so.”

      John S. McCright
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.

      MOST POPULAR ARTICLES

      Latest News

      Zeus Kerravala on Networking: Multicloud, 5G, and...

      James Maguire - December 16, 2022 0
      I spoke with Zeus Kerravala, industry analyst at ZK Research, about the rapid changes in enterprise networking, as tech advances and digital transformation prompt...
      Read more
      Applications

      Datadog President Amit Agarwal on Trends in...

      James Maguire - November 11, 2022 0
      I spoke with Amit Agarwal, President of Datadog, about infrastructure observability, from current trends to key challenges to the future of this rapidly growing...
      Read more
      Cloud

      IGEL CEO Jed Ayres on Edge and...

      James Maguire - June 14, 2022 0
      I spoke with Jed Ayres, CEO of IGEL, about the endpoint sector, and an open source OS for the cloud; we also spoke about...
      Read more
      IT Management

      Intuit’s Nhung Ho on AI for the...

      James Maguire - May 13, 2022 0
      I spoke with Nhung Ho, Vice President of AI at Intuit, about adoption of AI in the small and medium-sized business market, and how...
      Read more
      Applications

      Kyndryl’s Nicolas Sekkaki on Handling AI and...

      James Maguire - November 9, 2022 0
      I spoke with Nicolas Sekkaki, Group Practice Leader for Applications, Data and AI at Kyndryl, about how companies can boost both their AI and...
      Read more
      Logo

      eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site’s focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Advertisers

      Advertise with TechnologyAdvice on eWeek and our other IT-focused platforms.

      Advertise with Us

      Menu

      • About eWeek
      • Subscribe to our Newsletter
      • Latest News

      Our Brands

      • Privacy Policy
      • Terms
      • About
      • Contact
      • Advertise
      • Sitemap
      • California – Do Not Sell My Information

      Property of TechnologyAdvice.
      © 2022 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.

      ×