Close
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Logo
Logo
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Home Latest News
    • Storage

    Data Handling Eased

    By
    Dennis Callaghan
    -
    October 21, 2002
    Share
    Facebook
    Twitter
    Linkedin

      eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

      Enterprises faced with the daunting task of managing hundreds of thousands—if not millions—of records in data warehouses are finding that job scheduling applications can ease their data management headaches.

      Companies use job scheduling tools to automatically schedule migrations and conversions of data from flat files into data warehouses, specifying how often to update the data warehouse and to what folders to send data.

      SAS Institute Inc. recently developed integration between its SAS/Warehouse Administrator data warehousing application and Platform JobScheduler from Platform Computing Inc., in Markham, Ontario. The integration enables companies to apply Platforms workload management and distributed management capabilities to SAS data capturing and integration technologies to schedule, analyze and monitor data warehousing job flow with little human interaction.

      Spectra Marketing Systems Inc. is using the two technologies to better manage market research data it compiles on retail outlets for its packaged goods manufacturing clients.

      The Chicago-based market research company compiles data on 750,000 retail outlets. It must convert data from flat files into a relational format, store them in a data warehouse, then ship the appropriate reports from the data to clients on a monthly basis.

      “We need to automate as much of this process as we can to minimize the business impact,” said Dennis Klos, vice president of data development in Spectras Lancaster, Pa., office. “Weve saved quite a bit of time with the scheduling package.”

      A process that once took three weeks to complete now takes seven business days on average and runs automatically 24 hours a day without human intervention, Klos said. He said the job scheduling tools in Spectras Oracle Corp. database dont match the capabilities offered in Platform JobScheduler.

      “Its a huge efficiency gain that every one of our clients benefits from,” Klos said. “Because of the increased throughput, we can bring products to market faster.”

      Integrated tool sets are only one way enterprises are using job scheduling technologies to accelerate data warehouse administration.

      Asurion Inc., in San Mateo, Calif., provides wireless communication services such as roadside assistance and handset insurance and built its data warehouse on Sagent Technology Inc.s Data Flow Server, residing on a Microsoft Corp. SQL Server database.

      Sagent, of Mountain View, Calif., then helped the company to write Perl script to handle job scheduling, said Wayne Collins, manager of IT enrollment, whos based in Asurions Nashville, Tenn., office. “We get the flat files or text files and populate the data warehouse,” said Collins. “The Perl scripting moves the files to the proper places in the data warehouse. It moves it to where Sagent looks at it, then moves it to another folder to show that its been processed.”

      This process happens automatically on an hourly basis as Asurion processes more than 2 million records a day from its handset insurance enrollment database, he said. In addition, Asurions sales team gets updated reports on the data without human intervention.

      Because of the success of the initial project, Asurion is looking to expand its use of the Sagent-powered solution to its roadside assistance database, which would increase the workload to 5 million records a day, Collins said.

      There are other ways to integrate job scheduling tools with data warehousing applications. Before its integration with Platform JobScheduler, SAS, of Cary, N.C., like most data warehousing vendors, simply provided support for the Cron and AT commands in the Unix and Windows NT operating systems, respectively. Company officials said Platform JobScheduler provides more sophisticated job scheduling tools.

      SQL Server also includes its own job scheduling tool, SQL Server Enterprise Manager. Asurions Collins said the tools in SQL Server didnt meet his needs. But Enterprise Manager fits the bill for COBE Cardiovascular Inc., a manufacturer of cardiac surgery equipment.

      “It was a large part of why we chose SQL Server as our database because Enterprise Manager has the scheduler built in,” said Marsha Williams, vice president of IT at COBE, in Arvada, Colo.

      COBE uses Ottawa-based Cognos Inc.s DecisionStream extraction, transformation and loading tool to build its data warehouse. The scheduler in Enterprise Manager kicks off SQL statements, which call data from DecisionStream. DecisionStream repopulates the data warehouse, which contains about 338,000 base records, on a nightly basis, which takes about 2.5 hours, Williams said.

      Regardless of the tools a company requires, Williams said warehouses need automated scheduling. “If you never change your historical updates, then it really isnt an issue,” she said. “But Im not sure if theres a company around today that doesnt have to update their historical data.”

      Dennis Callaghan
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.

      MOST POPULAR ARTICLES

      Latest News

      Zeus Kerravala on Networking: Multicloud, 5G, and...

      James Maguire - December 16, 2022 0
      I spoke with Zeus Kerravala, industry analyst at ZK Research, about the rapid changes in enterprise networking, as tech advances and digital transformation prompt...
      Read more
      Applications

      Datadog President Amit Agarwal on Trends in...

      James Maguire - November 11, 2022 0
      I spoke with Amit Agarwal, President of Datadog, about infrastructure observability, from current trends to key challenges to the future of this rapidly growing...
      Read more
      Cloud

      IGEL CEO Jed Ayres on Edge and...

      James Maguire - June 14, 2022 0
      I spoke with Jed Ayres, CEO of IGEL, about the endpoint sector, and an open source OS for the cloud; we also spoke about...
      Read more
      Applications

      Kyndryl’s Nicolas Sekkaki on Handling AI and...

      James Maguire - November 9, 2022 0
      I spoke with Nicolas Sekkaki, Group Practice Leader for Applications, Data and AI at Kyndryl, about how companies can boost both their AI and...
      Read more
      IT Management

      Intuit’s Nhung Ho on AI for the...

      James Maguire - May 13, 2022 0
      I spoke with Nhung Ho, Vice President of AI at Intuit, about adoption of AI in the small and medium-sized business market, and how...
      Read more
      Logo

      eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site’s focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Advertisers

      Advertise with TechnologyAdvice on eWeek and our other IT-focused platforms.

      Advertise with Us

      Menu

      • About eWeek
      • Subscribe to our Newsletter
      • Latest News

      Our Brands

      • Privacy Policy
      • Terms
      • About
      • Contact
      • Advertise
      • Sitemap
      • California – Do Not Sell My Information

      Property of TechnologyAdvice.
      © 2022 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.

      ×