eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Amazon Web Services recently awarded prizes for innovative use of its Spot Instances, which are excess instances of the company’s industry-leading Amazon Elastic Compute Cloud computing capacity.
PiCloud and Princeton Consultants won the grand prize and runner-up awards for their use of AWS Spot Instances in the first annual Spotathon, said Stephen Elliott, senior product manager on the Amazon EC2 team, in a recent blog post. Amazon Web Services (AWS) recognized the winners at its re:Invent conference.
Spotathon is a contest to find the most innovative applications that leverage Amazon Web Services Spot Instances and help small companies achieve massive scale and save time and money. Spot Instances is a real-time market in which users bid for spare Amazon Elastic Cloud Compute (EC2) computing capacity.
PiCloud’s platform as a service for high-performance computing (HPC), batch processing and scientific computing took the grand prize of $2,500 in AWS credit in the Spotathon. PiCloud provides high-level APIs that scientists and engineers can use to submit units of computational work—like finding nucleotide sequences in a genome, conducting oil and gas geophysics simulations, or doing financial risk analytics—rather than provisioning, administering and tearing down instances themselves, Elliott said.
“By running 85% on Spot Instances, PiCloud provisions 50% more servers at the same cost, improves its customers’ experience by delivering results 33% faster, and saves 65% over the On-Demand price,” Elliott said in his post. “PiCloud has served thousands of researchers who have collectively processed over 100 million jobs, and is exemplary in how it uses AWS and Spot Instances to reduce researchers’ ‘time to science.'”
Princeton Consultants‘ high-frequency trading financial research application, OptiSpotter, took the runner-up award of $1,000 in AWS credit.
“In the investing world, as in many others, speed to result is a crucial competitive advantage. Princeton Consultants’ realized that the computational scale and cost-competitiveness that can be achieved on Spot Instances would allow startup hedge funds to master the sheer quantity of financial data (hundreds of terabytes) and compete against the dominant firms by enabling them to rapidly and inexpensively test and tune new investment theses,” Elliott said. “With OptiSpotter, researchers consume tens of thousands of instance hours on Spot and save up to 90% on their compute bill. More importantly, they can get feedback on their investment theses in hours or less, meaning they can iterate and tune an idea several times a day, rather than having to wait until the next morning (or for days) to back-test a new algorithm.”
For helping a startup hedge fund greatly accelerate its research and reduce costs, and compete successfully against some of the world’s largest funds, Princeton Consultants, an IT and management consulting firm stood out, Elliott said.
Princeton Consultants was retained by a high-frequency hedge fund to improve its ability to tune and devise new trading algorithms that trade hundreds of millions of dollars per day. A typical quantitative research project could require reading hundreds of terabytes of market data and tens of thousands of hours of processing. Some early quantitative research organizations are already using Amazon Web Services (AWS) for their research, but in general, they are using AWS On-Demand Instances, which allows for hourly rental of computers and disk.
Amazon Web Services Awards EC2 Spot Instances Users
Princeton Consultants built OptiSpotter, which uses custom algorithms to optimally bid on and consume AWS Spot Instances. Spot Instances have prices and availability that fluctuate constantly, and running Spot Instance jobs can be cancelled by Amazon at any time without notice. In exchange, Spot Instance hourly prices are often at 10 percent or less of On-Demand prices.
Elliott further explained the AWS Spot Instances offering: “Spot Instances are excess EC2 instances whose price is based on real-time supply and demand. When you request an EC2 Instance via a Spot request, you specify a bid price; as long as your bid exceeds the Spot price, you are provisioned a Spot Instance. When the Spot price exceeds your bid, your Spot Instance is interrupted.”
OptiSpotter won the Amazon award for its innovative approach that transforms research computing from massive, multi-hour On-Demand jobs into hundreds of smaller pieces that can effectively use Spot Instances. The hedge fund’s research turnaround time was sped up five to ten times—a critical competitive advantage in today’s fast-moving market conditions—and its costs were reduced 90 percent.
“Princeton Consultants’ OptiSpotter is an exciting, innovative application taking advantage of AWS Spot Instances that dramatically improves the speed and cost-effectiveness of big data quantitative research,” Matt Wood, chief data scientist at AWS, said in a statement. “The judges of our first Spotathon coding challenge were impressed with their creative approach.”
“We see a new world in research and data processing,” said Steve Sashihara, CEO of Princeton Consultants, in a statement. “For time-sensitive big data research, renting processing capacity from AWS is an attractive alternative to buying and maintaining supercomputers. Our approach and OptiSpotter application make access to AWS through the use of Spot Instances even more compelling.
Sashihara said he sees potential use of OptiSpotter by all sizes of organizations in which researchers seek significantly faster throughput and lower processing costs in the Amazon cloud. Princeton Consultants customizes OptiSpotter to the client’s needs.
“OptiSpotter maps massive, multi-hour jobs into thousands of small sub-jobs, queues them based on memory and I/O requirements, then it monitors the Spot price history and queues of outstanding jobs to determine the most efficient way to deploy Spot Instances,” Elliott said.
Meanwhile, AWS gave honorable mention to a couple of other organizations for their use of Spot Instances. The first is Numerate‘s drug discovery application built on its Numatix platform. Numatix accelerates drug discovery while reducing EC2 compute costs by more than 80 percent, Elliott said.
“Numerate’s proprietary machine-learning algorithms predict the properties of small (drug-like) molecules and run Numatix on EC2 Spot Instances to scale to 10,000 cores to search large sets of molecules (>100 million) and identify those likely to lead to new drugs,” Elliott said in his post. “All this for $100 per hour. Numerate’s use of Spot Instances enables them to search enormous chemistry spaces in hours, and flexibly decide how fast they require results and how deep to conduct their analyses. Numerate plans to open Numatix up for broader use beyond drug discovery and is another exemplary case of a powerful cloud solution that reduces computational costs and time-to-results so that scientists can rapidly iterate on their discoveries.”
The second honorable mention went to Lawrence Berkeley National Laboratories’ (LBL) Turbine Science Gateway (TSG), which supports the Department of Energy’s Carbon Capture Simulation Initiative (CCSI) by providing a Web application and execution environment for running and managing scientific applications and storing and archiving results, Elliott said.
“Utilizing TSG, simulation runs that would take months on a single machine can be done overnight on EC2, running tens of thousands of simulations on hundreds of Spot instances and saving over 70 percent on EC2 compute costs.”