eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
In the past couple of years, we’ve been hearing the term “cloud computing” a great deal. However, from my experience working with people across various industries, many are still unsure what exactly the term means.
Recently, Amazon entered the space with its own version of cloud computing. In this article, I’ll show exactly what Amazon is up to.
But First: Why Amazon?
Amazon isn’t the most likely company to get into the cloud computing business. It would seem that would be left to vendors such as IBM and Oracle.
Here’s how Amazon fell into this world. When a person visits the Amazon Web site and is logged in, he’ll see a storefront that has been customized for him. These recommendations aren’t somehow stored in cookies and figured out by the user’s browser; they are determined on the fly completely by the Amazon servers.
For a Web developer, figuring out such recommendations would be easy: When the user logs in, the latest purchases are looked up and a database query for similar items is done and they are shown on the page. Except there’s a huge problem: Can you realistically do such calculations from a few lines of code for millions of users simultaneously without taking minutes or hours?
Most Web servers, while certainly powerful, simply aren’t equipped for this kind of power. As such, the engineers at Amazon were forced to come up with some new ways of handling so many database transactions and software calculations so quickly. Their solution was to use a form of distributed computing where there are multiple computers, each devoted to simple tasks, all communicating and working together.
Can you bet on the cloud? Find out here.
As a software developer, many times I’ve been in a situation where I’ve had to explain to people that while lots of people can crank out some code, doing it right often requires teams of people, possibly even an entire company. That’s why, for example, you’ll find highly specialized software companies providing software to a very specific industry. Why can’t businesses just build their own software? They’re not in the business of doing so. Realistically, the most cost-efficient solution is often to just buy the software from a company dedicated to building such software.
There are exceptions to this rule. Sometimes the software a company needs simply doesn’t exist. Look at Amazon. Its primary business is selling books and, more recently, department store items. But to keep up with the sheer size of its customer base, Amazon needed both hardware and software that could handle its operations. Unfortunately, in Amazon’s earliest days, such systems simply didn’t exist. Amazon engineers had no choice but to build it themselves.
They were forced to build a massive, distributed system that could handle its retail operation. When it was finished, they realized that they had something big on their hands that could potentially be used by many other people. The result was AWS (Amazon Web Services), launched in 2002.
Amazons Elastic Compute Cloud
The umbrella system known as AWS encompasses many different technologies. One of the newest technologies is called Amazon Elastic Compute Cloud, or EC2 for short.
When Amazon engineers were first developing AWS, they decided that, in addition to simply distributing the work among separate computers, they would make sure the individual computers offered their services on an as-needed basis.
For example, if someone logs into a large site such as Amazon and wants to explore DVDs, there’s no reason for the server handling the comment boards to set aside storage space for that session if that person isn’t using the comment boards. Amazon has built its large distributed system to allocate resources only when they’re needed and then to let go of the resources when they’re no longer needed.
In a small Web server environment, such allocation is done quite a bit, as programmers know how to allocate objects when necessary. On a large system, this becomes especially important, and this is where the concept of cloud computing comes in. In the cloud computing world, each piece of the large Web server becomes a service, dedicated to a particular task. These services are run on separate hardware. Software running on other computers can access the service as needed.
That’s what cloud computing really is: multiple systems providing services to each other. Software functions as a service (hence the phrase “software as a service”). But in the case of cloud computing, these individual systems are usually themselves huge; they’re not just individual Web servers, but rather more likely huge data warehouses. When all the big pieces are working together as a whole spread out about the Internet, the whole shebang is called the cloud.
The modern cloud has advantages over SAAS, grids and utility computing. Find out why.
Now take this concept to the next level: The individual services are built so they’re only used when needed. Imagine that users want to make use of some of these services for their own software. They’d like to run some software on the Web that could benefit by being placed on one of these services and their software would make use of a few other services. They don’t need all of the services available, but only a portion.
That’s what Amazon’s EC2 is all about. They can simply rent space on the Amazon cloud and only pay for the parts they need and use. Their software can, while it’s running, allocate space (even the virtual equivalent of additional computers; I’ll get to that shortly). Everything will be available to the users, and they’ll only be charged for what they use.
To customers, this can be a huge benefit, because their software gets to run on a huge system that’s capable of crunching through millions of transactions a day; they don’t have to actually own the physical hardware. They also don’t need to simply subscribe to a hosting service where they get a virtual area of a single physical PC sitting in a data center. The result is that their software functions as powerfully as the big players.
This cloud approach to computing isn’t being done exclusively by Amazon. However, Amazon’s own system is called the Elastic Compute Cloud because engineers consider it elastic. A user’s software can request services on the fly and grow as needed while it’s running, and then let go of resources and shrink back down when there’s less need. In other words, the software is elastic, growing on an as-needed basis, with users charged for the services they use. That is contrary to a traditional hosting service, where users pay a fixed price per month for a fixed limit on resources.
There are some similarities to traditional hosting. When customers have an application running under EC2, they create a virtual server much like the virtual servers in smaller hosting providers. From there, their system can grow as needed by adding additional virtual servers. These virtual servers are basically the smallest atomic unit available.
Next, as their software runs, they might have coded their software to determine that it needs additional virtual machines while it’s running. Their software might need to do some serious number crunching, and they don’t want to devote their current virtual machine to this process. They can allocate a second virtual machine to this process, a virtual machine that has all the power of a single stand-alone PC. When the crunching is complete, the second virtual machine can provide the results of the crunching back to the first virtual machine, and then shut down so another Amazon customer can use the process time.
Fewer Development Nightmares
Building such an elastic computing environment would need some serious computing power and, likely, a server room full of computers devoted to the task. With multiple computers, engineers would likely have the additional requirement that they want the system to function on multiple virtual machines that are allocated as needed so that the system is efficient.
Suddenly, a small project is getting big. Young, ambitious programmers would probably salivate at the prospect of creating such a system. But remember what I said earlier. What business are you in? This system they’d be building is not trivial and is certainly not something a lone programmer could throw together over a weekend.
That’s why companies are intrigued by what Amazon offers: a massively parallel infrastructure that allows a customer’s application to scale on demand, all without them having to worry about the nuts and bolts of making it work or the financial issues of maintaining their own data center.
How many cloud computing platforms can we handle? Read more here.
In researching this article, I looked at much of the official documentation from Amazon, including one that describes the architecture. This article gives some great examples of how the EC2 architecture and other aspects of AWS can be a benefit; the one that caught my attention is an example of a system that needs to convert millions of pages of documents from one format, such as Microsoft Word, to another, such as PDF.
There are plenty of desktop applications that can do such conversions, and there are some that are available through a Web server, where a user can upload the document and transform it to PDF.
However, imagine hosting a Web site where users across the planet can upload their documents, and at times there could be tens of thousands of documents queued up. Such a job would be a nightmare. I would probably quit my job and move to another country the day such a system went live.
However, doing so in a cloud environment changes things. The massively parallel cloud environment could easily handle the system, and I probably wouldn’t even need to have my cell phone turned on; I could actually sleep through the night.
Technical Resources for EC2
The EC2 environment is built on some proven technologies, including Hadoop, an open-source framework for developing distributed applications that is written in Java (yes, Java really can handle heavy-duty tasks).
Earlier I mentioned that Amazon offers virtual machines. When setting up a virtual machine, users can choose what it should look like by specifying an AMI (Amazon Machine Image). For example, they can set up a virtual machine that is simply a standard Linux distribution based on a 2007 32-bit Xeon processor with 1.7GB of memory and 160GB of storage. From there they can then launch other virtual machines as needed. Right now the limit is 20, although a big customer can work with Amazon for more.
Take a look at the main EC2 page here.
This page is an overview of EC2 but also includes a list of what’s available when configuring an AMI, as well as links at the bottom to many developer pages and documentation.
Programmers will want to take a look at something one of the Hadoop developers has done, a sample that runs on EC2. The tutorial, written by Tom White, one of the Hadoop developers who has also worked on the EC2 project, includes sample code as well as a rundown of the architecture and how to get the most of it.
Amazon also has a huge set of resources available for developers. Included are tutorials, official documentation and tools.
In addition to going to the main developer page, I recommend checking out this page. It includes a link to a Getting Started Guide as well as a Developer Guide.
Amazon has made several tools available to help programmers, including command-line tools that they can run for configuring their systems. For example, one set of tools can be found here. These are the tools for registering and launching Web services.
In addition, developers can find more tools here. These are the tools for setting up a virtual machine.
Amazon also has a huge discussion forum with a lot of great questions and answers.
Conclusion
Early on, I was just as skeptical as the next guy about AWS, and especially Amazon’s break into the cloud world. Sure, it seemed cool, but I didn’t totally understand the point in it. Now I do. Imagine creating Web software that can easily scale so that millions of people can access it without it crashing. This is powerful stuff. That’s where something like EC2 comes in. I know I’m sold.
Senior Editor Jeff Cogswell can be reached at [email protected].