25

An Amazon EC2 compute cluster costs about \$800-\$1000 (depending on duty cycle) per physical CPU core over the course of 3 years. In our last round of hardware acquisition, my lab picked up 48 cores worth of hardware very similar to that of Amazon's clusters for about ~$300 a core.

Am I missing something here? Are there any situations in which it makes economic sense to build a cluster in the cloud for high CPU tasks such as molecular dynamics simulations? Or am I always better off just building and baby-sitting the dang machine myself?

(I should mention that my lab doesn't pay for electricity in our server room (at least not directly), but even with this benefit Amazon still seems extremely expensive).

tel
  • 1,380
  • 12
  • 23
  • 2
    One thing I can tell you is that you're missing some labor costs for cluster administration and maintenance. In an academic setting, someone has to be the cluster admin and do all of the grunt work, like OS system updates and upgrades, new software installation, tech support when the cluster goes down, and so on. These tasks are thankless, and depending on administrator skill, may require a lot of time. An EC2 cluster would cut down on the man-hours required for cluster maintenance. – Geoff Oxberry May 08 '12 at 07:02
  • 1
    Well, you've obviously got the hardware cheap. I've calculated the costs over 3 years for the 192-core cluster at my former workplace and it turned out to be over $850 per core per year. Not to mention all the problems we've had with the power system and the cooling... – Hristo Iliev May 08 '12 at 07:37
  • @GeoffOxberry You make a good point about the labor costs. In my lab I'm the admin, so I don't think about such things. As frustrating as it can be sometimes, it's a labor of love. If I had the cash and somewhere to put it, I'd follow the (sparse) instructions on this blog and try to build my own Watson. – tel May 08 '12 at 16:22
  • @HristoIliev I don't even think it was that cheap. If you're willing to take the "lightly used" clusters from Dell, you can push the cost down to ~$200 a core. Can you tell me more about the hardware at your former workplace? – tel May 08 '12 at 16:26
  • 1
    It is a custom built system with 12 twin Supermicro dual-socket Xeon chassis (24 nodes, 48 E5420s) with 16 GiB ECC RAM on each node, one single-socket Nehalem machine with 2 Tesla M2090s, a 24-port InfiniBand switch, one file server with 4 disks, two 10 kVA UPSs, two air conditioners. We obtained it in several phases throughout a three years project period. Not the greatest of the greatest but still ~100k EUR in total (best academic prices in Bulgaria). – Hristo Iliev May 08 '12 at 17:05

5 Answers5

16

The main advantage, in my opinion, of using Cloud-based resources is flexibility, i.e. if you have a fluctuating workload, you only pay for what you need.

If this is not the case in your application, i.e. you know you will have a quantifiable and constant workload, then you're probably better-off building your own cluster. In the Cloud, you pay for flexibility, and if you don't need flexibility, you would be paying for something you don't need.

If your workload is flexible but somewhat intense and relies on certain hardware features (see aeismail's answer), you may want to try sharing a cluster with other people in your university to amortize the idle cycles. My old university runs such a shared cluster with a "Shareholder Model" in which every group is guaranteed a share of the computing power proportional to their investment in the hardware and idle cycles can be used by anyone. The only difficulty is centralizing the cluster administration.

Pedro
  • 9,573
  • 1
  • 36
  • 45
  • 3
    Spot on, the key being 'quantifiable and constant'. Usually workload varies significantly, and it's quite possible for underusage to make that cost much higher than $300/core. Also, in setting up cloud compute it is easy to scale up to more instances if needed temporarily (the week before a conference?). – Phil H May 08 '12 at 11:03
  • 1
    +1 for this. My use of clusters goes from 0 to a period case of "You did what!?". I can't afford to pay for the second one to be around whenever I need it. – Fomite May 09 '12 at 09:24
7

There are some things to worry about when doing cloud computing with MD simulations. For instance, you need to worry about the physical layout of the processors in the server farm where these jobs will be running. The reason is that, depending upon the size of your simulations, and the kinds of calculations you're running (for instance, systems with electrostatics), you may be heavily reliant on FFT's—and pushing electrons around to different processors in a gigantic cluster could become a very time-consuming part of the total compute time.

Also, for something as data-intensive as MD, you'll want to make sure that you have fast upload and download connections to the servers, as well as sensible limits on data storage. Otherwise, a lot of the cost savings could get sucked away in lost productivity and storage fees.

For what it's worth, our institute just purchased about ~240 cores for our local cluster at a cost of under 500€ per core. That cost includes hosting and administration, plus service, on our campus for four years. On an annualized basis that seems ridiculously cheap. I think that's probably the best of both worlds—local access, but professionally maintained without needing our own IT team.

aeismail
  • 3,523
  • 1
  • 18
  • 31
6

I have no firsthand experience with cloud services like Amazon's EC2, but the actual cost per core is likely much greater than you cite: it's the cost of initial purchase, electricity, cooling, space in a building, replacement hardware. Plus the cost of administration: setting up the OS and cluster services, keeping the OS up to date, troubleshooting the queue, etc. I would not at all be surprised if the sum of all of this is twice the cost of initial purchase. Of course you gain flexibility.

To me, the model comes down to scale: If you have a truly large cluster (1000 cores or more) then you can amortize work time, repairs, system administration because there is enough to do to keep a professional busy. If you have a small cluster where it's not worth having a dedicated person do it, then it's likely that you make someone do it whose first job should be to do science, and in that case this person's time is poorly spent on such adminstratorial jobs. This is where services-on-demand such as cloud servers shine.

Wolfgang Bangerth
  • 55,373
  • 59
  • 119
  • At my lab the electricity, cooling, and space for our cluster is all paid for out of the cut our university takes out of our grants for facilities fees. This cut is the same whether we're running a cluster or not. Do you know if there's a similar situation at most universities, or are most labs stuck paying for cooling etc. directly out of pocket? – tel May 08 '12 at 16:33
  • 1
    I believe most labs have the same arrangement as yours, but it would be wrong to ignore these costs anyway. Someone will have to shoulder them, even if it's not you personally. It might be worth asking the department/university to get a bigger share of the indirect cost returned to you if you agree to build your cluster virtually in the cloud rather than physically on campus. – Wolfgang Bangerth May 08 '12 at 21:46
6

As a supplement to some of the already excellent answers, there's another factor to consider:

  • Regardless of the costs, how are you going to pay for it?

I've encountered a non-trivial number of grants that will not under any circumstances pay for hardware expenses, but will pay for computing time on something like EC2. So under some funding circumstances, while you might be able to fund a small "testbed" cluster with unstructured funds or a lab startup package, for larger-scale projects it may be the only way to have your computing costs funded.

Consider the NIH:

ADP/Computer Services: The services you include here should be research specific computer services- such as reserving computing time on supercomputers or getting specialized software to help run your statistics. This section should not include your standard desktop office computer, laptop, or the standard tech support provided by your institution. Those types of charges should come out of the F&A costs.

While it's possible to put cluster machines down under the $5,000+ equipment heading, and you can make a good argument for it, I've found both reviewers who are skittish about it, and universities that are hesitant about the ongoing costs of maintaining such a system.

Some grants are even more strict. One grant I currently have reads as follows:

Funds may also not be used for computer hardware

It's often simply easier to get a cluster paid for by direct costs if its EC2-based or one of its many analogs than actually buying the hardware, especially if your institution is stingy with the indirect costs. This may not be the case for you, but its the case for some.

Fomite
  • 2,383
  • 1
  • 18
  • 25
  • I think that's not a good argument. I am only familiar with the funding system in the US but there if you put a certain amount of money into the "Equipment" category you can use it to buy a cluster. Of course, if that category is empty, then you've requested the wrong set of dollars. Equipment actually has a nice advantage in that if it's a single piece of equipment that's above $5k purchase price you don't pay overhead on it at all (at least at our university). – Wolfgang Bangerth May 09 '12 at 16:52
  • @WolfgangBangerth See my edit for details - putting it under the "Equipment" category may not be allowable. – Fomite May 09 '12 at 18:50
  • Yes, if these are the conditions then there's nothing you can do. But I've found that if you have a good case why you ask for equipment money in a grant then reviewers usually go with it -- it's not usually a large fraction of the overall sum anyway. Of course, if the money hasn't been budgeted to begin with, there's nothing you can do after the fact. – Wolfgang Bangerth May 10 '12 at 09:46
0

There are a lot of god answers already. I just want to add two small points.

1.) I know that IBM is offering something similar. So depending on your application it might be worth comparing the different offers. Not only with respect to the cost but also the equipment.

2.) Of course most labs are too small to be able to invest into their own cluster. Thus the question is whether it is not possible to gather several labs together and invest in a cluster that is shared between the different labs. This has besides the obvious cost sharing effect also the advantage of a more even load distribution.

Azrael3000
  • 307
  • 1
  • 8