2

For hosting our web sites in a Windows load-balanced environment we've traditionally used DFSR to sync folders between servers with our hosting company.

We're looking to migrate a lot of our sites to Amazon EC2 and I'm playing around with load balanced setups at the moment. That's all going fine, but now I'm onto trying to get folders/drives to sync between web servers.

Ideally I'd like to use DFSR, but it relies on having a domain controller in order to sync folders. After some research into setting up a DC on EC2 it appears that this is quite tricky (and somewhat outside my area of expertise as a developer) but I've seen it said once or twice that it's not necessarily that reliable?

So far alternatives I've looked into:

  • Network share from www1 to www2 - if this is on a single web server then it provides a single point of failure (plus we've had performance issues with this).
  • Network share on a separate "micro" Windows instance - again, provides a single point of failure as if this instance goes down then the whole site
  • Cached network drive - I investigated using a cached network drive to get around the "single point of failure" issue, however, it's tricky to get a persistent networked drive that IIS will be able to see, and even then, I couldn't see a way to cache it (plus I'm not sure how quickly changes would be replicated between machines).
  • Robocopy - from what I've seen this this could be useful if I ran it every minute, but it looks like it won't copy locked files? Also, there would be a lag in sync-times

Sorry if that's not explained very well or I've missed something obvious. Has anyone come up with a good, reliable method of syncing folders between web servers on Amazon EC2 which is as good as DFSR (or close)?

tristankoffee
  • 129
  • 1
    windows 2012 has the ability to support file server scale out, I'm just not sure if amazon support this process and if its economical. – tony roth Aug 12 '13 at 15:01
  • Amazon says that 2012 pricing is the same as any other version of windows, and that you can create a Storage Space on EBS volumes. So, this could easily be the winner. – mfinni Aug 12 '13 at 15:29
  • Interesting - I'll take a look at that today - thanks guys :) – tristankoffee Aug 13 '13 at 08:44
  • How often do your files really change? – Dusan Bajic Aug 13 '13 at 08:49
  • @dusan.bajic, it depends on the client and website, we generally need it for CMS sites where the client will be regularly uploading content and assets. – tristankoffee Aug 13 '13 at 16:38
  • @mfinni - I've been reading up on storage spaces and I can't see a solid example of using Storage Spaces to mirror content across two servers. In fact, I saw it said that SS aren't a replication solution - do you have an example of using it to replicate content across two servers? – tristankoffee Aug 13 '13 at 16:39
  • Nope, I was riffing on tony's comment. Without a domain, you can't do DFSR, AFAICT. And if storage spaces don't do replication, then that's not an alternative. The proper answer is to use highly-available shared storage if out-of-sync copies will ever be a problem. DFSR would present the same problem; it's not instantaneous, and it doesn't handle multiple-server locking. – mfinni Aug 13 '13 at 16:43
  • why is setting up a dc in ec2 so difficult? – tony roth Aug 13 '13 at 18:51
  • also I was not thinking of storage spaces, just guest level clustering via smb3. – tony roth Aug 13 '13 at 18:54
  • @mfinni - we've been using DFSR for years and it's met our needs without any issue - it's always been pretty much instantaneous even when releasing large amounts of files it syncs within seconds. – tristankoffee Aug 14 '13 at 08:45
  • @tonyroth I'm not a server admin and I have no experience in setting up DC. All the guides I've seen online indicate that I'd need to spin up at least 2 more servers and that, ideally start using a VPC in order to get private static IPs. This presents two problems 1) cost; we're using EC2 primarily to reduce cost, so having to spin up extra servers just as DCs 2) my complete lack of experience in this area would be a cause for concern should something go wrong and cause outages for our clients. If you know of a good guide that shows how to set up a DC simply in the cloud that would be awesome – tristankoffee Aug 14 '13 at 08:48
  • @tonyroth That's not what a Scale Out File Server is or how it works. – MDMarra Aug 08 '15 at 00:47
  • Check out cloudberry lab's s3 sync software called cloudberry box –  Dec 08 '15 at 19:23

1 Answers1

1

Use S3 for user generated content (one bucket that all webservers will have access to), and keep application on EC2 webservers?

Dusan Bajic
  • 2,076
  • Thanks for the suggestion, but we are moving existing applications onto EC2, some of which would require a fair amount of dev to integrate with S3. Do you know of a reliable way of mapping S3 to a drive on Windows server? I'd be looking for something that works like Dropbox but for S3 (i.e. content is stored locally, but synced between S3 as it changes locally/on S3) – tristankoffee Aug 14 '13 at 08:51
  • You can find some third party tools recommendations on AWS forums but I believe none of them is officially supported by Amazon. I've successfully used s3.exe from http://s3.codeplex.com/ to sync windows folder with s3 bucket but not for real-time sync. – Dusan Bajic Aug 14 '13 at 10:11