0

Companies such as Amazon, Etsy, Google, Facebook and others are publicly saying they deploy software thousands time per day.

What are they doing that is so different from other companies? How can other software companies deploy software thousands times per day, what is stopping them from doing so today?

030
  • 13,235
  • 16
  • 74
  • 173
Evgeny Zislis
  • 8,963
  • 5
  • 38
  • 72
  • 1
    This question is too broad to be answered in a couple of phrases, so, here is the broad answer: You need to automate the way you build, test and deliver software. So, what is stopping other companies to do the same? They don't have that level of automation. – Gepser Hoil Feb 28 '17 at 17:59
  • The main thing that those companies have are hundreds of employees who are able to push code because the deployment process is automated. If you only have 20 developers, you won't be deploying thousands of times a day. So it's not really a devops question, just a math question. – avi Feb 28 '17 at 18:06
  • I'm voting to close this question as off-topic because it's more a question about how development groups inside of large cooperations work than it is about DevOps itself. – avi Feb 28 '17 at 18:08
  • Related: https://devops.stackexchange.com/questions/2635/from-6-months-to-16-seconds-grokking-devops-speedup-data-from-devops-report-by – Dan Cornilescu Nov 30 '17 at 07:39

3 Answers3

3

The "how" of extremely high-frequency deployment is all about very mature engineering practices and tooling. The "unicorns" (Gene Kim's term for the type of company you list) have some of the best talent in the industry, and invest in the best technology.

One of several key practical differences between these unicorns and most other orgs is the level of automation they employ. It is very difficult to automate the quality checks needed to maintain a constant flow of work from DEV, through testing environments, to Production. Development and Operations must maintain high service levels internally. Ops must provide self-service resources so developers aren't waiting on tickets to get new infrastructure.

Dave Swersky
  • 4,068
  • 2
  • 20
  • 33
  • According to Adrian Cockroft, other companies had these people before. Yet they didn't achieve such speed. He doesn't explain exactly how Netflix does this though. https://image.slidesharecdn.com/monktoberfestpdf-141003084814-phpapp01/95/monktoberfest-fast-delivery-65-638.jpg – Evgeny Zislis Feb 28 '17 at 18:08
2

These big companies are using unconventional solutions, such as the BitTorrent deployment system and geographically distributed content delivery networks (CDNs). When Facebook updates its code or generates a new build, the binary files needs be pushed to all of the company's servers (~1.5GB), so they created its own custom BitTorrent tracker designed to obtain data slices from other servers that are on the same node/rack which reduces total latency.

Source: Exclusive: a behind-the-scenes look at Facebook release engineering

Facebook deployment takes the following steps:

  1. Update goes to an "a2" tier which rolls update out to a small, random collection of users.
  2. Check-in procedure on company IRC where all developers who submitted code for the update have confirm that they are ready to respond if there is a problem when the update goes out.
  3. Head of release engineering issues command in terminal to begin deployment
  4. Watch web-based monitoring dashboards as the update rolls out.

Frequency of deployments is one minor update on most business days, one major update on a weekly basis.

Source: How Facebook Does Deployment

kenorb
  • 7,841
  • 12
  • 40
  • 77
1

Not exactly a full answer, but you need mostly three things:

  • Automation of the delivery pipeline (Continuous Delivery)
  • A very, very strong test system (including canary release)
  • Good coding practices to ensure versions can run concurrently on the same environment.

This usually include tooling for code review, code linting, unit testing, orchestration, configuration management, integration testing and a strong culture for the change.

Tensibai
  • 11,366
  • 2
  • 35
  • 62