Those who do not understand UNIX are condemned to reinvent it, poorly.

Henry Spencer

Who am I?

CTO Office

CI/CD and Automation

Zuul

Ansible

Who am I?

Developer Infrastructure Core Team

Shade PTL

Technical Committee (for another week)

Former Foundation Board of Directors

Infrastructure PTL Emeritus

In the beginning ...

Google's architecture features clusters of more than 15,000 commodity class PCs with fault-tolerant software. This architecture achieves superior performance at a fraction of the cost of a system built from fewer, but more expensive, high-end servers.

Web Search for a Planet: The Google Cluster Architecture

Luiz Andre Barroso, Jeffrey Dean, Urs Hölzle - 2003

https://research.google.com/pubs/pub49.html

• Assume failure
• Use cheap servers, not expensive servers
• Scale out, not up

Shared Nothing

Who was I?

Professional Services

High Availability

Scaling

Cluster

I asked for a crossover cable...

Why MySQL? Wouldn’t NoSQL databases, for example, be better suited for the massive workloads seen at Facebook?

I have not been able to find a transactional NoSQL database better than InnoDB. And it’s easy to understand how MySQL Replication works, which makes much easier to fix problems in production.

Yoshinori Matsunobu, Facebook Engineering, 2014

Lessons from MySQL

• Simple is better than complex
• Know how your software works
• How something fails is more important than how something works

Who was I?

Professional Services

High Availability

Scaling

Cluster

Lessons from Sun

• Simple is better than complex
• Assume failure
• Use cheap servers, not expensive servers
• Scale out, not up

Who was I?

Fork of MySQL

Modern C++0x

Microkernel Design

Why have one when you can have two at twice the price?

"A Database For The Cloud"

• Removed bloat (triggers, stored procedures, mediumint)
• Sensible Defaults - no config needed
• Moved data dictionary into InnoDB tablespace
• Immediate Ancestor of OpenStack's Gating

We used to sit around in the Unix Room saying, 'What can we throw out? Why is there this option?' It's often because there is some deficiency in the basic design — you didn't really hit the right design point. Instead of adding an option, think about what was forcing you to add that option.

Doug McIlroy, 2005

UNIX philosophy

• Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".
• Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.
• Design and build software, even operating systems, to be tried early, ideally within weeks. Don't hesitate to throw away the clumsy parts and rebuild them.
• Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you've finished using them.

Doug McIlroy, Bell System Technical Journal, 1978

the power of a system comes more from the relationships among programs than from the programs themselves

The Unix Programming Environment

Brian Kernighan and Rob Pike

Cloud Native Is ...

• Architectural and operational approach
• Assume cloud
• Assume failures
• Microservices
• Containerized?

12 Factor Application

I. Codebase - One codebase tracked in revision control, many deploys

II. Dependencies - Explicitly declare and isolate dependencies

III. Config - Store config in the environment

IV. Backing services - Treat backing services as attached resources

V. Build, release, run - Strictly separate build and run stages

VI. Processes - Execute the app as one or more stateless processes

VII. Port binding - Export services via port binding

VIII. Concurrency - Scale out via the process model

IX. Disposability - Maximize robustness with fast startup and graceful shutdown

X. Dev/prod parity - Keep development and production as similar as possible

XI. Logs - Treat logs as event streams

XII. Admin processes - Run admin/management tasks as one-off processes

This is awesome

Except for III

I use config files

Ooops

VI. Stateless

If /dev/null is fast in web scale I will use it. Is it web scale?

Use a service to store your data - like a database

Is that database service web scale?

If /dev/null is fast in web scale I will use it. Is it web scale?

MongoDB is web scale

http://www.mongodb-is-web-scale.com

The tragedy of modern man is not that he knows less and less about the meaning of his own life, but that it bothers him less and less.

Václav Havel

As computation continues to move into the cloud, the computing platform of interest no longer resembles a pizza box or a refrigerator, but a warehouse full of computers. ... in other words, we must treat the datacenter itself as one massive warehouse-scale computer.

The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines

Luiz André Barroso, Urs Hölzle - 2009

https://research.google.com/pubs/pub35290.html

The use of top-down design methods and high-level languages in producing portable applications software is well established. By applying the same principles at the systems programming level, portability can be extended to the operating system itself. Although the Unix operating system was developed for a specific computer (the DEC PDPll), its concise and elegant design and the careful selection of 'primitives' which it provides make it an ideal candidate for portability.

UNIX: a portable operating system?

Richard Miller - 1978

If the datacenter is the new computer ...

Then the power of OpenStack is as a portable Operating System

Just as MVS was an Operating System for IBM System/370 ...

AWS is an Operating System specific to Amazon Datacenters

Google Cloud is an Operating System specific to Google Datacenters

If sets of cheaper commodity servers are superior to fancy custom built high-end hardware ...

Then commodity data centers with a common Operating System should be better than fancy data centers controlled by one or two companies

Linux won the battle for single-computer Operating Systems.

Let's win the battle for a Free, Open and Portable Operating System for warehouse-scale computers.