基于容器的分布式系统设计模式
Design patterns for container-based distributed systems Brendan Burns David Oppenheimer Google
1 引言（Introduction）
20世纪80年代末90年代初，面向对象编程革新了软件开发，普及了以模块化组件构建应用的理念。如今，分布式系统开发正经历一场类似的革命：由容器化软件组件构建的微服务架构日益流行。凭借容器边界形成的隔离特性，容器特别适合作为分布式系统的基础“对象”。随着这种架构风格走向成熟，设计模式开始涌现——正如当年面向对象编程一样，原因也相同：以对象（或容器）为思考单元，抽象底层代码细节，最终提炼出适用于各类应用与算法的高层模式。
In the late 1980s and early 1990s, object-oriented programming revolutionized software development, popularizing the approach of building of applications as collections of modular components. Today we are seeing a similar revolution in distributed system development, with the increasing popularity of microservice architectures built from containerized software components. Containers are particularly well-suited as the fundamental “object” in distributed systems by virtue of the walls they erect at the container boundary. As this architectural style matures, we are seeing the emergence of design patterns, much as we did for object-oriented programs, and for the same reason – thinking in terms of objects (or containers) abstracts away the low-level details of code, eventually revealing higher-level patterns that are common to a variety of applications and algorithms.
本文梳理了基于容器的分布式系统中涌现的三类设计模式：单容器管理模式、单节点多容器协作模式、面向分布式算法的多节点模式。与此前的面向对象模式一样，这些分布式计算模式沉淀了最佳实践、简化开发流程，并提升所构建系统的可靠性。
This paper describes three types of design patterns that we have observed emerging in container-based distributed systems: single-container patterns for container management, single-node patterns of closely cooperating containers, and multi-node patterns for distributed algorithms. Like object-oriented patterns before them, these patterns for distributed computation encode best practices, simplify development, and make the systems where they are used more reliable.
2 分布式系统设计模式（Distributed system design patterns）
面向对象编程普及数年后，设计模式出现并被系统化整理。这些模式将解决特定常见编程问题的通用方法规范化、法典化。这种规范化进一步提升了编程整体水平：让经验不足的开发者也能写出工程化良好的代码，并推动可复用库的发展，让代码更可靠、开发更高效。
After object-oriented programming had been used for some years, design patterns emerged and were documented. These patterns codified and regularized general approaches to solving particular common programming problems. This codification further improved the general state of the art in programming because it made it easier for less experienced programmers to produce well-engineered code, and led to the development of reusable libraries that made code more reliable and faster to develop.
当今分布式系统工程的现状，更接近20世纪80年代初的编程世界，而非面向对象开发时代。但MapReduce模式成功将“大数据”编程能力普及到众多领域与开发者群体，这清晰表明：一套合适的模式体系能显著提升分布式系统编程的质量、速度与易用性。即便如此，MapReduce的成功仍很大程度局限于单一语言——Apache Hadoop生态主要用Java开发并面向Java。要构建真正全面的分布式系统设计模式集，需要一种通用、语言无关的载体来表示系统的基本单元。
The state-of-the-art in distributed system engineering today looks significantly more like the world of early 1980s programming than it does the world of object-oriented development. Yet it’s clear from the success of the MapReduce pattern in bringing the power of “Big Data” programming to a broad set of fields and developers, that putting in place the right set of patterns can dramatically improve the quality, speed, and accessibility of distributed system programming. But even the success of MapReduce is largely limited to a single programming language, insofar as the Apache Hadoop ecosystem is primarily written in and for Java. Developing a truly comprehensive suite of patterns for distributed system design requires a very generic, language-neutral vehicle to represent the atoms of the system.
所幸过去两年间，Linux容器技术迎来爆发式普及。容器与容器镜像，正是分布式系统模式研发所需的抽象单元。截至目前，容器与容器镜像之所以广受欢迎，核心原因是它提供了从开发到生产全流程更优质、更可靠的软件交付方式。凭借密封封装、自带依赖、原子化部署信号（成功/失败），容器大幅改进了数据中心与云端软件部署的现有方案。但容器的潜力远不止是更好的部署工具——我们认为它注定会成为面向对象系统中“对象”的等价物，并由此推动分布式系统设计模式的发展。后续章节将阐述这一判断，并介绍未来几年可规范与指导分布式系统工程的新兴模式。
Thus it is fortunate that the last two years have seen a dramatic rise in adoption of Linux container technology. The container and the container image are exactly the abstractions needed for the development of distributed systems patterns. To date, containers and container images have achieved the large measure of their popularity simply by being a better, more reliable method for delivering software from development all the way through production. By being hermetically sealed, carrying their dependencies with them, and providing an atomic deployment signal (“succeeded”/“failed”), they dramatically improve on the previous state of the art in deploying software in the datacenter or cloud. But containers have the potential to be much more than just a better deployment vehicle – we believe they are destined to become analogous to objects in object-oriented software systems, and as such will enable the development of distributed system design patterns. In the following sections we explain why we believe this to be the case, and describe some patterns that we see emerging to regularize and guide the engineering of distributed systems over the coming years.
3 单容器管理模式（Single-container management patterns）
容器为定义接口提供了天然边界，与对象边界高度相似。容器可通过该接口对外暴露应用功能，同时提供面向管理系统的钩子。
The container provides a natural boundary for defining an interface, much like the object boundary. Containers can expose not only application-specific functionality, but also hooks for management systems, via this interface.
传统容器管理接口非常有限，容器仅对外提供三个基础操作：run()、pause()、stop()。这套接口虽可用，但更丰富的接口能为开发者与运维人员带来更大价值。得益于现代编程语言几乎普遍支持HTTP服务、JSON等数据格式广泛普及，很容易定义基于HTTP的管理API：容器在主功能之外，在指定端点启动Web服务即可“实现”该接口。
The traditional container management interface is extremely limited. A container effectively exports three verbs: run(), pause(), and stop(). Though this interface is useful, a richer interface can provide even more utility to system developers and operators. And given the ubiquitous support for HTTP web servers in nearly every modern programming language and widespread support for data formats like JSON, it is easy to define an HTTP-based management API that can be “implemented” by having the container host a web server at specific endpoints, in addition to its main functionality.
向上：容器可对外暴露丰富的应用信息，包括应用专属监控指标（QPS、健康状态等）、开发者关注的剖析信息（线程、栈、锁竞争、网络消息统计等）、组件配置与日志。典型案例：Kubernetes、Aurora、Marathon等容器管理系统支持用户通过指定HTTP端点（如/health）定义健康检查。而上述“向上”API中其他能力的标准化支持目前仍较少。
In the “upward” direction the container can expose a rich set of application information, including application-specific monitoring metrics (QPS, application health, etc.), profiling information of interest to developers (threads, stack, lock contention, network message statistics, etc.), component configuration information, and component logs. As a concrete example of this, Kubernetes, Aurora, Marathon, and other container management systems allow users to define health checks via specified HTTP endpoints (e.g. “/health”). Standardized support for other elements of the “upward” API we have described is more rare.
向下：容器接口为定义生命周期提供天然位置，让编写受管理系统控制的软件组件更简单。例如，集群管理系统通常为任务分配优先级，高优先级任务在集群超配时仍保证运行；系统会驱逐已运行的低优先级任务，待资源充足后重新调度。直接杀死低优先级任务虽可实现驱逐，但会给开发者带来沉重负担——需在代码任意位置处理意外终止。若在应用与管理系统之间定义正式生命周期，应用组件遵循约定契约，更易被管理；开发者可依赖契约，系统开发更简单。例如，Kubernetes利用Docker的“优雅删除”特性，在发送SIGKILL前，提前通过SIGTERM告知容器即将终止，让应用完成正在处理的请求、刷盘落态等清理工作。可进一步扩展该机制，支持状态序列化与恢复，大幅降低有状态分布式系统的状态管理难度。
In the “downward” direction, the container interface provides a natural place to define a lifecycle that makes it easier to write software components that are controlled by a management system. For example, a cluster management system will typically assign “priorities” to tasks, with high-priority tasks guaranteed to run even when the cluster is oversubscribed. This guarantee is enforced by evicting already-running lower-priority tasks, that will then have to wait until resources become available. Eviction can be implemented by simply killing the lower-priority task, but this puts an undue burden on the developer to respond to arbitrary death anywhere in their code. If instead, a formal lifecycle is defined between application and management system, then the application components become more manageable, since they conform to a defined contract, and the development of the system becomes easier, since the developer can rely on the contract. For example, Kubernetes uses a “graceful deletion” feature of Docker that warns a container, via the SIGTERM signal, that it is going to be terminated, an application-defined amount of time before it is sent the SIGKILL signal. This allows the application to terminate cleanly by finishing in-flight operations, flushing state to disk, etc. One can imagine extending such a mechanism to provide support for state serialization and recovery that makes state management significantly easier for stateful distributed systems.
更复杂生命周期的典型案例：Android Activity模型，包含一系列回调（onCreate()、onStart()、onStop()等），以及系统触发这些回调的正式状态机。若无这套规范生命周期，开发健壮可靠的Android应用会困难得多。映射到容器系统，可抽象为容器创建、启动、终止前等时机触发的应用自定义钩子。容器支持的另一类“向下”API示例是“自我复制”（用于服务扩容）。
As a concrete example of a more complex lifecycle, consider the Android Activity model, which features a series of callbacks (e.g. onCreate(), onStart(), onStop(), ...) and a formally defined state machine for how the system triggers these callbacks. Without this formal lifecycle, robust, reliable Android applications would be significantly harder to develop. In the context of container-based systems, this generalizes to application-defined hooks that are invoked when a container is created, when it is started, just before termination, etc. Another example of a “downward” API that a container might support is “replicate yourself” (to scale up the service).
4 单节点多容器应用模式（Single-node, multi-container application patterns）
除单容器接口外，跨容器的设计模式也开始涌现。我们此前已总结若干此类模式。这类单节点模式由共生容器组成，被协同调度到同一台主机。因此，容器管理系统必须支持将多个容器作为原子单元协同调度——Kubernetes称之为Pod，Nomad称之为任务组（task groups），这是启用本节模式的必备功能。
Beyond the interface of a single container, we also see design patterns emerging that span containers. We have previously identified several such patterns. These single-node patterns consist of symbiotic containers that are co-scheduled onto a single host machine. Container management system support for co-scheduling multiple containers as an atomic unit, an abstraction Kubernetes calls “Pods” and Nomad calls “task groups,” is thus a required feature for enabling the patterns we describe in this section.
4.1 边车模式（Sidecar pattern）
多容器部署中最常见的模式是边车模式。边车容器扩展并增强主容器能力。例如，主容器是Web服务器，搭配“日志保存”边车容器，从本地磁盘收集Web服务器日志并流式上传到集群存储系统。另一常见案例：Web服务器从本地磁盘提供服务，边车容器定期从Git仓库、内容管理系统或其他数据源同步内容。这两类案例在谷歌广泛使用。同一机器上的容器可共享本地磁盘卷，是边车模式可行的基础。
The first and most common pattern for multi-container deployments is the sidecar pattern. Sidecars extend and enhance the main container. For example, the main container might be a web server, and it might be paired with a “logsaver” sidecar container that collects the web server’s logs from local disk and streams them to a cluster storage system. Another common example is a web server that serves from local disk content that is populated by a sidecar container that periodically synchronizes the content from a git repository, content management system, or other data source. Both of these examples are common at Google. Sidecars are possible because containers on the same machine can share a local disk volume.
虽然可将边车功能直接集成到主容器，但拆分容器有多重优势：
- 容器是资源计量与分配单元，可分别配置Web服务器容器（保证低延迟响应）与日志保存容器（利用空闲CPU）。
- 容器是打包单元，服务与日志保存拆分后，可由不同团队分别开发、独立测试与联合测试。
- 容器是复用单元，边车可与各类主容器搭配（如日志保存容器可用于所有产生日志的组件）。
- 容器是故障隔离边界，系统可优雅降级（如日志保存故障时Web服务仍可继续）。
- 容器是部署单元，各功能可独立升级与回滚。
While it is always possible to build the functionality of a sidecar container into the main container, there are several benefits to using separate containers. First, the container is the unit of resource accounting and allocation, so for example a web server container’s cgroup can be configured so that it provides consistent low-latency responses to queries, while the logsaver container is configured to scavenge spare CPU cycles when the web server is not busy. Second, the container is the unit of packaging, so separating serving and log saving into different containers makes it easy to divide responsibility for their development between two separate programming teams, and allows them to be tested independently as well as together. Third, the container is the unit of reuse, so sidecar containers can be paired with numerous different “main” containers (e.g. a log saver container could be used with any component that produces logs). Fourth, the container provides a failure containment boundary, making it possible for the overall system to degrade gracefully (for example, the web server can continue serving even if the log saver has failed). Lastly, the container is the unit of deployment, which allows each piece of functionality to be upgraded and, when necessary, rolled back, independently.
4.2 大使模式（Ambassador pattern）
第二类模式是大使模式。大使容器代理主容器的进出通信。例如，应用使用memcache协议，搭配twemproxy大使容器。应用以为自己在连接本地单机memcache，实际twemproxy将请求分片到集群中多个memcache节点。该模式从三方面简化开发：只需按连接本地单机服务编程；本地可直接运行真实memcache独立测试；twemproxy大使可复用于不同语言编写的应用。同一机器容器共享本地网络接口，是大使模式可行的基础。
The next pattern that we have observed is the ambassador pattern. Ambassador containers proxy communication to and from a main container. For example, a developer might pair an application that is speaking the memcache protocol with a twemproxy ambassador. The application believes that it is simply talking to a single memcache on localhost, but in reality twemproxy is sharding the requests across a distributed installation of multiple memcache nodes elsewhere in the cluster. This container pattern simplifies the programmer’s life in three ways: they only have to think and program in terms of their application connecting to a single server on localhost, they can test their application standalone by running a real memcache instance on their local machine instead of the ambassador, and they can reuse the twemproxy ambassador with other applications that might even be coded in different languages. Ambassadors are possible because containers on the same machine share the same localhost network interface.
4.3 适配器模式（Adapter pattern）
单节点最后一类模式是适配器模式。与大使模式为应用简化外部世界视图相反，适配器为外部世界提供应用的简化、统一视图，通过标准化多容器的输出与接口实现。典型案例：用适配器保证系统所有容器拥有一致的监控接口。当前应用暴露指标的方式五花八门（JMX、statsd等），若所有应用提供统一监控接口，单一监控工具更易采集、聚合与展示指标。谷歌通过代码规范实现这一点，但仅适用于从零构建的软件。适配器模式可让遗留与开源应用在不修改源码的前提下对外呈现统一接口。主容器可通过本地回环或共享卷与适配器通信。现有部分监控方案虽支持多种后端，但在监控系统内嵌入应用专属代码，关注点分离不如适配器模式清晰。
The final single-node pattern we have observed is the adapter pattern. In contrast to the ambassador pattern, which presents an application with a simplified view of the outside world, adapters present the outside world with a simplified, homogenized view of an application. They do this by standardizing output and interfaces across multiple containers. A concrete example of the adapter pattern is adapters that ensure all containers in a system have the same monitoring interface. Applications today use a wide variety of methods to export their metrics (e.g. JMX, statsd, etc). But it is easier for a single monitoring tool to collect, aggregate, and present metrics from a heterogenous set of applications if all the applications present a consistent monitoring interface. Within Google, we have achieved this via code convention, but this is only possible if you build your software from scratch. The adapter pattern enables the heterogenous world of legacy and open-source applications to present a uniform interface without requiring modification of the original application. The main container can communicate with the adapter through localhost or a shared local volume. Note that while some existing monitoring solutions are able to communicate with multiple types of back-ends, they use application-specific code in the monitoring system itself, which provides a less clean separation of concerns.
5 多节点应用模式（Multi-node application patterns）
跳出单节点协作容器，模块化容器让构建多节点协同分布式应用更简单。本节介绍三类分布式系统模式，同样依赖Pod抽象的系统支持。
Moving beyond cooperating containers on a single machine, modular containers make it easier to build coordinated multi-node distributed applications. We describe three of these distributed system patterns next. Like the patterns in the previous section, these also require system support for the Pod abstraction.
5.1 主节点选举模式（Leader election pattern）
分布式系统最常见问题之一是主节点选举。副本常用于负载分担，更复杂场景是从一组副本中选出“主节点”，其余副本在主节点故障时快速接管。系统可并行执行多组选举（如为多个分片分别选主）。现有大量选举库，但理解与正确使用复杂，且局限于特定编程语言。替代方案是使用选举容器：一组选举容器分别与需要选主的应用实例协同调度，彼此完成选举，并通过本地回环向应用容器提供简化HTTP API（becomeLeader、renewLeadership等）。选举容器可由领域专家一次性构建，简化接口可被任意语言的应用开发者复用，体现软件工程的抽象与封装最佳实践。
One of the most common problems in distributed systems is leader election. While replication is commonly used to share load among multiple identical instances of a component, another, more complex use of replication is in applications that need to distinguish one replica from a set as the “leader.” The other replicas are available to quickly take the place of the leader if it fails. A system may even run multiple leader elections in parallel, for example to determine the leader of each of multiple shards. There are numerous libraries for performing leader election. They are generally complicated to understand and use correctly, and additionally, they are limited by being implemented in a particular programming language. An alternative to linking a leader election library into the application is to use a leader election container. A set of leader-election containers, each one co-scheduled with an instance of the application that requires leader election, can perform election amongst themselves, and they can present a simplified HTTP API over localhost to each application container that requires leader election (e.g. becomeLeader, renewLeadership, etc.). These leader election containers can be built once, by experts in this complicated area, and then the subsequent simplified interface can be re-used by application developers regardless of their choice of implementation language. This represents the best of abstraction and encapsulation in software engineering.
5.2 工作队列模式（Work queue pattern）
工作队列与主节点选举一样是成熟领域，有大量框架实现，但同样能从面向容器架构受益。此前系统中，框架限制程序运行在单一语言环境（如Python的Celery），或任务分发与二进制部署由实现者自行处理（如Condor）。容器提供run()与mount()接口，让构建通用工作队列框架变得简单：可接收任意容器化处理代码与任意数据，搭建完整工作队列系统。开发者只需构建能从文件系统读取输入、输出结果的容器，即可作为工作队列的一个阶段。开发完整工作队列的其余工作，均可由可复用的通用框架处理。
Although work queues, like leader election, are a well-studied subject with many frameworks implementing them, they too are an example of a distributed system pattern that can benefit from container-oriented architectures. In previous systems, the framework limited programs to a single language environment (e.g. Celery for Python), or the distribution of work and binary were exercises left to the implementer (e.g. Condor). The availability of containers that implement the run() and mount() interfaces makes it fairly straightforward to implement a generic work queue framework that can take arbitrary processing code packaged as a container, and arbitrary data, and build a complete work queue system. The developer only has to build a container that can take an input data file on the filesystem, and transform it to an output file; this container would become one stage of the work queue. All of the other work involved in developing a complete work queue can be handled by the generic work queue framework that can be reused whenever such a system is needed.
5.3 分发/聚合模式（Scatter/gather pattern）
最后介绍的分布式系统模式是分发/聚合模式。外部客户端向“根/父”节点发送初始请求，根节点将请求分发到大量服务器并行计算；各分片返回部分结果，根节点汇总为单一响应返回给客户端，搜索引擎广泛使用该模式。开发这类分布式系统涉及大量样板代码：请求分发、结果收集、客户端交互等。这些代码通用性强，可像面向对象编程一样重构：只要容器实现指定接口，即可用同一套实现适配任意容器。具体来说，实现分发/聚合系统需提供两类容器：叶子节点容器（执行分片计算并返回结果）、合并容器（汇总所有叶子输出并生成最终响应）。用户只需提供实现简单接口的容器，即可构建任意深度的分发/聚合系统。
The last distributed systems pattern we highlight is scatter/gather. In such a system, an external client sends an initial request to a “root” or “parent” node. This root fans the request out to a large number of servers to perform computations in parallel. Each shard returns partial data, and the root gathers this data into a single response to the original request. This pattern is common in search engines. Developing such a distributed system involves a great deal of boilerplate code: fanning out the requests, gathering the responses, interacting with the client, etc. Much of this code is quite generic, and again, as in object-oriented programming, can be refactored in such a way that a single implementation can be provided that can be used with arbitrary containers so long as they implement a specific interface. In particular, to implement a scatter/gather system, a user is required to supply two containers. First, the container that implements the leaf node computation; this container performs the partial computation and returns the corresponding result. The second container is the merge container; this container takes the aggregated output of all of the leaf containers, and groups them into a single response. It is easy to see how a user can implement a scatter/gather system of arbitrary depth simply by providing containers that implement these relatively simple interfaces.
6 相关工作（Related work）
面向服务架构（SOA）早于容器化分布式系统，二者有诸多共性：均强调带清晰接口、通过网络通信的可复用组件。区别在于：SOA组件粒度更大、耦合更松散；SOA组件常实现业务逻辑，本文聚焦更接近通用库的组件，简化分布式系统构建。“微服务”是近年出现的术语，用于描述本文讨论的组件类型。
Service-oriented architectures (SOA) pre-date, and share a number of characteristics with, container-based distributed systems. For example, both emphasize reusable components with well-defined interfaces that communicate over a network. On the other hand, components in SOA systems tend to be larger-grain and more loosely-coupled than the multi-container patterns we have described. Additionally, components in SOA often implement business activities, while the components we have focused on here are more akin to generic libraries that make it easier to build distributed systems. The term “microservice” has recently emerged to describe the types of components we have discussed in this paper.
网络化组件的标准化管理接口理念至少可追溯到SNMP，SNMP主要面向硬件组件管理，微服务/容器系统尚未形成统一管理标准。这并未阻碍各类容器管理系统的发展，包括Aurora、ECS、Docker Swarm、Kubernetes、Marathon、Nomad等。
The concept of standardized management interfaces to networked components dates back at least to SNMP. SNMP focuses primarily on managing hardware components, and no standard has yet emerged for managing microservice/container-based systems. This has not prevented the development of numerous container management systems, including Aurora, ECS, Docker Swarm, Kubernetes, Marathon, and Nomad.
第5节提到的所有分布式算法都有悠久历史。GitHub上有大量主节点选举实现，但多以库而非独立组件形式存在。流行的工作队列实现包括Celery、Amazon SQS等。分发/聚合模式已被列为企业集成模式。
All of the distributed algorithms we mentioned in Section 5 have a long history. One can find a number of leader election implementations in Github, though they appear to be structured as libraries rather than standalone components. There are a number of popular work queue implementations, including Celery and Amazon SQS. Scatter-gather has been identified as an Enterprise Integration Pattern.
7 结论（Conclusion）
正如面向对象编程催生并规范了面向对象“设计模式”，容器架构正推动基于容器的分布式系统设计模式的出现与定型。本文梳理了三类新兴模式：面向系统管理的单容器模式、紧密协作的单节点多容器模式、面向分布式算法的多节点模式。在所有场景中，容器提供与面向对象系统中对象相同的价值：便于多团队分工实现、在新场景复用组件；此外还具备分布式系统专属优势：支持组件独立升级、混合语言开发、系统整体优雅降级。我们相信容器模式体系将持续丰富，未来几年会像当年面向对象编程一样革新分布式系统编程，实现分布式系统开发的标准化与规范化。
Much as object-oriented programming led to the emergence and codification of object-oriented “design patterns,” we see container architectures leading to design patterns for container-based distributed systems. In this paper we identified three types of patterns we have seen emerging: single-container patterns for system management, single-node patterns of closely-cooperating containers, and multi-node patterns for distributed algorithms. In all cases, containers provide many of the same benefits as objects in object-oriented systems, such as making it easy to divide implementation among multiple teams and to reuse components in new contexts. In addition, they provide some benefits unique to distributed systems, such as enabling components to be upgraded independently, to be written in a mixture of languages, and for the system a whole to degrade gracefully. We believe that the set of container patterns will only grow, and that in the coming years they will revolutionize distributed systems programming much as object-oriented programming did in earlier decades, in this case by enabling a standardization and regularization of distributed system development.
8 致谢（Acknowledgements）
想法并非凭空产生。本文工作深受与Brian Grant、Tim Hockin、Joe Beda、Craig McLuckie交流的启发。
Ideas don’t simply appear in our heads from a vaccum. The work in this paper has been influenced heavily by conversations with Brian Grant, Tim Hockin, Joe Beda and Craig McLuckie.