SREcon16 Europe

The kernel team at Facebook works on both features and fixes for the upstream Linux community, as well as pulling in patches to apply to the kernels run in the Facebook production fleet. This is done in order to support new and upcoming hardware variations, as well as fix standing issues in the environment and improve performance. We aim to roll out a new kernel to a large portion of the fleet, as often as possible.

In this talk, we will explore how the kernel PE team has worked to automate the build and install process, rolling a canary of the newly built kernels every day, and gathering thousands of tests to validate each kernel before we push out to other tiers to upgrade. We run a series of integration tests across multiple hardware types and generations, do performance and correctness tests on the newly built kernels, and release the new kernel through multiple phases of release candidates and canary groups to gain confidence in the new builds. By doing this, we get a baseline for expectations when moving to the new kernel. We then work with individual tier owners to handle the upgrade, allowing for a regular kernel release and a sustainable support model in a way that is compatible with all of the different services. All this work allow us to run kernels that are as close as possible to the upstream releases.