SREcon16 Europe

In Dropbox automation, to bridge the gap between “scripts” and “fully automatic automation”, we’ve introduced a concept of “Human Authorized Execution”. This means that a tool automatically finds problems and decides how to fix them, but a human operator is required to audit the tool’s decisions before the automation may run.

Why do we need this? Frankly, it’s terrifying to have automation run fully automatically. With a human involved, their subconscious can answer a really important question: Why might I NOT want to run this script? If we took a simple approach, for instance deploying a cron job to run our scripts whenever alerts fire, then we would lose that human’s sense of paranoia and danger.

At Dropbox, we’ve built an alert auto-remediation platform called Naoru, which forces us to build our automation in a way that adheres to these principles. In this talk we will discuss the thought process we bring towards building trustworthy automation, how Naoru forces our engineers to follow these philosophies, and how we’ve driven our infrastructure organization towards a culture of embracing trustworthy automation.