SREcon16 Europe

One of the key principles of SRE is always knowing your service is broken before your customers notice. Network devices are typically black boxes that fail in mysterious ways. The tradiational methods of network monitoring don't scale well. They also have a key fundamental flaw. Most network monitoring involves asking a network device if it's dropping packets. If that device is fundamentally unhealthy, should you really trust what it tells you about what it is doing?

Facebook has taken a different approach to network monitoring where we actively probe our datacenter networks from as many locations as possible to ensure the network is behaving as we expect. The interesting past comes when we detect loss between different hosts on the network. How do we discover which network device or individual link in the thousands that are available is the root cause of this loss? By combining some magic requests with some basic math, can we automate that detection and triangulate it to a specific device that does not even know it's dropping packets?