Toil and its budget

Toil is the operational work that keeps a service running and produces nothing lasting. Google defines it as manual, repetitive, automatable, tactical work that grows with the size of the service. Manual certificate renewals, restarts after every OOM kill, copy-pasting the same dashboard queries, and babysitting a flaky deploy are all toil. None of it makes the system better.

The big idea

Think of a shop owner who spends every morning re-sorting the stockroom by hand. The work is needed, but it adds nothing. Each new branch of the shop means another morning of sorting. The owner who builds a shelving system stops the work from growing with the shop.

Walking through it

Google's SRE book sets a target of keeping toil under about 50% of an SRE's time. The rest goes to engineering that removes future toil and risk. Product engineers on a "you build it, you run it" rotation need a similar explicit budget. Without one, operations will eat the roadmap slowly.

Measure it simply to start:

• Hours per week on interrupts, compared with project work.
• Pages per shift, and the share that were actionable.
• Recurring tickets of the same kind.

Then remove the largest sources, using automation, better alerts, or by deleting the feature that causes them.

Common mistakes

• Counting toil by feel. Measure hours and pages, or the budget is a wish.
• Accepting recurring tickets as the job. The same ticket twice means a fix is due.
• Automating the rare case first. Start with the work that repeats most often.

Check yourself

1. A team spends 70% of its time on interrupts. What does Google's guidance say about this? It is above the roughly 50% target. Engineering time to remove the toil is no longer protected.
2. Is a one-time manual step before a launch toil? It can be a short, time-boxed trade-off, as long as the automation is planned. Permanent manual work is toil.

Key takeaways

• Toil is manual, repetitive, automatable work that grows with the service.
• Keep it under about half of the team's time, and measure it.
• Automate the top offenders, and time-box any manual work you accept.

Further learning

The SRE book chapter on toil defines the term and gives examples of what to automate first.

Eliminating Toil (Google SRE book)Learn what toil is in SRE, how it affects operational work, and why minimizing it is crucial for efficiency and morale.sre.google