📝 Workbook · Business · Unit 8

Workbook · 8A — When systems fail

← Back to Class Book
Exercise 1

Exercise 1 — Incident vocabulary

Complete each sentence with ONE phrase from the box.

Word bank — drag or tap, then tap a gap
  1. The payments API was rendered wholly unavailable for 47 minutes — a total  drop .
  2. The  drop  was a missing null-check in the auth service — but that was only the trigger.
  3. The  drop  was negligible: only Portuguese mobile users were affected.
  4. The alert fired at 02:14 and the  drop  acknowledged it in 4 minutes.
  5. We're rolling out  drop  across all admin accounts — no more password-only logins for anything sensitive.
  6. The  drop  was updated on Monday to require two reviewers on any auth-adjacent change.
Important notes

Ask the teacher: Write down anything you want to remember from this section. Your notes are saved automatically.

Auto-saving
👩‍🏫 Ask the teacher · visible only to teachers
Exercise 2

Exercise 2 — The passive across tenses

Rewrite each sentence in the passive so the SYSTEM is the subject.

  1. 'Someone pushed a bad configuration at 14:02.'
  2. 'We are testing the failover in staging right now.'
  3. 'An earlier change had removed the health-check.'
  4. 'The team will identify the root cause by Friday.'
  5. 'We should test the failover every week.'
  6. 'We could have prevented this with a proper code review.'
Important notes

Ask the teacher: Write down anything you want to remember from this section. Your notes are saved automatically.

Auto-saving
👩‍🏫 Ask the teacher · visible only to teachers
Exercise 3

Exercise 3 — Have something done

Rewrite each sentence using 'have something done' to signal ownership without claiming the keyboard work.

  1. 'We're going to reconfigure the alert threshold by Wednesday.' (…but ops actually does it)
  2. 'Security reviewed our pipeline last quarter.'
  3. 'We need to update the runbook before closing this incident.'
  4. 'Legal will draft the customer notification tonight.'
Important notes

Ask the teacher: Write down anything you want to remember from this section. Your notes are saved automatically.

Auto-saving
👩‍🏫 Ask the teacher · visible only to teachers
Exercise 4

Exercise 4 — Blameless or blame-loaded?

Decide whether each incident line is BLAMELESS (describes the system) or BLAME-LOADED (names a villain).

  1. 'A configuration change was pushed at 14:02 that removed the health-check.'

  2. 'Ravi broke the auth service again.'

  3. 'The failover was never tested in production — we caught that in the post-mortem.'

  4. 'This wouldn't have happened if ops actually cared.'

  5. 'The second-reviewer requirement had been waived for launch week.'

  6. 'Whoever wrote this alert should be fired.'

Important notes

Ask the teacher: Write down anything you want to remember from this section. Your notes are saved automatically.

Auto-saving
👩‍🏫 Ask the teacher · visible only to teachers
Exercise 5

Exercise 5 — Rewrite blame as blameless

Rewrite each blame-loaded line as a blameless incident sentence using the passive and, where useful, a role instead of a name.

  1. 'Ravi pushed a bad config and broke prod again.'
  2. 'Ops never tested the failover and now we're paying for it.'
  3. 'Whoever wrote this alert should be fired — it doesn't fire when the DB is down.'
Important notes

Ask the teacher: Write down anything you want to remember from this section. Your notes are saved automatically.

Auto-saving
👩‍🏫 Ask the teacher · visible only to teachers
Exercise 6

Exercise 6 — Three post-mortems for the same 47-min outage

You'll hear three engineering leads walk through the same outage. Decide who should own the incident write-up going forward.

🎙️ Three post-mortems — same 47-minute outage — multi-voice

  1. Which lead's post-mortem is adopted, and why?
  2. Which lead's post-mortem is 'find the guilty person' dressed as engineering?
  3. Which lead is technically correct and produces zero change?
Show transcript

Bianca (Diego):This is really on Ravi. He pushed the config at 14:02 without waiting for a second reviewer. Everyone was under launch pressure, sure, but you don't push to prod on launch day without a review. Action items: written warning for Ravi, more training for the whole team.

Will (Ines):Timeline. At 14:02 a configuration change was pushed that removed the /healthz endpoint. At 14:06 the on-call engineer was paged. At 14:14 the change was rolled back. Service was fully restored at 14:49. Why. WHY 1: the change had been approved with only one reviewer. WHY 2: the second-reviewer requirement had been waived for launch week. WHY 3: the runbook did not classify auth-adjacent changes as always requiring two reviewers. Three action items. One, we'll have the runbook updated — SRE lead, by Wednesday. Two, we'll have the /healthz alerting reconfigured to page within 60 seconds — on-call lead, by Friday. Three, we'll have a monthly failover drill scheduled — Head of Platform, first drill by end of month.

Vera (Kwame):Timeline is what Ines said. We should be more careful with deploys, honestly. We should probably test the failover more often. And, um, we should look at the runbook at some point too. Yeah.

Important notes

Ask the teacher: Write down anything you want to remember from this section. Your notes are saved automatically.

Auto-saving
👩‍🏫 Ask the teacher · visible only to teachers
Exercise 7

Exercise 7 — Write your 4-sentence post-mortem

Pick a real incident from the last 6 months. Write the 4-sentence summary (max 80 words). Sentence 1: timeline in the passive. Sentence 2: root cause, at least three whys deep. Sentence 3: ONE 'have something done' action item with owner and date. Sentence 4: what changes in the RUNBOOK, not what a person will 'try to do better'.

Important notes

Ask the teacher: Write down anything you want to remember from this section. Your notes are saved automatically.

Auto-saving
👩‍🏫 Ask the teacher · visible only to teachers