---
name: debugging-detective
description: Find the real root cause of bugs systematically instead of guessing. Use when something is broken, flaky, slow, behaves differently between environments, or a previous fix did not work. Covers reproduction, hypothesis testing, bisecting, logging, race conditions, and writing a regression test before the fix.
---

# Debugging Detective

You debug like a senior engineer: evidence first, one hypothesis at a time, and no fix without a confirmed cause. Guess-and-patch is banned.

## The loop

1. **State the symptom precisely.** Expected vs actual, exact error text, where it happens, since when, how often. Rewrite vague reports ("it's broken") into a testable statement.
2. **Reproduce it.** Find the smallest, fastest, most reliable reproduction. A bug you cannot reproduce is a bug you cannot prove fixed. If it is intermittent, measure the failure rate (for example 3 in 20 runs) so you can tell later whether a change helped.
3. **Gather evidence before theorising.** Read the full stack trace (the first frame in *our* code matters most), logs around the timestamp, network requests, recent commits, environment and dependency versions.
4. **List hypotheses and rank them** by likelihood × cheapness to test. Write them down.
5. **Test one hypothesis at a time** with an experiment whose result would distinguish it from the others. Change one variable per experiment.
6. **Confirm the root cause** by explaining the whole chain: trigger → mechanism → symptom. If any link is "probably", you are not done.
7. **Write a failing test** that captures the bug, then fix, then watch it pass.
8. **Look for siblings.** The same mistake often exists elsewhere (same pattern, same author, copy-pasted code). Search for it.

## Narrowing techniques

- **Git bisect** when it used to work: `git bisect start`, `git bisect bad`, `git bisect good <sha>`, then `git bisect run <test command>` to automate.
- **Binary search the code path**: comment out or short-circuit half, see which half the bug lives in, repeat.
- **Diff the environments** when it "works on my machine": runtime version, env vars, feature flags, data, locale, time zone, OS, CPU architecture, build mode (dev vs prod), caches, CDN.
- **Minimise the input**: shrink the failing payload or dataset until removing anything else makes the bug disappear.
- **Read the source of the dependency** rather than assuming its behaviour. Check its changelog and open issues for your version.

## Instrumentation

- Prefer a debugger with breakpoints and conditional breakpoints over scattered `console.log`.
- When logging, log the *values* that test your hypothesis, with a unique prefix you can grep and remove: `[dbg-cart] subtotal=%o items=%o`.
- Add timestamps and request or correlation IDs when tracking async flows.
- Remove all temporary instrumentation before finishing.

## Classic root causes to check

- **Async and ordering**: missing `await`, unhandled promise rejections, stale closures, event handlers registered twice, effects without cleanup, race between two requests where the slower one wins.
- **State and caching**: stale cache (HTTP, CDN, build, ORM, memoised selectors), mutation of shared objects, singletons leaking state between requests or tests.
- **Data**: `null` vs `undefined` vs empty string, off-by-one, floating point money (use integer minor units), integer overflow, Unicode normalisation, case sensitivity, trailing whitespace.
- **Time**: time zones, DST transitions, server vs client clocks, month indexing from 0, dates parsed without a zone.
- **Configuration**: wrong env var, missing in one environment, read at build time instead of runtime.
- **Concurrency**: two workers processing the same job, non-atomic read-modify-write, missing unique constraints, lack of idempotency.
- **Flaky tests**: shared state between tests, reliance on order, real timers or network, unawaited assertions, fixed sleeps instead of waiting for a condition.

## Performance bugs

Measure, do not guess. Profile (browser Performance panel, `node --cpu-prof`, `EXPLAIN ANALYZE`), find the hottest path, fix that, measure again. Report before and after numbers.

## Rules

- Never "fix" by adding a retry, a `setTimeout`, a broader `try/catch`, or `!important` without understanding why it was needed. These hide bugs.
- Never disable a failing test to make CI green.
- If you are stuck after three failed hypotheses, step back: re-read the original symptom and question an assumption you have treated as fact.

## Output

Report: **Symptom**, **Reproduction steps**, **Root cause** (the full causal chain, with file and line), **Fix**, **Regression test added**, **Other places with the same pattern**, and anything still uncertain.

---

**Go deeper:** [Observability & Incident Response](https://www.ben-toms.com/skill-packs/observability-incident-response) is a premium skill pack with linked skills, production templates, scripts and eval prompts.