In short: Capability numbers that look better than reality mean decisions made on false confidence. A one-character default in a common Python library is enough to cause it.
If you’ve ever built process capability reporting in Python, there’s a good chance you’ve shipped this bug without knowing it. It doesn’t throw an error. It doesn’t crash. It just quietly makes your process look more capable than it actually is.
What Cp/Cpk and Pp/Ppk are for
In statistical process control, capability indices answer one question: given the natural variation in a process, how much room is there between that variation and the specification limits?
- Cp/Cpk use within-subgroup variation. They describe potential capability, assuming the process is stable and in control.
- Pp/Ppk use overall variation across the entire dataset. They describe actual, observed performance, including any drift or shifts between subgroups.
Both depend entirely on getting the standard deviation right. That’s exactly where the bug hides.
The subtle difference: population vs. sample standard deviation
Population standard deviation divides by n. Sample standard deviation divides by n − 1 (Bessel’s correction), because a sample systematically underestimates the true variance of the population it was drawn from. Any time you’re computing a capability index from a sample of process readings, which is always, since you never have the entire population of a process’s output, you want sample standard deviation.
NumPy’s np.std() defaults to ddof=0, which is population variance. Getting sample variance requires explicitly passing ddof=1. That one-character omission is the entire bug, and it’s an easy one to introduce because the function runs fine either way. There’s no error, no warning, just a slightly wrong number.
How the error compounds
Population standard deviation is always smaller than (or equal to) sample standard deviation for the same data. Since capability indices divide the spec width by a multiple of sigma, a smaller sigma produces a larger, falsely better-looking, capability index.
A worked example
Here’s the bug in action, using a realistic spec window: an upper spec limit of 441.664, a lower spec limit of 415.936, and a process target of 428.800.
Below is a 30-point sample of process readings around that target (illustrative data shaped to this spec window, not a live production pull):
| # | Value | # | Value | # | Value |
|---|---|---|---|---|---|
| 1 | 432.24 | 11 | 427.34 | 21 | 432.18 |
| 2 | 427.87 | 12 | 428.27 | 22 | 429.07 |
| 3 | 428.67 | 13 | 429.77 | 23 | 427.95 |
| 4 | 429.54 | 14 | 428.05 | 24 | 433.06 |
| 5 | 426.72 | 15 | 427.83 | 25 | 428.69 |
| 6 | 428.28 | 16 | 425.02 | 26 | 425.69 |
| 7 | 428.09 | 17 | 429.29 | 27 | 427.67 |
| 8 | 424.63 | 18 | 428.73 | 28 | 423.79 |
| 9 | 430.71 | 19 | 429.08 | 29 | 430.79 |
| 10 | 429.79 | 20 | 425.22 | 30 | 427.89 |
Mean: 428.397
Population std (ddof=0, the bug): 2.1527
Sample std (ddof=1, the correct version): 2.1895
That’s a small-looking difference, about 1.7%, but it flows straight through to Ppk:
| Using population std (bug) | Using sample std (correct) | |
|---|---|---|
| Ppu | 2.0543 | 2.0198 |
| Ppl | 1.9296 | 1.8972 |
| Ppk | 1.9296 | 1.8972 |
A 1.7% understatement of sigma turns into a matching 1.7% overstatement of Ppk. On this sample the process is comfortably capable either way, so the conclusion doesn’t flip. But the gap matters more the closer a process sits to a capability threshold like 1.33, and it grows with smaller sample sizes, where the n vs. n − 1 difference is larger. A marginal process sitting right at a pass/fail line is exactly the case where this bug is most likely to change the answer.
The fix
The fix is less about the math (a one-character change: ddof=1) and more about the engineering discipline around it:
def _sample_sigma(values):
"""Single source of truth for sample standard deviation across all
capability and drift calculations."""
return float(np.std(values, ddof=1))Every place that previously computed its own standard deviation, whether Ppk, sigma drift or anything else, calls this one helper instead. That eliminates the chance of three different definitions of “sigma” quietly disagreeing with each other inside the same system.
Making it visible, not just correct
A capability index is a single number, and a single number invites exactly the kind of blind trust worth guarding against. The more honest presentation is a histogram of recent samples with the spec limits (USL/LSL) drawn directly on it, and the capability value labeled next to the distribution it was computed from, not floating alone on a tile. Seeing the spread of the data against the limits makes it obvious when a “good” Ppk is riding on a small sample or an off-center distribution, in a way the number alone never communicates.
The broader lesson
This is a useful lesson beyond SPC: library defaults are not the same as domain-correct defaults. ddof=0 is a perfectly reasonable default for a general-purpose numerical library. It is not a reasonable default for a capability index with a formal statistical definition. Any time a domain-specific formula touches a general-purpose function, it’s worth explicitly checking whether the library’s default assumption matches the formula’s assumption, rather than trusting that “it ran without an error” means “it’s correct.”
If you’re building or auditing SPC software, the fifteen minutes it takes to search for every std() call and check its ddof is some of the highest-leverage code review time you can spend.
This is Part 1 of a 7-part series on building a real-time process monitoring system. Part 2 covers a related mistake in EWMA/CUSUM control limits.