Acceptance Sampling: What a Clean Sample of Fifty Actually Proves

A supplier delivers ten thousand parts. Quality control inspects fifty, finds nothing wrong, and the delivery is accepted. The same routine appears in data work: a new feed arrives, an analyst checks a hundred rows, everything looks right, the feed goes live.

Neither check establishes what people take from it. A batch running at two percent defective passes an inspection of fifty with nothing found 36 percent of the time. The inspection did not show the batch was clean. It showed the batch was not disastrous, which is a much weaker claim, and the arithmetic that says how much weaker has been settled since the 1920s.

What a Clean Sample Rules Out

A sample of <div class="math-expression math-expression--source" data-math-alt="n" data-math-source="n" aria-label="n" tabindex="0">\(n\)</div> with no defects is consistent with any rate low enough that <div class="math-expression math-expression--source" data-math-alt="n" data-math-source="n" aria-label="n" tabindex="0">\(n\)</div> clean draws are unremarkable. The highest rate that would still produce a clean sample five percent of the time is the useful bound, and it falls only as fast as <div class="math-expression math-expression--source" data-math-alt="1/n" data-math-source="1/n" aria-label="1/n" tabindex="0">\(1/n\)</div>.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import numpy as np
from scipy import stats

RNG = np.random.default_rng(41)


def accept_prob(n, c, p):
    """Chance a lot with defect rate p passes a plan that inspects n and
    allows at most c defects."""
    return stats.binom.cdf(c, n, p)


for n in (20, 50, 100, 300, 1000):
    p95 = 1 - 0.05 ** (1 / n)          # highest rate that still passes 5% of the time
    print(f"inspect {n:5d}, find none: rates above {p95:6.2%} are rejected 95% of the time, "
          f"and a lot at {p95 / 2:6.2%} passes {accept_prob(n, 0, p95 / 2):5.1%} of the time")

Fifty clean items rule out rates above about six percent and say nothing about anything below three. That is the honest reading of the routine inspection, and it is the same rule of three that governs any zero-count observation: the upper bound is roughly three divided by the sample size.

The Operating Characteristic

A plan is a pair: inspect <div class="math-expression math-expression--source" data-math-alt="n" data-math-source="n" aria-label="n" tabindex="0">\(n\)</div>, accept if at most <div class="math-expression math-expression--source" data-math-alt="c" data-math-source="c" aria-label="c" tabindex="0">\(c\)</div> are defective. Its operating characteristic is the chance of acceptance at each true rate, and it is what makes two plans comparable.

1
2
3
4
5
6
7
plans = [(50, 0), (100, 1), (200, 3), (500, 8)]
rates = (0.001, 0.005, 0.01, 0.02, 0.05)
header = "  plan      " + "".join(f"{r:>10.1%}" for r in rates)
print(header)
for n, c in plans:
    row = "".join(f"{accept_prob(n, c, r):>10.1%}" for r in rates)
    print(f"  n={n:<4d} c={c:<2d}{row}")

Read the columns rather than the rows. Every plan here accepts a one-in-a-thousand lot and rejects a five-percent lot, and they differ in the middle, which is where real suppliers live. The first plan rejects four good lots in ten at a one percent rate; the last accepts nine in ten. Larger samples buy discrimination, not severity: the bottom row is both kinder to good lots and harsher on bad ones.

Choosing a Plan From Two Risks

The design question is not "how many should we check" but "which two rates matter". An acceptable quality level is the rate a good supplier should almost always pass; a tolerance level is the rate that must almost always be caught. Fix those two and the plan follows.

1
2
3
4
5
6
7
8
9
10
11
12
AQL, LTPD = 0.01, 0.05      # acceptable rate, and the rate that must be caught
print(f"accept 95% of lots at {AQL:.0%}, reject 90% of lots at {LTPD:.0%}")
best = None
for c in range(0, 12):
    n = c + 1
    while n < 5000:
        if accept_prob(n, c, AQL) >= 0.95 and accept_prob(n, c, LTPD) <= 0.10:
            print(f"  c = {c:2d}: n = {n:4d}   passes at AQL {accept_prob(n, c, AQL):5.1%}, "
                  f"passes at LTPD {accept_prob(n, c, LTPD):5.1%}")
            best = best or (n, c)
            break
        n += 1

Accept numbers of zero, one and two appear nowhere in the table, because no sample size satisfies both constraints with them. That is the central and least intuitive result here: a stricter rule does not make a better plan. The smallest plan meeting both risks inspects 132 and tolerates three defects.

The Price of Zero Tolerance

Zero-defect rules are popular because they sound rigorous. What they actually do is shift the entire burden onto the supplier's good batches.

1
2
3
4
for c, n in ((0, 45), (1, 77), (2, 105), (3, 132)):
    print(f"c = {c}, n = {n:3d}: passes at 1% {accept_prob(n, c, 0.01):5.1%}, "
          f"passes at 5% {accept_prob(n, c, 0.05):5.1%}, "
          f"producer's risk at 1% {1 - accept_prob(n, c, 0.01):5.1%}")

All four plans are equally severe on a five-percent lot, catching ninety percent of them. They differ entirely in what they do to a supplier running at one percent, which the contract calls acceptable: the zero-tolerance plan rejects more than a third of those batches, the tolerant plan one in twenty-three. The cost of those false rejections lands as returned shipments, expedited replacements and an adversarial relationship with a supplier who is meeting the agreement.

Two Stages Instead of One

Most lots are clearly fine or clearly bad, and only the ambiguous ones need a full inspection. A two-stage plan exploits that: inspect a first sample, accept immediately if it is clean enough, reject immediately if it is bad enough, and inspect more only in between.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
def double_plan(p, n1, c1, r1, n2, c2, rng, reps=20000):
    """Inspect n1. Accept at c1 or fewer, reject at r1 or more, otherwise
    inspect n2 more and judge on the total."""
    first = rng.binomial(n1, p, reps)
    decided = (first <= c1) | (first >= r1)
    second = rng.binomial(n2, p, reps)
    total = first + second
    accept = np.where(decided, first <= c1, total <= c2)
    inspected = np.where(decided, n1, n1 + n2)
    return accept.mean(), inspected.mean()


single_n, single_c = 132, 3
for p in (0.005, 0.01, 0.02, 0.05):
    acc_d, avg_n = double_plan(p, 80, 1, 4, 80, 4, np.random.default_rng(3))
    print(f"true rate {p:5.1%}: single plan n={single_n} accepts "
          f"{accept_prob(single_n, single_c, p):5.1%} inspecting {single_n} every time; "
          f"double plan accepts {acc_d:5.1%} inspecting {avg_n:5.0f} on average")

At the rate a good supplier runs, the two-stage plan inspects 95 items instead of 132, a 28 percent saving, while accepting slightly more often. The saving is largest exactly where most lots sit. The cost is a little less discrimination at the bad end, 13.8 percent acceptance instead of 9.9, which can be tuned back by tightening the second-stage accept number.

When the Lot Is Small

The binomial calculation assumes each inspected item is drawn from an unchanged population, which is fine for a lot of ten thousand and wrong for a lot of two hundred.

1
2
3
4
5
6
for lot in (200, 500, 2000, 100000):
    n, c, defects = 50, 0, int(round(0.02 * lot))
    hyper = stats.hypergeom.cdf(c, lot, defects, n)
    binom = accept_prob(n, c, 0.02)
    print(f"lot of {lot:6d} with 2% defective: exact {hyper:6.2%}, "
          f"binomial approximation {binom:6.2%}, difference {hyper - binom:+.2%}")

The approximation errs in the safe direction, overstating the chance a bad lot passes, and the error is worth carrying only when the sample is a substantial fraction of the lot. A practical rule: use the exact calculation when the sample exceeds about a tenth of the lot.

The Same Arithmetic for a Data Audit

None of this is specific to physical parts. Sampling rows from a table to check a transformation is the same problem with different vocabulary, and the sample sizes people use are just as habitual.

1
2
3
4
5
6
7
8
for rows, sample in ((2_000_000, 200), (2_000_000, 1000), (2_000_000, 5000)):
    for true_rate in (0.001, 0.005):
        r = np.random.default_rng(5)
        found = r.binomial(sample, true_rate, 20000)
        print(f"{rows:,} rows, sample {sample:5d}, true error rate {true_rate:.1%}: "
              f"finds nothing {np.mean(found == 0):5.1%} of the time, "
              f"expected errors in the sample {sample * true_rate:5.1f}, "
              f"implied errors in the table {rows * true_rate:8,.0f}")

Two hundred rows checked by hand is the most common data audit there is, and against a table carrying two thousand bad rows it comes back clean four times in five. The audit is not weak because the checker is careless; it is weak because 200 is the wrong number. Moving to a thousand rows changes the answer from "we found nothing" to "we expect to see about one, and we did".

The last column is the one to put in the report. A one-in-a-thousand error rate sounds tolerable and means two thousand broken rows, which may or may not be acceptable depending on what reads them.

What to Do

1. State the two rates before choosing a sample size: the rate a good supplier or pipeline should pass, and the rate that must be caught. The plan follows from them.
2. Stop using zero-defect acceptance as a default. It is the harshest plan on batches that meet the agreement and no harsher on the ones that do not.
3. Publish the operating characteristic alongside the plan, so both sides can see what each rate implies rather than arguing about a single number.
4. Use two-stage sampling when inspection is expensive. It typically cuts the average inspected by a quarter at the rates suppliers actually run.
5. Switch to the exact finite-lot calculation when the sample exceeds about a tenth of the lot.
6. Size data audits the same way, and report the implied number of affected rows rather than the rate. Two hundred rows finds nothing in a table with thousands of errors.

References

• Dodge, H. F., & Romig, H. G. (1959). Sampling Inspection Tables: Single and Double Sampling (2nd ed.). Wiley.
• Schilling, E. G., & Neubauer, D. V. (2017). Acceptance Sampling in Quality Control (3rd ed.). CRC Press.
• Montgomery, D. C. (2019). Introduction to Statistical Quality Control (8th ed.). Wiley.
• Wald, A. (1947). Sequential Analysis. Wiley.
• Hanley, J. A., & Lippman-Hand, A. (1983). If nothing goes wrong, is everything all right? Interpreting zero numerators. JAMA, 249(13), 1743-1745.
• American Society for Quality (2018). ANSI/ASQ Z1.4: Sampling Procedures and Tables for Inspection by Attributes. ASQ.