Robustness

A QCA solution is conditional on decisions the data do not make for you: where the calibration anchors sit, how consistent a row must be to count as sufficient, how many cases a row needs. Reporting one solution from one set of those choices hides how much of the result was the choice rather than the evidence.

from setqca import RobustnessGrid, robustness_analysis

analysis = robustness_analysis(
    data,
    outcome="SURV",
    conditions=["DEV", "URB", "LIT"],
    grid=RobustnessGrid(
        consistency=[0.75, 0.80, 0.85, 0.90],
        pri=[0.50, 0.60, 0.70],
        frequency=[1, 2],
    ),
)
print(analysis)
print(analysis.to_frame())

What comes back

Robustness of the conservative solution
Specifications: 24 (21 produced a solution, 3 did not)
Baseline: cons=0.85, pri=0.6, n=1

Stable terms (1):
  DEV*LIT*STB — 21/21 specifications
Threshold-sensitive terms (2):
  DEV*URB — 6/21 specifications
  URB*STB — 4/21 specifications
Baseline terms that do not survive: URB*STB

Stability is not validity: a mis-specified model can be perfectly stable.

Four buckets, each answering a different question:

Accessor	Question
stable_terms()	Which paths survive nearly every cutoff?
fragile_terms()	Which appear only under some?
disappearing_terms()	Which baseline paths do not survive?
emerging_terms()	Which stable paths does the baseline miss?

The threshold defaults to 0.8 and is adjustable on each call.

Failures are recorded, not dropped

A specification that produces no solution gets a row with a failure message and NaN fit, rather than vanishing. "The model collapses above 0.9" is a finding about your data, and silently omitting those rows would make the surviving ones look more robust than they are.

Sweeping calibration anchors

Calibration is where substantive judgement enters, so it is also where a result is most easily manufactured. calibration_robustness recalibrates from the raw measures for each anchor combination:

from setqca.analysis.robustness import calibration_robustness

analysis = calibration_robustness(
    raw_data,
    outcome="SURV",
    conditions=["DEV", "URB"],
    grid=RobustnessGrid(
        consistency=[0.80],
        anchors={"DEV": [(10, 50, 90), (20, 50, 80), (10, 40, 90)]},
    ),
    outcome_anchors=(10, 50, 90),
    base_anchors={"URB": (10, 50, 90)},
)

The input is raw, so every condition needs anchors from one source or the other — swept in the grid, or fixed through base_anchors. Passing calibrated data to this function, or a grid with anchors to robustness_analysis, is an error rather than a silent mis-read.

Comparing solutions

Textual identity is the strictest comparison and often the least informative. Four scales are available:

from setqca.analysis.robustness import solution_similarity

similarity = solution_similarity(left_terms, right_terms, data)
similarity.identical  # exact set equality
similarity.term_overlap  # Jaccard over terms
similarity.configurational  # Jaccard over the literals used
similarity.membership  # fuzzy Jaccard over case membership

The last is the one that catches agreement the text hides: two solutions can be written differently and still select the same cases. A and A+A*B are textually distinct and have membership similarity 1.0, because the second term adds nothing.

analysis.similarity_to_baseline()

Robustness is not validity

A path that appears under every threshold is stable, not true.

Stability says the finding does not depend on one arbitrary cutoff. It says nothing about whether the conditions are causally relevant, whether the calibration was substantively sensible, whether the cases were well chosen, or whether an omitted condition is doing the work. A thoroughly mis-specified model can be perfectly stable — sweeping thresholds cannot detect a problem that lives in the model rather than the cutoffs.

Nothing in this module reports a verdict. The measures are descriptive; the interpretation is yours.

setqca.analysis.robustness

Sensitivity of a QCA result to the choices that produced it.

A QCA solution is conditional on decisions the data do not make for you: where the calibration anchors sit, how consistent a row must be to count as sufficient, how many cases a row needs. Reporting one solution from one set of those choices hides how much of the result was the choice rather than the evidence.

This module runs the analysis across a grid of those choices and reports which paths survive.

What robustness is not

A path that appears under every threshold is stable, not true. Stability says the finding does not depend on one arbitrary cutoff. It says nothing about whether the conditions are causally relevant, whether the calibration was substantively sensible, or whether the case selection was sound. A thoroughly mis-specified model can be perfectly stable.

Nothing here reports a verdict. The measures are descriptive, and the interpretation is the researcher's.

Specification dataclass

Specification(
    consistency: float,
    pri: float,
    frequency: int,
    anchors: tuple[
        tuple[str, tuple[float, float, float]], ...
    ] = (),
)

One combination of analytical choices.

RobustnessGrid dataclass

RobustnessGrid(
    consistency: Sequence[float] = (0.75, 0.8, 0.85),
    pri: Sequence[float] = (0.0,),
    frequency: Sequence[int] = (1,),
    anchors: Mapping[
        str, Sequence[tuple[float, float, float]]
    ] = dict(),
)

The analytical choices to sweep.

Parameters:

Name	Type	Description	Default
consistency	sequence of float	Inclusion and PRI cutoffs to try.	(0.75, 0.8, 0.85)
pri	sequence of float	Inclusion and PRI cutoffs to try.	(0.75, 0.8, 0.85)
frequency	sequence of int	Frequency cutoffs to try.	(1,)
anchors	mapping of str to sequence of (float, float, float)	Alternative calibration anchors per condition, as (full_out, crossover, full_in). Only usable with raw data through :func:calibration_robustness.	dict()

specifications

specifications() -> Iterator[Specification]

Yield every combination in the grid, in a deterministic order.

Source code in src/setqca/analysis/robustness.py

def specifications(self) -> Iterator[Specification]:
    """Yield every combination in the grid, in a deterministic order."""
    names = sorted(self.anchors)
    anchor_options: list[list[tuple[str, tuple[float, float, float]]]] = [
        [(name, tuple(anchor)) for anchor in self.anchors[name]]  # type: ignore[misc]
        for name in names
    ]
    combinations: Sequence[tuple[tuple[str, tuple[float, float, float]], ...]] = (
        list(product(*anchor_options)) if anchor_options else [()]
    )
    for consistency, pri, frequency, anchors in product(
        self.consistency, self.pri, self.frequency, combinations
    ):
        yield Specification(
            consistency=float(consistency),
            pri=float(pri),
            frequency=int(frequency),
            anchors=tuple(anchors),
        )

RobustnessRun dataclass

RobustnessRun(
    specification: Specification,
    terms: frozenset[str],
    consistency: float,
    coverage: float,
    implicants: int,
    literals: int,
    solutions: int,
    failure: str | None = None,
)

The outcome of one specification.

A specification that produces no solution is recorded rather than dropped: "the model collapses above 0.85" is itself a finding.

succeeded property

succeeded: bool

Return whether a solution was produced.

TermStability dataclass

TermStability(
    term: str,
    appearances: int,
    total: int,
    in_baseline: bool,
)

How often one term survived the sweep.

share property

share: float

Return the proportion of successful runs containing the term.

stable

stable(threshold: float = DEFAULT_STABILITY) -> bool

Return whether the term appears in at least threshold of runs.

Source code in src/setqca/analysis/robustness.py

def stable(self, threshold: float = DEFAULT_STABILITY) -> bool:
    """Return whether the term appears in at least ``threshold`` of runs."""
    return self.share >= threshold

SolutionSimilarity dataclass

SolutionSimilarity(
    identical: bool,
    term_overlap: float,
    configurational: float,
    membership: float,
)

Several ways two solutions can resemble each other.

Attributes:

Name	Type	Description
identical	bool	The two term sets are equal.
term_overlap	float	Jaccard index over term sets: exact string agreement.
configurational	float	Jaccard index over the literals used, so solutions that differ in how terms are cut but use the same conditions still score highly.
membership	float	Fuzzy Jaccard over case membership in the solution, Σ min(a, b) / Σ max(a, b). Two solutions can differ textually and still select the same cases; this is what notices that.

RobustnessAnalysis dataclass

RobustnessAnalysis(
    grid: RobustnessGrid,
    runs: tuple[RobustnessRun, ...],
    baseline: Specification,
    family: str,
    data: DataFrame,
)

The result of sweeping a grid of analytical choices.

successful property

successful: tuple[RobustnessRun, ...]

Return runs that produced a solution.

failed property

failed: tuple[RobustnessRun, ...]

Return specifications under which the model produced nothing.

baseline_terms property

baseline_terms: frozenset[str]

Return the terms of the baseline specification, if it succeeded.

term_stability

term_stability() -> tuple[TermStability, ...]

Return every term seen, with how often it survived.

Source code in src/setqca/analysis/robustness.py

def term_stability(self) -> tuple[TermStability, ...]:
    """Return every term seen, with how often it survived."""
    successful = self.successful
    total = len(successful)
    seen: dict[str, int] = {}
    for run in successful:
        for term in run.terms:
            seen[term] = seen.get(term, 0) + 1
    baseline = self.baseline_terms
    return tuple(
        sorted(
            (
                TermStability(
                    term=term,
                    appearances=count,
                    total=total,
                    in_baseline=term in baseline,
                )
                for term, count in seen.items()
            ),
            key=lambda item: (-item.appearances, item.term),
        )
    )

stable_terms

stable_terms(
    threshold: float = DEFAULT_STABILITY,
) -> tuple[str, ...]

Return terms appearing in at least threshold of successful runs.

Source code in src/setqca/analysis/robustness.py

def stable_terms(self, threshold: float = DEFAULT_STABILITY) -> tuple[str, ...]:
    """Return terms appearing in at least ``threshold`` of successful runs."""
    return tuple(item.term for item in self.term_stability() if item.stable(threshold))

fragile_terms

fragile_terms(
    threshold: float = DEFAULT_STABILITY,
) -> tuple[str, ...]

Return terms that appear, but in fewer than threshold of runs.

Source code in src/setqca/analysis/robustness.py

def fragile_terms(self, threshold: float = DEFAULT_STABILITY) -> tuple[str, ...]:
    """Return terms that appear, but in fewer than ``threshold`` of runs."""
    return tuple(item.term for item in self.term_stability() if not item.stable(threshold))

disappearing_terms

disappearing_terms(
    threshold: float = DEFAULT_STABILITY,
) -> tuple[str, ...]

Return baseline terms that do not survive the sweep.

Source code in src/setqca/analysis/robustness.py

def disappearing_terms(self, threshold: float = DEFAULT_STABILITY) -> tuple[str, ...]:
    """Return baseline terms that do not survive the sweep."""
    return tuple(
        item.term
        for item in self.term_stability()
        if item.in_baseline and not item.stable(threshold)
    )

emerging_terms

emerging_terms(
    threshold: float = DEFAULT_STABILITY,
) -> tuple[str, ...]

Return stable terms the baseline did not report.

Source code in src/setqca/analysis/robustness.py

def emerging_terms(self, threshold: float = DEFAULT_STABILITY) -> tuple[str, ...]:
    """Return stable terms the baseline did not report."""
    return tuple(
        item.term
        for item in self.term_stability()
        if not item.in_baseline and item.stable(threshold)
    )

similarity_to_baseline

similarity_to_baseline() -> tuple[
    tuple[Specification, SolutionSimilarity], ...
]

Compare every successful run against the baseline solution.

Source code in src/setqca/analysis/robustness.py

def similarity_to_baseline(self) -> tuple[tuple[Specification, SolutionSimilarity], ...]:
    """Compare every successful run against the baseline solution."""
    baseline = self.baseline_terms
    return tuple(
        (run.specification, solution_similarity(baseline, run.terms, self.data))
        for run in self.successful
    )

to_frame

to_frame() -> DataFrame

Return one row per specification.

Returns:

Type	Description
DataFrame	Columns consistency_cutoff, pri_cutoff, frequency_cutoff, anchors, solution, consistency, coverage, n_implicants, n_literals, n_solutions and failure.

Source code in src/setqca/analysis/robustness.py

def to_frame(self) -> pd.DataFrame:
    """Return one row per specification.

    Returns
    -------
    pandas.DataFrame
        Columns ``consistency_cutoff``, ``pri_cutoff``, ``frequency_cutoff``,
        ``anchors``, ``solution``, ``consistency``, ``coverage``,
        ``n_implicants``, ``n_literals``, ``n_solutions`` and ``failure``.
    """
    return pd.DataFrame(
        {
            "consistency_cutoff": [run.specification.consistency for run in self.runs],
            "pri_cutoff": [run.specification.pri for run in self.runs],
            "frequency_cutoff": [run.specification.frequency for run in self.runs],
            "anchors": [
                "; ".join(f"{name}={anchor}" for name, anchor in run.specification.anchors)
                for run in self.runs
            ],
            "solution": [" + ".join(sorted(run.terms)) for run in self.runs],
            "consistency": [run.consistency for run in self.runs],
            "coverage": [run.coverage for run in self.runs],
            "n_implicants": [run.implicants for run in self.runs],
            "n_literals": [run.literals for run in self.runs],
            "n_solutions": [run.solutions for run in self.runs],
            "failure": [run.failure for run in self.runs],
        }
    )

solution_similarity

solution_similarity(
    left: frozenset[str],
    right: frozenset[str],
    data: DataFrame,
) -> SolutionSimilarity

Compare two solutions on four scales, from strictest to loosest.

Parameters:

Name	Type	Description	Default
left	frozenset of str	Solution terms, in standard QCA notation.	required
right	frozenset of str	Solution terms, in standard QCA notation.	required
data	DataFrame	Calibrated data, used for the membership comparison.	required

Returns:

Type	Description
SolutionSimilarity	Exact identity, term overlap, configurational overlap and membership agreement.

Source code in src/setqca/analysis/robustness.py

def solution_similarity(
    left: frozenset[str], right: frozenset[str], data: pd.DataFrame
) -> SolutionSimilarity:
    """Compare two solutions on four scales, from strictest to loosest.

    Parameters
    ----------
    left, right : frozenset of str
        Solution terms, in standard QCA notation.
    data : pandas.DataFrame
        Calibrated data, used for the membership comparison.

    Returns
    -------
    SolutionSimilarity
        Exact identity, term overlap, configurational overlap and membership
        agreement.
    """
    left_membership = _membership(left, data)
    right_membership = _membership(right, data)
    union = float(np.maximum(left_membership, right_membership).sum())
    membership = (
        1.0 if union == 0.0 else float(np.minimum(left_membership, right_membership).sum()) / union
    )
    return SolutionSimilarity(
        identical=left == right,
        term_overlap=_jaccard(left, right),
        configurational=_jaccard(_literals(left), _literals(right)),
        membership=membership,
    )

robustness_analysis

robustness_analysis(
    data: DataFrame,
    *,
    outcome: str,
    conditions: Sequence[str],
    grid: RobustnessGrid | None = None,
    family: str = "conservative",
    directional_expectations: Mapping[str, Direction]
    | None = None,
    case_id: str | None = None,
) -> RobustnessAnalysis

Sweep truth-table thresholds and report which paths survive.

Parameters:

Name	Type	Description	Default
data	DataFrame	Calibrated memberships in [0, 1].	required
outcome	str	Name of the outcome column.	required
conditions	sequence of str	Condition columns.	required
grid	RobustnessGrid	Choices to sweep. Defaults to three consistency cutoffs.	None
family	str	Which solution family to track.	"conservative"
directional_expectations	mapping	Required when family is "intermediate".	None
case_id	str	Column holding case labels.	None

Returns:

Type	Description
RobustnessAnalysis	Every specification's result, with term stability across the sweep.

Raises:

Type	Description
ValueError	If the grid specifies calibration anchors, which need raw data — use :func:calibration_robustness for those.

Source code in src/setqca/analysis/robustness.py

def robustness_analysis(
    data: pd.DataFrame,
    *,
    outcome: str,
    conditions: Sequence[str],
    grid: RobustnessGrid | None = None,
    family: str = "conservative",
    directional_expectations: Mapping[str, Direction] | None = None,
    case_id: str | None = None,
) -> RobustnessAnalysis:
    """Sweep truth-table thresholds and report which paths survive.

    Parameters
    ----------
    data : pandas.DataFrame
        Calibrated memberships in ``[0, 1]``.
    outcome : str
        Name of the outcome column.
    conditions : sequence of str
        Condition columns.
    grid : RobustnessGrid, optional
        Choices to sweep. Defaults to three consistency cutoffs.
    family : str, default "conservative"
        Which solution family to track.
    directional_expectations : mapping, optional
        Required when ``family`` is ``"intermediate"``.
    case_id : str, optional
        Column holding case labels.

    Returns
    -------
    RobustnessAnalysis
        Every specification's result, with term stability across the sweep.

    Raises
    ------
    ValueError
        If the grid specifies calibration anchors, which need raw data — use
        :func:`calibration_robustness` for those.
    """
    grid = grid or RobustnessGrid()
    if grid.anchors:
        raise ValueError(
            "Calibration anchors need raw, uncalibrated data; use calibration_robustness instead."
        )
    specifications = list(grid.specifications())
    runs = tuple(
        _run_one(
            data,
            specification,
            outcome=outcome,
            conditions=conditions,
            family=family,
            directional_expectations=directional_expectations,
            case_id=case_id,
        )
        for specification in specifications
    )
    return RobustnessAnalysis(
        grid=grid,
        runs=runs,
        baseline=specifications[len(specifications) // 2],
        family=family,
        data=data,
    )

calibration_robustness

calibration_robustness(
    raw: DataFrame,
    *,
    outcome: str,
    conditions: Sequence[str],
    grid: RobustnessGrid,
    outcome_anchors: tuple[float, float, float],
    base_anchors: Mapping[str, tuple[float, float, float]]
    | None = None,
    family: str = "conservative",
    directional_expectations: Mapping[str, Direction]
    | None = None,
    case_id: str | None = None,
) -> RobustnessAnalysis

Sweep calibration anchors as well as thresholds, starting from raw data.

Calibration is where substantive judgement enters, so it is also where a result is most easily manufactured. This recalibrates from the raw measures for every anchor combination in the grid.

Parameters:

Name	Type	Description	Default
raw	DataFrame	Uncalibrated measures.	required
outcome	str	Name of the outcome column.	required
conditions	sequence of str	Condition columns.	required
grid	RobustnessGrid	Must specify anchors for at least one condition.	required
outcome_anchors	tuple of float	Anchors used to calibrate the outcome, held fixed across the sweep.	required
base_anchors	mapping of str to tuple of float	Anchors for conditions the grid does not sweep. Every condition needs anchors from one source or the other, since the input is raw.	None
family	str	Which solution family to track.	"conservative"
directional_expectations	mapping	Required when family is "intermediate".	None
case_id	str	Column holding case labels.	None

Returns:

Type	Description
RobustnessAnalysis	As :func:robustness_analysis, with anchors recorded per specification.

Raises:

Type	Description
ValueError	If the grid specifies no anchors, or a condition has no anchors at all.
KeyError	If anchors name a condition outside the model.

Source code in src/setqca/analysis/robustness.py

def calibration_robustness(
    raw: pd.DataFrame,
    *,
    outcome: str,
    conditions: Sequence[str],
    grid: RobustnessGrid,
    outcome_anchors: tuple[float, float, float],
    base_anchors: Mapping[str, tuple[float, float, float]] | None = None,
    family: str = "conservative",
    directional_expectations: Mapping[str, Direction] | None = None,
    case_id: str | None = None,
) -> RobustnessAnalysis:
    """Sweep calibration anchors as well as thresholds, starting from raw data.

    Calibration is where substantive judgement enters, so it is also where a
    result is most easily manufactured. This recalibrates from the raw measures
    for every anchor combination in the grid.

    Parameters
    ----------
    raw : pandas.DataFrame
        **Uncalibrated** measures.
    outcome : str
        Name of the outcome column.
    conditions : sequence of str
        Condition columns.
    grid : RobustnessGrid
        Must specify ``anchors`` for at least one condition.
    outcome_anchors : tuple of float
        Anchors used to calibrate the outcome, held fixed across the sweep.
    base_anchors : mapping of str to tuple of float, optional
        Anchors for conditions the grid does **not** sweep. Every condition
        needs anchors from one source or the other, since the input is raw.
    family : str, default "conservative"
        Which solution family to track.
    directional_expectations : mapping, optional
        Required when ``family`` is ``"intermediate"``.
    case_id : str, optional
        Column holding case labels.

    Returns
    -------
    RobustnessAnalysis
        As :func:`robustness_analysis`, with anchors recorded per specification.

    Raises
    ------
    ValueError
        If the grid specifies no anchors, or a condition has no anchors at all.
    KeyError
        If anchors name a condition outside the model.
    """
    if not grid.anchors:
        raise ValueError("calibration_robustness needs a grid with anchors.")
    base = dict(base_anchors or {})
    unknown = (set(grid.anchors) | set(base)) - set(conditions)
    if unknown:
        raise KeyError(f"Anchors reference unknown conditions: {sorted(unknown)}")
    unanchored = [name for name in conditions if name not in grid.anchors and name not in base]
    if unanchored:
        raise ValueError(
            f"The input is raw, so every condition needs anchors; missing: {unanchored}. "
            "Supply them through base_anchors, or sweep them in the grid."
        )

    calibrated_outcome = calibrate_direct(
        raw[outcome].to_numpy(),
        full_out=outcome_anchors[0],
        crossover=outcome_anchors[1],
        full_in=outcome_anchors[2],
    )

    specifications = list(grid.specifications())
    runs: list[RobustnessRun] = []
    reference: pd.DataFrame | None = None

    for specification in specifications:
        frame = pd.DataFrame(index=raw.index)
        anchors = dict(specification.anchors)
        for name in conditions:
            low, crossover, high = anchors.get(name) or base[name]
            frame[name] = calibrate_direct(
                raw[name].to_numpy(), full_out=low, crossover=crossover, full_in=high
            )
        frame[outcome] = calibrated_outcome
        if case_id is not None:
            frame[case_id] = raw[case_id].to_numpy()
        if reference is None:
            reference = frame

        runs.append(
            _run_one(
                frame,
                specification,
                outcome=outcome,
                conditions=conditions,
                family=family,
                directional_expectations=directional_expectations,
                case_id=case_id,
            )
        )

    assert reference is not None
    return RobustnessAnalysis(
        grid=grid,
        runs=tuple(runs),
        baseline=specifications[len(specifications) // 2],
        family=family,
        data=reference,
    )