Reproducibility and Limits

Run it

One command rebuilds everything from seed 34: the simulation, models, analyses, tests and this site.

Rscript run_all.R

The last run passed 106 checks (34 simulation, 72 artifact). Packages are locked in renv.lock (R 4.5.2; probcal 0.2.0, forecastdom 0.1.0, falsifyr 1.0.0). The generator file’s checksum is d23c42f7; any edit changes it. Result checksums and the full session record are in the source repository.

What is fixed

Every analysis choice lives in params.yaml, which the scripts read. The table shows the main ones.

w <- function(n) paste(range(pp("windows", n)), collapse = "–")
tab <- data.frame(
  Setting = c("Random seed", "Windows: classifier / overlay fit / overlay test",
              "Score sharpening", "Risk-band boundaries", "Resampling draws",
              "Trimming rule", "Deletion budget", "Significance level"),
  Value = c(pp("seed"), sprintf("%s / %s / %s", w("classifier_train"), w("calibration_fit"), w("overlay_eval")),
            pp("vendorscore", "sharpen"), sprintf("%s, %s", pct(pp("calibration", "band_low_upper"), 0), pct(pp("calibration", "band_medium_upper"), 0)),
            format(pp("calibration", "bootstrap_draws"), big.mark = ","), sprintf("top %s of patch latency", pct(1 - pp("fragility", "trim_percentile"), 0)),
            pp("fragility", "deletion_budget"), pct(pp("fragility", "alpha"), 0)),
  Basis = c("assumption", "assumption", "planted", "assumption", "convention", "assumption", "assumption", "convention"))
knitr::kable(tab)

Table 1: Main analysis settings and their basis. The full register, with sensitivity ranges, is in params.yaml.

Setting	Value	Basis
Random seed	34	assumption
Windows: classifier / overlay fit / overlay test	1–30 / 31–36 / 37–48	assumption
Score sharpening	1.5	planted
Risk-band boundaries	1%, 3%	assumption
Resampling draws	1,000	convention
Trimming rule	top 1% of patch latency	assumption
Deletion budget	80	assumption
Significance level	5%	convention

How to read the table. Each row is a setting that shapes the results. The last column says whether it is an analyst’s assumption, a standard convention, or a feature planted in the synthetic data.

What the table tells you. Most settings are assumptions. Changing the windows, risk bands or deletion budget would change the reported numbers, so each is recorded in one place.

Limitations

• Synthetic data. Confounding, regime shift and overconfidence are planted. Real use needs NorthPoint’s vendor-month panel, incident definition and dated SOC 2 reports.
• Few events. 75 evaluation incidents and 18 forecast months limit power; more history could change pages 2 and 3.
• Thin overlay window. Six months and 28 incidents is little. A real overlay needs a refit schedule and tolerances set in advance.
• Information timing. Start-of-month availability of patch latency and financial health is assumed; real feeds need timestamps.
• One dataset. Seed 34 was chosen in development; the seed sweep on page 4 checks repeatability only.
• Partial validation. Scores and outcomes only; the vendor’s internals stay unseen.