This page documents how to rebuild every number on the site, and checks a few of those claims live: the tests, the environment status and the parameter staleness check re-run every time this page is rendered (it is excluded from the project-wide freeze: auto cache).
One command
From a clean clone:
Rscript-e'renv::restore()'# once: package versions from renv.lockRscript run_all.R # simulations -> mlReserve experiment -> site in docs/
run_all.R runs run_simulations.R (replications, control study, break-even), installs the pinned mlReserve fork into an isolated library if needed and runs run_mlreserve.R, then calls quarto render.
The package versions are pinned in renv.lock. Status of the project library against the lockfile:
No issues found -- the project is in a consistent state.
The experimental fork is installed from GitHub at the pinned commit into lib_mlreserve/, outside renv, so installing it cannot change the CRAN ChainLadder version.
Randomness
All randomness is seeded explicitly, so results do not depend on how many cores are used (tested). A master seed in params.yaml (master_seed) generates a seed for every stream:
Seeds depend on the replication number, the scenario index and the event-type index, so the same replication in different scenarios shares nothing accidentally, and the baseline and control arms of the control study share seeds on purpose (paired comparison). Bootstrap and frequency-severity simulations take a seed derived the same way.
Parameters
Every simulator parameter lives in one file. status: assumption marks values that are plausible choices, not calibrated estimates.
# Single source of truth for all simulator parameters.# status: "assumption" = chosen for plausibility, not calibrated to data.# Every value is synthetic; replace with client data in a real engagement.master_seed:20261009firm:aum_usd:5.0e+9 # stylised mid-size buy-side firm; used only to express losses in bps of AUMstatus: assumptionhorizon:n_origin:40 # origin quarters (10 years); evaluation date = end of quarter 40n_lag:40 # discovery lags 0..39 inside the window; lag >= 40 is "tail"status: assumptionreporting_threshold_usd:0 # events below this loss are not recorded (collection threshold); real databases often use $10k-$20k # status: assumptionexposure:base:1.0growth_per_quarter:0.01 # ~4% p.a. volume growth; range 0 - 0.03status: assumptionreplication:n_reps:200n_boot:1000 # BootChainLadder resamples per triangleruns: # scenario -> granularities scored (annual = 4 quarters per period)baseline:[annual, quarterly]heavy_tail_shock:[annual]control_improvement:[annual]exposure_growth:[annual]overdispersion:[annual]reporting_threshold_10k:[annual]recent_quarters:4 # "recent" = latest 4 origin quarters (1 annual period)status: assumptionmlreserve: # experimental comparator, see mlreserve.qmdn_reps:30commit: 30cc7a8402f29713d3f53861e42b092cbc6946f1 # thierrymoudiki/ChainLadder (fork of mages/ChainLadder)configs: # blog setting first; the rest are variants incl. the package default (factor features), all scored on the same replicationsrf_asinh_numeric:{learner: ranger,features: numeric,transform: asinh}rf_raw_numeric:{learner: ranger,features: numeric,transform: none}rf_raw_both:{learner: ranger,features: both,transform: none}rf_raw_factor:{learner: ranger,features: factor,transform: none} # package default featuressvm_raw_numeric:{learner: svm,features: numeric,transform: none}svm_raw_factor:{learner: svm,features: factor,transform: none}status: assumptioncontrols: # detection controls; effects are applied as shifts to the lag distributionrecon_automation:description: Automated daily position/cash reconciliation against prime-broker statementsannual_cost_usd:400000 # assumption; range 250k - 600klag_shift:nav_error:{slow_meanlog:-0.4,w_fast:0.10}unauthorized_trading:{slow_meanlog:-0.4}status: assumptionindependent_pnl_verification:description: Independent P&L and limit-breach verification with prime-broker position feedannual_cost_usd:700000 # assumption; range 400k - 1.2Mlag_shift:trade_error:{w_fast:0.05}nav_error:{slow_meanlog:-0.3}unauthorized_trading:{slow_meanlog:-0.8,w_fast:0.15}status: assumptionevent_types:trade_error: # Basel: Execution, Delivery & Process Mgmtlambda:60 # events / quarter at exposure 1; range 30 - 120lag:{w_fast:0.85,p_fast:0.80,slow_meanlog:0.5,slow_sdlog:0.8}severity:{meanlog:7.8,sdlog:1.2,p_tail:0.03,tail_q:0.95,xi:0.2,beta:8000,lag_elasticity:0.02}status: assumptionnav_error: # Basel: EDPM / Clients, Products & Business Practiceslambda:6 # range 3 - 15lag:{w_fast:0.50,p_fast:0.40,slow_meanlog:1.5,slow_sdlog:0.8}severity:{meanlog:10.3,sdlog:1.4,p_tail:0.05,tail_q:0.95,xi:0.3,beta:80000,lag_elasticity:0.05}status: assumptionunauthorized_trading: # Basel: Internal Fraud (incl. limit and mandate breaches)lambda:0.3 # ~1.2 per year; range 0.1 - 0.8lag:{w_fast:0.10,p_fast:0.30,slow_meanlog:2.0,slow_sdlog:0.9}severity:{meanlog:12.4,sdlog:1.6,p_tail:0.08,tail_q:0.90,xi:0.6,beta:1500000,lag_elasticity:0.08}status: assumptionscenarios:baseline:{}heavy_tail_shock:injected_events: # absolute-dollar events added to the baseline-{type: unauthorized_trading,origin:12,lag:15,severity:60000000}-{type: unauthorized_trading,origin:20,lag:22,severity:40000000}control_improvement: # time remaining to discovery, measured from max(origin, quarter 24), falls by `reduction` (stochastic rounding)control_shift:{calendar_quarter:24,reduction:0.5}exposure_growth:exposure:{growth_per_quarter:0.03}overdispersion: # negative-binomial counts with the same mean; var = mu + mu^2 / size (assumption)count_dist: negbincount_size:{trade_error:20,nav_error:6,unauthorized_trading:1}reporting_threshold_10k: # losses below $10k are never recorded (assumption)reporting_threshold_usd:10000
The file holds 110 leaf values in 10 blocks marked as assumptions. No parameter is calibrated to data in this version.
Data lineage
Generated data are outputs, not sources of truth: the generator and params.yaml are. Content hashes (SHA-256 of the deserialised object, serialised with R serialization version 3 so they do not depend on the R release) let you confirm you reproduced the same results.
How to read the table Each row is a stored results file, with a short fingerprint of its contents and the number of rows.
What the table tells you If you rebuild the project and get the same fingerprints, you have reproduced the same results; a different fingerprint means the inputs, code or software versions differ.
Staleness check:params.yaml has not changed since the results were generated (results are current). Results were produced 2026-10-11 22:25:57 UTC with 200 replications on 10 workers.
How to read the table Each row is a group of automated checks on the code, counting how many ran and how many failed, errored or were skipped.
What the table tells you All 106 checks across 39 tests ran with 0 failures and 0 errors, so the calculations behind the site, from triangle building to scoring, behaved as designed when this page was last built.
What the tests cover:
Triangle construction reconciles exactly to the event-level data (counts and amounts, quarterly and annual) and never leaks undiscovered events into the observed region.
The generator is deterministic for a seed, rejects invalid parameters, keeps lags non-negative under the control shift, injects shock events exactly, and the control-improvement shift reduces time remaining by the stated fraction measured from the later of the shift quarter and the origin (events originating after the shift are not collapsed to lag 0).
The negative-binomial scenario has the same mean count as Poisson but larger variance, and the reporting threshold removes only losses below it.
Estimators: the in-house chain-ladder matches the package on a dense triangle, tolerates empty development columns, never throws (errors become failed), and sparse triangles are reported as failed or suspect, never as silent zeros. The Mack implementation with empty-column handling reproduces ChainLadder::MackChainLadder on a dense triangle (IBNR and standard error), runs where the package stops, and a zero bootstrap estimate is flagged suspect. Bornhuetter-Ferguson, Benktander and the benchmark development pattern are checked for coherence (shares in [0, 1], Benktander between BF and chain-ladder, pattern matches simulation).
Scoring and control functions are checked against hand-computed values.
A replication run is identical on 1 and 2 cores.
Caching and runtime
Pages cache their chunk results with Quarto freeze: auto; a page re-executes when its source changes. To force a full rebuild, delete _freeze/ and docs/. The heavy computation is not in the pages at all: it is in run_simulations.R, which took 7.9 minutes for the replications and 4 seconds for the control study and break-even on this run.
Known limits of reproducibility
Bit-for-bit equality is expected only on the same platform and package versions. Floating-point and BLAS differences across machines can change low-order digits, and occasionally the optimiser path of the Clark fits.
The mlReserve fork is a moving target on GitHub; the pinned commit keeps the experiment fixed, but the fork could disappear.
Quarto and Typst versions affect rendering, not the numbers.
Limitations and model risk
This project tests how estimation methods behave on data we generated. That is its strength (the true answer is known) and its main limitation (the answer holds only for this generator).
Key limitations
Synthetic, uncalibrated parameters. Frequencies, lag distributions, severities and control effects are assumptions chosen for plausibility. Different values would change the numbers, and may change which method works.
The generator and the methods share structure. Discovery lag is independent of origin period apart from the explicit scenarios, which is close to what chain-ladder assumes. Real loss databases have reporting-policy changes, system migrations and batch back-fills that create calendar-period effects the baseline does not contain.
Loss is booked in full at discovery. Real events also have payment and recovery development after discovery. This is a second triangle layer that is not modelled.
Event types are independent. In reality a market shock raises trade errors, NAV errors and unauthorized-position losses together.
Control effects are assumed. The savings come from a posited link between time undiscovered and loss size, and an assumed shift in the lag distribution. We have no evidence for either magnitude.
Frequency x severity ignores parameter uncertainty, so its intervals understate uncertainty by construction.
Interval method. Mack and Clark intervals use a lognormal approximation to the total; the bootstrap uses empirical quantiles. Other choices would change coverage numbers somewhat.
Annual vs. quarterly. Headline results use annual triangles because quarterly triangles are mostly unusable for the standard methods. A real firm with a long, dense history might do better at finer granularity.
One firm size. A $5B stylised firm; no other sizes were tested.
Poisson frequency, no clustering. Baseline counts are independent Poisson. Real operational-loss counts are overdispersed, and common-cause failures (for example a pricing feed that fails across many funds) produce batches of errors. The negative-binomial scenario adds overdispersion within an event type, but not clustering across types or across quarters.
No reporting threshold in the baseline. Real loss databases only record losses above a collection threshold, often $10k to $20k. The median simulated trade error is about $2.4k, so a real database would hold far fewer trade errors. A $10k threshold is run as a sensitivity scenario.
Some “method failures” are package limits, not method limits.ChainLadder::MackChainLadder() stops on triangles with empty development columns. Results with empty-column handling (factor 1) are reported next to the package results.
Frequency x severity donor sparsity. At long lags few observed events exist, so severities are resampled from the few that do (or from all events when none are nearby, which drops the lag uplift). A variant that pools severities and re-applies an estimated uplift is reported.
Every control result scales with the severity-growth parameter \(e\). With \(e = 0\) incurred loss is the same with and without the control (see Control Value).
Pure IBNR only. Loss is booked in full at discovery, so there is no development on events already known (provisions, recoveries, insurance). The terminology here is IBNR, not IBNER.
The benchmark development pattern is the generator’s own. The Bornhuetter-Ferguson variant with an external pattern shows what an accurate prior pattern buys; with a pattern that is itself wrong it would do worse, and the study does not test that.
What would change the conclusions
If we found…
Effect
Real unauthorized-trading events are far more frequent than ~1 per year
Standard methods become usable; the “cannot run” finding would weaken
Detection lags are stable over time in real data
The control-improvement failure matters less in practice
Loss size does not grow with time undiscovered
Control savings on expected loss largely disappear
Strong calendar-period effects in real data
All development methods would need an explicit calendar term
Losses arrive in clusters from a common failure
Chain-ladder variance is understated; Bornhuetter-Ferguson with a pooled pattern is preferred
Real collection threshold of $10k or more
The trade-error triangle is mostly empty and the standard methods have little to work with
What client data would replace each assumption
Synthetic parameter
Client source
Event frequency by type
Internal loss event database, trade-break and reconciliation logs
Discovery-lag distribution
Timestamps for occurrence, detection and booking in the loss database
Severity and tail
Booked losses; external data (for example consortium loss data) for the tail
Severity growth with lag
Cases where the same error type was found early and late
Control effect on lag
Before/after detection times around past control changes
Control cost
Vendor quotes and internal run cost
Firm size
Actual AUM and trade volumes
Use of the results
The numbers on this site are illustrations of method behaviour. They should not be quoted as estimates of any firm’s undiscovered losses or as a benchmark for the value of a particular control.