An experimental comparator, kept out of the main Results page because it does not feed the conclusions on the Overview.
mlReserve() comes from a personal fork of the ChainLadder package (thierrymoudiki/ChainLadder, pinned to commit 30cc7a84, package version 0.2.22), not from CRAN. Its published evidence is a blog post using simulated insurance triangles. It was installed into an isolated library so the CRAN ChainLadder used on every other page is unaffected. Nothing on this page feeds the executive summary.
mlReserve() fits a machine-learning regression (here a random forest from ranger or a support vector machine from e1071) to the observed cells of a triangle, using origin and development period as features, and predicts the unobserved cells. It is a different idea from chain-ladder’s multiplicative development factors.
Same simulated databases as the baseline validation (replications 1-30), annual dollar triangles, scored on the undiscovered loss inside the window. Four configurations are run and all are reported; none was chosen using the true values. The first is the setting used in the blog post. Point estimates only (no bootstrap intervals).
tibble(Configuration = names(cfg$configs),
Learner = map_chr(cfg$configs, "learner"), Features = map_chr(cfg$configs, "features"), Transform = map_chr(cfg$configs, "transform")) |>
gt::gt()| Configuration | Learner | Features | Transform |
|---|---|---|---|
| rf_asinh_numeric | ranger | numeric | asinh |
| rf_raw_numeric | ranger | numeric | none |
| rf_raw_both | ranger | both | none |
| rf_raw_factor | ranger | factor | none |
| svm_raw_numeric | svm | numeric | none |
| svm_raw_factor | svm | factor | none |
How to read the table
Each row is one setting of the machine-learning method: which learning algorithm, which inputs and whether the data were transformed first.
What the table tells you
Six settings were tried and all are reported, including the setting used in the method’s published blog post, and none was chosen after seeing the true answer.
ref <- res$reps |>
filter(scenario == "baseline", granularity == "annual", kind == "amount", rep <= cfg$n_reps,
method %in% c("mack", "boot", "freq_x_sev")) |>
transmute(type, config = method_lab[method], est = if_else(status == "ok", est, NA_real_), truth = truth_window, rep)
mlr <- ml$results |> transmute(type, config = paste("mlReserve:", config), est = if_else(status == "ok", est, NA_real_), truth, rep)
cmp <- bind_rows(ref, mlr)
sc_ml <- cmp |>
summarise(
usable = sum(is.finite(est)),
rel_bias = sum(est - truth, na.rm = TRUE) / sum(truth[is.finite(est)]),
median_ratio = median(est / truth, na.rm = TRUE),
.by = c(type, config)
) |>
mutate(type = factor(type_lab[type], levels = type_lab)) |>
arrange(type, config)sc_ml |>
gt::gt(groupname_col = "type") |>
gt::fmt_percent(rel_bias, decimals = 0) |>
gt::fmt_number(median_ratio, decimals = 2) |>
gt::cols_label(config = "Method", usable = "Usable fits", rel_bias = "Relative bias", median_ratio = "Median estimate / truth") |>
gt::sub_missing(missing_text = "-")| Method | Usable fits | Relative bias | Median estimate / truth |
|---|---|---|---|
| Trade errors | |||
| Bootstrap | 30 | −1% | 0.97 |
| Frequency x severity | 30 | −7% | 0.91 |
| Mack (package) | 30 | −1% | 0.96 |
| mlReserve: rf_asinh_numeric | 30 | 19,597% | 30.89 |
| mlReserve: rf_raw_both | 30 | 1,813% | 19.88 |
| mlReserve: rf_raw_factor | 30 | 3,888% | 41.31 |
| mlReserve: rf_raw_numeric | 30 | 3,666% | 38.19 |
| mlReserve: svm_raw_factor | 30 | 4,085% | 43.21 |
| mlReserve: svm_raw_numeric | 30 | 4,482% | 46.25 |
| NAV errors | |||
| Bootstrap | 30 | −1% | 0.86 |
| Frequency x severity | 30 | −16% | 0.78 |
| Mack (package) | 30 | −4% | 0.84 |
| mlReserve: rf_asinh_numeric | 30 | 339,214% | 87.44 |
| mlReserve: rf_raw_both | 30 | 212% | 3.03 |
| mlReserve: rf_raw_factor | 30 | 321% | 4.22 |
| mlReserve: rf_raw_numeric | 30 | 300% | 4.03 |
| mlReserve: svm_raw_factor | 30 | 321% | 4.46 |
| mlReserve: svm_raw_numeric | 30 | 315% | 3.92 |
| Unauthorized trading | |||
| Bootstrap | 0 | - | - |
| Frequency x severity | 30 | 37% | 1.40 |
| Mack (package) | 0 | - | - |
| mlReserve: rf_asinh_numeric | 30 | 17,670,308% | 135.61 |
| mlReserve: rf_raw_both | 30 | 633% | 5.05 |
| mlReserve: rf_raw_factor | 30 | 492% | 5.65 |
| mlReserve: rf_raw_numeric | 30 | 432% | 3.73 |
| mlReserve: svm_raw_factor | 30 | 114% | 2.32 |
| mlReserve: svm_raw_numeric | 30 | 29% | 1.26 |
How to read the table
Each block is an event type. “Median estimate / truth” shows how many times the true loss the method estimated in a typical database, so 1.00 is exact, 2.00 is twice too high and 0.50 is half.
What the table tells you
On NAV errors the machine-learning settings are at best 3.0 times the truth, against 0.84 for the standard Mack method. For trade errors the best is 20 times, so the experimental method overstates the missing loss by large multiples where standard methods are accurate.
cmp |>
filter(is.finite(est), truth > 0) |>
mutate(ratio = est / truth, type = factor(type_lab[type], levels = type_lab),
ml = str_detect(config, "^mlReserve")) |>
ggplot(aes(ratio, fct_rev(config), fill = ml)) +
geom_vline(xintercept = 1, colour = "grey40") +
geom_boxplot(outlier.size = 0.6, linewidth = 0.3) +
facet_wrap(~type, nrow = 1) +
scale_x_log10(labels = \(x) paste0(x, "x")) +
scale_fill_manual(values = c(`TRUE` = "#E69F00", `FALSE` = "#0072B2"), guide = "none") +
labs(x = "Estimate / truth (log scale; 1x = exact)", y = NULL, title = "mlReserve vs. classical methods on the same databases") +
theme_op()
How to read the plot
Each box shows how far the estimate was from the truth across the simulated databases on a multiplicative scale: 1x is exact, 10x is ten times too high. Blue boxes are the standard methods and orange boxes are the machine-learning settings.
What the plot tells you
The orange boxes sit far to the right of the blue ones for trade and NAV errors, so the experimental method consistently overstates. For unauthorized trading all methods are widely scattered, so the plot gives no reason to prefer the machine-learning approach there either.
mlsc <- sc_ml |> filter(str_detect(config, "^mlReserve"))
closest <- \(t) mlsc |> filter(type == t) |> slice_min(abs(log(median_ratio)), n = 1)
worst <- \(t) mlsc |> filter(type == t) |> slice_max(abs(log(median_ratio)), n = 1)
mack <- \(t) sc_ml |> filter(type == t, config == "Mack (package)")
nav_c <- closest("NAV errors"); trade_c <- closest("Trade errors"); un_c <- closest("Unauthorized trading")
asinh_trade <- sc_ml |> filter(type == "Trade errors", config == "mlReserve: rf_asinh_numeric")mlReserve configuration overestimates on the two event types where classical methods work. For NAV errors the closest configuration (mlReserve: rf_raw_both) has a median estimate/truth ratio of 3, against 0.84 for Mack. For trade errors the closest (mlReserve: rf_raw_both) is 20x the truth, against 0.96 for Mack.asinh transform, applied to dollar values, gives a median ratio of 31 for trade errors, worse than the same learner with no transform in the cases we ran; it may be calibrated for much smaller numeric scales, and we did not test rescaling the data first.Taken together, this is a limited test, on one data-generating process, with no tuning, of an unreleased fork. It gives no support for using mlReserve here, and it does not show that the approach cannot work with more data or tuning.