From a62db7537c20d0418fd00197d8183aae1432d58c Mon Sep 17 00:00:00 2001 From: Jeffrey Girard Date: Wed, 21 Oct 2026 22:55:49 -0500 Subject: [PATCH 01/13] M151: start, branch cut from 3af50898 --- cairn/ROADMAP.md | 2 +- cairn/milestones/M151-time-series-tuner-cut.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/cairn/ROADMAP.md b/cairn/ROADMAP.md index 11e677e8..77e0710b 100644 --- a/cairn/ROADMAP.md +++ b/cairn/ROADMAP.md @@ -4,7 +4,7 @@ _Last hygiene check: 2026-10-20 triage pass over 41 candidates and 20 Known issu ## Milestones | ID | Title | Status | Depends on | Priority | File/Archive | |---|---|---|---|---|---| -| M151 | The four other tuners are tested on one inner sliding design, and the help claims only what the tests run | planned | — | high | milestones/M151-time-series-tuner-cut.md | +| M151 | The four other tuners are tested on one inner sliding design, and the help claims only what the tests run | in-progress | — | high | milestones/M151-time-series-tuner-cut.md | | M152 | The suite drops tests that repeat a shared path, cannot fail, or test a dependency | planned | — | normal | milestones/M152-suite-shared-path-prune.md | | M153 | The devel-vctrs job runs the vctrs test files on each push and the whole suite each week | planned | — | normal | milestones/M153-devel-vctrs-subset.md | | M154 | The wait ledger finds each call by its test block, so moving a line breaks no test | planned | M152 | normal | milestones/M154-wait-ledger-rekey.md | diff --git a/cairn/milestones/M151-time-series-tuner-cut.md b/cairn/milestones/M151-time-series-tuner-cut.md index 6a6ac12a..a59cd9ac 100644 --- a/cairn/milestones/M151-time-series-tuner-cut.md +++ b/cairn/milestones/M151-time-series-tuner-cut.md @@ -1,13 +1,13 @@ # M151: The four other tuners are tested on one inner sliding design, and the help claims only what the tests run -- **Status:** planned +- **Status:** in-progress - **Priority:** high - **Depends on:** — - **Driving RR:** — - **Principles touched:** — - **Resolves:** — - **Surface tier:** user-facing — the help of `nested_resamples()` and `nested_tune_grid()` and `NEWS.md` state a narrower tested set -- **Branch/PR:** — +- **Branch/PR:** m151-time-series-tuner-cut ## Goal From 176ce6666b6f9168db6c8caa4f1238861d87bec1 Mon Sep 17 00:00:00 2001 From: Jeffrey Girard Date: Wed, 21 Oct 2026 22:58:13 -0500 Subject: [PATCH 02/13] M151 T2: drop the inner tuner matrix and the sliding outer final fits --- DESCRIPTION | 6 +- .../milestones/M151-time-series-tuner-cut.md | 4 +- tests/testthat/test-time-series-designs.R | 28 +-- .../test-time-series-inner-bayes-anneal.R | 191 ------------------ tests/testthat/test-time-series-inner-other.R | 62 +----- tests/testthat/test-time-series-inner-race.R | 120 ----------- 6 files changed, 11 insertions(+), 400 deletions(-) delete mode 100644 tests/testthat/test-time-series-inner-bayes-anneal.R delete mode 100644 tests/testthat/test-time-series-inner-race.R diff --git a/DESCRIPTION b/DESCRIPTION index c52f2686..95892c11 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -61,8 +61,8 @@ Config/Needs/website: tidymodels, tidyverse/tidytemplate, pkgdown Config/testthat/edition: 3 Config/testthat/parallel: true Config/testthat/start-first: parallel-identity, time-series-designs, - time-series-inner-bayes-anneal, nested-tune-race-oracles, - nested-tune-grid-oracles, nested-tune-bayes-oracles, nested-results-print, nested-tune-finalize, + nested-tune-race-oracles, nested-tune-grid-oracles, + nested-tune-bayes-oracles, nested-results-print, nested-tune-finalize, time-series-bayes, model-spec-input, nested-tune-grid-failures, - time-series-race, time-series-inner-race, time-series-pairs-other + time-series-race, time-series-pairs-other Config/roxygen2/version: 8.1.0 diff --git a/cairn/milestones/M151-time-series-tuner-cut.md b/cairn/milestones/M151-time-series-tuner-cut.md index a59cd9ac..aa5a5dc9 100644 --- a/cairn/milestones/M151-time-series-tuner-cut.md +++ b/cairn/milestones/M151-time-series-tuner-cut.md @@ -50,7 +50,7 @@ Cut the suite time that the inner-design tuner matrix costs. The help and `NEWS. ## Tasks - [ ] T1: Record the branch point. Run `benchmarks/test-blocks.R`, `benchmarks/coverage-lines.R` and `benchmarks/profile-tests.R 1` on it in the background, with outputs in the scratchpad. -- [ ] T2: Delete the files and blocks that Scope In names. If no other file calls a fixture helper, delete it. Update `DESCRIPTION` start-first. Check `helper-time-budget.R` for rows on the deleted files. +- [x] T2: Delete the files and blocks that Scope In names. If no other file calls a fixture helper, delete it. Update `DESCRIPTION` start-first. Check `helper-time-budget.R` for rows on the deleted files. - [ ] T3: Rewrite the help section and sentences, and run `devtools::document()`. Rewrite the `NEWS.md` bullets. Write each claim against a head test block. - [ ] T4: Write the D-entry, and widen the inner-designs candidate row. - [ ] T5: Run the three scripts at the head and compare. Write the block comparison, a claim ledger with one row per help or `NEWS.md` sentence, the timing and the coverage results to the Review section. @@ -68,6 +68,8 @@ Cut the suite time that the inner-design tuner matrix costs. The help and `NEWS. - 2026-10-21: split: the four requested pieces became M151 to M155; the ledger re-key and the comment cleanup were split into M154 and M155, because one goal needed "and". - 2026-10-21: inbox sweep: one open issue (#91, docs pass), no overlap; no open PRs. - 2026-10-21: lessons harvested: M51 (a CI leg's time moves with the runner, so the timing criteria use local serial runs on one machine). +- 2026-10-21: T1 started: branch point 3af50898 checked out as a detached worktree in the scratchpad; test-blocks and coverage-lines run there in the background; the branch-point profile runs alone after them, so its timing has no contention. +- 2026-10-21: T2 done: deleted the two inner tuner files, the four map-with-tuner blocks and the three sliding outer final fits; no helper became unused (each is called from another test file); start-first drops the two files; the ledger has no rows on them; the O2 and O3 oracle headers no longer name the deleted tests. The edited files and the start-first guard pass. ## Decisions diff --git a/tests/testthat/test-time-series-designs.R b/tests/testthat/test-time-series-designs.R index 3acf42bd..59c8e31f 100644 --- a/tests/testthat/test-time-series-designs.R +++ b/tests/testthat/test-time-series-designs.R @@ -18,10 +18,10 @@ # O3 -- type "live" (reference implementation). Source: tune::tune_grid(), # tune::select_best() and fit() run by hand under the final fit's # `tuning_seed` and `fit_seed`, on an inner design built on the full data -# from the fixture's literal `rolling_origin()` call. Pinned by the four -# "the final fit on a ... design matches a hand-rolled reference" tests, -# one per outer design. Satisfies M108 AC4 and M111 AC4, and backs the -# help's claim for `nested_final_fit()` on each design. +# from the fixture's literal `rolling_origin()` call. Pinned by "the final +# fit on a rolling-origin design matches a hand-rolled reference". The +# final fit reads the data, not the outer splits, so one outer design +# covers its path. # # The M110 oracles for the Bayesian, racing and annealing tuners are recorded # in test-time-series-bayes.R, test-time-series-race.R and @@ -150,12 +150,6 @@ test_that("the final fit on a rolling-origin design matches a hand-rolled refere expect_final_matches_reference(d, ts_rolling_nested(d)) }) -test_that("the final fit on a sliding-window design matches a hand-rolled reference", { - skip_if_no_engines() - d <- make_reg_data() - expect_final_matches_reference(d, ts_sliding_nested(d)) -}) - # ---- Index-based and period-based designs (M111) ----------------------------- # # The O1, O2 and O3 oracles above, run on `sliding_index()` and @@ -230,20 +224,6 @@ for (design in names(TS_NEW_OUTER_CALLS)) { expect_outer_identical(lean, ref) expect_inner_identical(lean, ref) }) - - test_that( - sprintf( - "the final fit on a %s design matches a hand-rolled reference", - design - ), - { - skip_if_no_engines() - d <- make_ts_data() - folds <- build(d) - expect_s3_class(folds$splits[[1]], TS_SPLIT_CLASS[[design]]) - expect_final_matches_reference(d, folds) - } - ) } # ---- AC5: predictions and augment() ----------------------------------------- diff --git a/tests/testthat/test-time-series-inner-bayes-anneal.R b/tests/testthat/test-time-series-inner-bayes-anneal.R deleted file mode 100644 index 9c2a38ac..00000000 --- a/tests/testthat/test-time-series-inner-bayes-anneal.R +++ /dev/null @@ -1,191 +0,0 @@ -# nested_tune_bayes() and nested_tune_sim_anneal() on the three inner sliding -# designs under an outer `rolling_origin()` (M120), and their final fits on -# the three designs (M120, M140). The fixtures are TS_INNER_DESIGNS in -# helper-orchestration.R. DESIGN Conventions: oracles are recorded in the test -# file that asserts them. -# -# O1 -- type "live" (reference implementation). Source: -# reference_nested_bayes_loop() and reference_nested_anneal_loop() in -# helper-orchestration.R, written from the seed contract rather than from -# the driver. Unlike the outer design, the inner design reaches the tuner: -# each fold hands it that fold's inner resamples. Pinned by the -# " matches its reference loop on an inner ... design" tests. -# Satisfies M120 AC1 for these tuners. -# -# O2 -- type "live" (reference implementation). Source: -# reference_bayes_final_fit() and reference_anneal_final_fit() in -# helper-orchestration.R, handed the fixture's literal inner call as -# `inner_design`, so the reference builds the inner design on the full data -# from its own spelling of the call. Pinned by "the final fit on an -# inner ... result matches its reference". Satisfies M120 AC2 for these -# tuners on the sliding-period design, and M140 AC1 on the sliding-window -# and sliding-index designs. Each run comes from the fixture cache that the -# O1 test filled. -# -# O1 and O2 check that the orchestrators give what tune and finetune give when -# run by hand. The estimate itself adds nothing new for these designs, so no -# second oracle type is asked of them here. - -skip_heavy_on_cran() - -ts_inner_bayes_run <- function(wf, folds, p, ms) { - set.seed(22) - memoised(nested_tune_bayes( - wf, - folds, - iter = 2, - initial = 3, - param_info = p, - metrics = ms - )) -} - -ts_inner_anneal_run <- function(wf, folds, ms, ctrl) { - set.seed(24) - memoised(nested_tune_sim_anneal( - wf, - folds, - iter = 2, - initial = 3, - metrics = ms, - control = ctrl - )) -} - -for (design in names(TS_INNER_DESIGNS)) { - spec <- TS_INNER_DESIGNS[[design]] - - test_that( - sprintf( - "nested_tune_bayes() matches its reference loop on an inner %s design", - design - ), - { - skip_if_no_bayes_fixture() - d <- spec$data() - wf <- bayes_workflow(d) - folds <- spec$build(d) - expect_s3_class( - folds$inner_resamples[[1]]$splits[[1]], - spec$split_class - ) - p <- bayes_param_info(wf) - ms <- ts_metrics() - - res <- ts_inner_bayes_run(wf, folds, p, ms) - ref <- memoised(reference_nested_bayes_loop( - wf, - folds, - iter = 2, - initial = 3, - objective = tune::exp_improve(), - param_info = p, - metrics = ms, - seed = 22, - metric_name = "rmse" - )) - - expect_ts_matches_reference(res, ref) - } - ) - - test_that( - sprintf( - "nested_tune_sim_anneal() matches its reference loop on an inner %s design", - design - ), - { - skip_if_no_anneal_fixture() - d <- spec$data() - wf <- det_workflow(d) - folds <- spec$build(d) - expect_s3_class( - folds$inner_resamples[[1]]$splits[[1]], - spec$split_class - ) - ms <- ts_metrics() - ctrl <- anneal_control() - - res <- ts_inner_anneal_run(wf, folds, ms, ctrl) - ref <- memoised(reference_nested_anneal_loop( - wf, - folds, - iter = 2, - initial = 3, - metrics = ms, - seed = 24, - metric_name = "rmse", - control = ctrl - )) - - expect_ts_matches_reference(res, ref) - } - ) -} - -for (design in names(TS_INNER_DESIGNS)) { - spec <- TS_INNER_DESIGNS[[design]] - - test_that( - sprintf( - "the Bayesian final fit on an inner %s result matches its reference", - design - ), - { - skip_if_no_bayes_fixture() - d <- spec$data() - wf <- bayes_workflow(d) - folds <- spec$build(d) - p <- bayes_param_info(wf) - ms <- ts_metrics() - - res <- ts_inner_bayes_run(wf, folds, p, ms) - set.seed(41) - final <- nested_final_fit(wf, res) - ref <- reference_bayes_final_fit( - wf, - d, - iter = 2, - initial = 3, - objective = tune::exp_improve(), - param_info = p, - metrics = ms, - seed = 41, - metric_name = "rmse", - inner_design = spec$inner - ) - expect_ts_final_matches(final, ref, d, split_class = spec$split_class) - } - ) - - test_that( - sprintf( - "the annealing final fit on an inner %s result matches its reference", - design - ), - { - skip_if_no_anneal_fixture() - d <- spec$data() - wf <- det_workflow(d) - folds <- spec$build(d) - ms <- ts_metrics() - ctrl <- anneal_control() - - res <- ts_inner_anneal_run(wf, folds, ms, ctrl) - set.seed(43) - final <- nested_final_fit(wf, res) - ref <- reference_anneal_final_fit( - wf, - d, - iter = 2, - initial = 3, - metrics = ms, - seed = 43, - metric_name = "rmse", - control = ctrl, - inner_design = spec$inner - ) - expect_ts_final_matches(final, ref, d, split_class = spec$split_class) - } - ) -} diff --git a/tests/testthat/test-time-series-inner-other.R b/tests/testthat/test-time-series-inner-other.R index a9edd920..ee1734c2 100644 --- a/tests/testthat/test-time-series-inner-other.R +++ b/tests/testthat/test-time-series-inner-other.R @@ -15,9 +15,7 @@ # helper-orchestration.R, which runs each workflow of the set through its # own orchestrator by hand under the same seed. Pinned by the "a workflow # set on an inner ... design matches the hand calls" tests. Satisfies M120 -# AC3 for nested_workflow_map(). The same oracle pins "a workflow set run -# with fn = ... on an inner sliding-period design matches the hand calls", -# one test for each tuner other than grid. Satisfies M140 AC2. +# AC3 for nested_workflow_map(). skip_heavy_on_cran() @@ -114,61 +112,3 @@ for (design in names(TS_INNER_DESIGNS)) { } ) } - -# The map with each tuner other than grid, on the inner sliding-period design -# alone (M140). Each tuner's own run is tested on all three inner designs in -# test-time-series-inner-bayes-anneal.R and test-time-series-inner-race.R. -for (fn in c( - "nested_tune_bayes", - "nested_tune_race_anova", - "nested_tune_race_win_loss", - "nested_tune_sim_anneal" -)) { - test_that( - sprintf( - "a workflow set run with fn = %s on an inner sliding-period design matches the hand calls", - fn - ), - { - skip_if_no_wset_fixture(fn) - spec <- TS_INNER_DESIGNS[["sliding-period"]] - d <- spec$data() - wset <- wset_two(d) - folds <- spec$build(d) - expect_s3_class( - folds$inner_resamples[[1]]$splits[[1]], - spec$split_class - ) - ms <- ts_metrics() - - set.seed(27) - res <- rlang::inject(nested_workflow_map( - wset, - fn = fn, - resamples = folds, - metrics = ms, - !!!wset_map_args(fn) - )) - - expect_identical(res$wflow_id, c("tuned", "fixed")) - for (i in seq_len(nrow(wset))) { - wf <- wset$info[[i]]$workflow[[1L]] - hand <- hand_call(fn, wf, folds, ms, seed = 27) - expect_true(all(hand$.completed)) - # A race that drops no candidate scores as grid does on this fixture, - # so the tuner that ran is compared as well. - expect_identical( - attr(res$result[[i]], "procedure")$tuner, - attr(hand, "procedure")$tuner - ) - expect_identical(res$result[[i]]$.tuning_seed, hand$.tuning_seed) - expect_identical( - res$result[[i]]$.outer_fit_seed, - hand$.outer_fit_seed - ) - expect_identical(res$result[[i]]$.metrics, hand$.metrics) - expect_identical(res$result[[i]]$.selected, hand$.selected) - } - } - ) -} diff --git a/tests/testthat/test-time-series-inner-race.R b/tests/testthat/test-time-series-inner-race.R deleted file mode 100644 index 77c18e52..00000000 --- a/tests/testthat/test-time-series-inner-race.R +++ /dev/null @@ -1,120 +0,0 @@ -# The two racing tuners on the three inner sliding designs under an outer -# `rolling_origin()` (M120), and their final fits on the three designs (M120, -# M140). The fixtures are TS_INNER_DESIGNS in helper-orchestration.R, each -# with at least three inner resamples in every fold, which a racer at -# `burn_in = 2` needs. DESIGN Conventions: oracles are recorded in the test -# file that asserts them. -# -# O1 -- type "live" (reference implementation). Source: -# reference_nested_race_loop() in helper-orchestration.R, written from the -# seed contract rather than from the driver. Unlike the outer design, the -# inner design reaches the racer: each fold hands it that fold's inner -# resamples. Pinned by the " matches its reference loop on an inner -# ... design" tests. Satisfies M120 AC1 for these tuners. -# -# O2 -- type "live" (reference implementation). Source: -# reference_race_final_fit() in helper-orchestration.R, handed the fixture's -# literal inner call as `inner_design`, so the reference builds the inner -# design on the full data from its own spelling of the call. Pinned by "the -# final fit on an inner ... result matches its reference". -# Satisfies M120 AC2 for these tuners on the sliding-period design, and -# M140 AC1 on the sliding-window and sliding-index designs. Each run comes -# from the fixture cache that the O1 test filled. -# -# O1 and O2 check that the orchestrators give what finetune gives when run by -# hand. The estimate itself adds nothing new for these designs, so no second -# oracle type is asked of them here. - -skip_heavy_on_cran() - -ts_inner_race_run <- function(fn, wf, folds, g, ms, ctrl) { - set.seed(23) - memoised(race_call_by_name( - fn, - wf, - folds, - grid = g, - metrics = ms, - control = ctrl - )) -} - -for (design in names(TS_INNER_DESIGNS)) { - spec <- TS_INNER_DESIGNS[[design]] - - for (fn in RACERS) { - test_that( - sprintf( - "%s matches its reference loop on an inner %s design", - fn, - design - ), - { - skip_if_no_race_fixture(fn) - d <- spec$data() - wf <- det_workflow(d) - folds <- spec$build(d) - expect_s3_class( - folds$inner_resamples[[1]]$splits[[1]], - spec$split_class - ) - ms <- ts_metrics() - g <- det_grid() - ctrl <- race_control() - - res <- ts_inner_race_run(fn, wf, folds, g, ms, ctrl) - ref <- memoised(reference_nested_race_loop( - fn, - wf, - folds, - grid = g, - metrics = ms, - seed = 23, - metric_name = "rmse", - control = ctrl - )) - - expect_ts_matches_reference(res, ref) - } - ) - } -} - -for (design in names(TS_INNER_DESIGNS)) { - spec <- TS_INNER_DESIGNS[[design]] - - for (fn in RACERS) { - test_that( - sprintf( - "the %s final fit on an inner %s result matches its reference", - fn, - design - ), - { - skip_if_no_race_fixture(fn) - d <- spec$data() - wf <- det_workflow(d) - folds <- spec$build(d) - ms <- ts_metrics() - g <- det_grid() - ctrl <- race_control() - - res <- ts_inner_race_run(fn, wf, folds, g, ms, ctrl) - set.seed(42) - final <- nested_final_fit(wf, res) - ref <- reference_race_final_fit( - fn, - wf, - d, - grid = g, - metrics = ms, - seed = 42, - metric_name = "rmse", - control = ctrl, - inner_design = spec$inner - ) - expect_ts_final_matches(final, ref, d, split_class = spec$split_class) - } - ) - } -} From 288335fc9ce85647d2d085aeb623ca6f4bb471f9 Mon Sep 17 00:00:00 2001 From: Jeffrey Girard Date: Wed, 21 Oct 2026 22:59:28 -0500 Subject: [PATCH 03/13] M151 T3: the help and NEWS name the tested set after the cut --- NEWS.md | 18 ++++++++--------- R/nested-resamples.R | 20 ++++++++++--------- .../milestones/M151-time-series-tuner-cut.md | 3 ++- man/nested_resamples.Rd | 20 ++++++++++--------- 4 files changed, 33 insertions(+), 28 deletions(-) diff --git a/NEWS.md b/NEWS.md index 4bb40ad4..a96310b7 100644 --- a/NEWS.md +++ b/NEWS.md @@ -290,7 +290,9 @@ `rsample::sliding_index()` or `rsample::sliding_period()` design with an inner `rsample::rolling_origin()` design. `nested_tune_grid()` and `nested_fit_resamples()` are tested on all four, and so is - `nested_final_fit()` for their results. `nested_tune_bayes()`, + `nested_final_fit()` for `nested_fit_resamples()` results. + `nested_final_fit()` for `nested_tune_grid()` results is tested on the + rolling-origin design. `nested_tune_bayes()`, `nested_tune_race_anova()`, `nested_tune_race_win_loss()` and `nested_tune_sim_anneal()` are tested on the rolling-origin design, and so is `nested_final_fit()` for their results, plus the sliding-window @@ -304,15 +306,13 @@ `rsample::sliding_period()` design is tested under `nested_tune_grid()`, and so is `nested_final_fit()` for its results. `nested_resamples()` builds these three designs with the same splits as - `rsample::nested_cv()`. These designs are also tested under + `rsample::nested_cv()`. `nested_workflow_map()` is tested on these three + designs with its default `fn`. `nested_fit_resamples()` is tested to + accept these designs. It checks the inner design but fits nothing on it. `nested_tune_bayes()`, `nested_tune_race_anova()`, - `nested_tune_race_win_loss()` and `nested_tune_sim_anneal()`, and so is - `nested_final_fit()` for the results of those four tuners. - `nested_workflow_map()` is tested on these three designs with its default - `fn`. With `fn` set to each of those four tuners, it is tested on the - sliding-period design alone. `nested_fit_resamples()` is tested to accept - these designs. It checks the inner design but fits nothing on it. Any - other inner design is not tested. + `nested_tune_race_win_loss()` and `nested_tune_sim_anneal()` are not + tested on these designs. They are tested with an inner sliding design on + the sliding-window pair below. Any other inner design is not tested. * An outer `rsample::sliding_window()`, `rsample::sliding_index()` or `rsample::sliding_period()` design with an inner design of one of these diff --git a/R/nested-resamples.R b/R/nested-resamples.R index 6a0248cf..9c0b9a98 100644 --- a/R/nested-resamples.R +++ b/R/nested-resamples.R @@ -65,16 +65,18 @@ #' #' - Outer [rsample::rolling_origin()] or a sliding design, with an inner #' [rsample::rolling_origin()]. [nested_tune_grid()], -#' [nested_fit_resamples()] and their final fits are tested on all four. -#' The four other tuners, their final fits and [nested_workflow_map()] are -#' tested on the rolling-origin outer design. The final fit of -#' [nested_tune_bayes()] is also tested on the sliding-window outer design. -#' - Outer [rsample::rolling_origin()] with an inner sliding design. All five -#' tuners and their final fits are tested on all three. -#' [nested_workflow_map()] is tested on all three with its default `fn`, -#' and with each of the four other tuners on sliding-period alone. +#' [nested_fit_resamples()] and the final fit of [nested_fit_resamples()] +#' are tested on all four. The final fit of [nested_tune_grid()], the four +#' other tuners, their final fits and [nested_workflow_map()] are tested on +#' the rolling-origin outer design. The final fit of [nested_tune_bayes()] +#' is also tested on the sliding-window outer design. +#' - Outer [rsample::rolling_origin()] with an inner sliding design. +#' [nested_tune_grid()] and its final fit are tested on all three. +#' [nested_workflow_map()] is tested on all three with its default `fn`. #' [nested_fit_resamples()] accepts these designs. It checks the inner -#' design but fits nothing on it. +#' design but fits nothing on it. The four other tuners are not tested on +#' these designs. They are tested with an inner sliding design on the +#' sliding-window pair below. #' - A sliding design in both loops, which gives nine pairs. #' [nested_tune_grid()] is tested on all nine. On the sliding-window pair, #' the four other tuners, [nested_workflow_map()] with its default `fn`, diff --git a/cairn/milestones/M151-time-series-tuner-cut.md b/cairn/milestones/M151-time-series-tuner-cut.md index aa5a5dc9..3d6f7989 100644 --- a/cairn/milestones/M151-time-series-tuner-cut.md +++ b/cairn/milestones/M151-time-series-tuner-cut.md @@ -51,7 +51,7 @@ Cut the suite time that the inner-design tuner matrix costs. The help and `NEWS. - [ ] T1: Record the branch point. Run `benchmarks/test-blocks.R`, `benchmarks/coverage-lines.R` and `benchmarks/profile-tests.R 1` on it in the background, with outputs in the scratchpad. - [x] T2: Delete the files and blocks that Scope In names. If no other file calls a fixture helper, delete it. Update `DESCRIPTION` start-first. Check `helper-time-budget.R` for rows on the deleted files. -- [ ] T3: Rewrite the help section and sentences, and run `devtools::document()`. Rewrite the `NEWS.md` bullets. Write each claim against a head test block. +- [x] T3: Rewrite the help section and sentences, and run `devtools::document()`. Rewrite the `NEWS.md` bullets. Write each claim against a head test block. - [ ] T4: Write the D-entry, and widen the inner-designs candidate row. - [ ] T5: Run the three scripts at the head and compare. Write the block comparison, a claim ledger with one row per help or `NEWS.md` sentence, the timing and the coverage results to the Review section. - [ ] T6: Run the `verify` slot. @@ -70,6 +70,7 @@ Cut the suite time that the inner-design tuner matrix costs. The help and `NEWS. - 2026-10-21: lessons harvested: M51 (a CI leg's time moves with the runner, so the timing criteria use local serial runs on one machine). - 2026-10-21: T1 started: branch point 3af50898 checked out as a detached worktree in the scratchpad; test-blocks and coverage-lines run there in the background; the branch-point profile runs alone after them, so its timing has no contention. - 2026-10-21: T2 done: deleted the two inner tuner files, the four map-with-tuner blocks and the three sliding outer final fits; no helper became unused (each is called from another test file); start-first drops the two files; the ledger has no rows on them; the O2 and O3 oracle headers no longer name the deleted tests. The edited files and the start-first guard pass. +- 2026-10-21: T3 done: `?nested_resamples` and the two `NEWS.md` bullets now name the grid final fit on the rolling-origin outer design alone, and say that the four other tuners are not tested under an outer rolling-origin with an inner sliding design; `?nested_tune_grid` needed no change, since it names only grid. `document()` raised `Config/roxygen2/version` to the local 8.1.1, and that line was put back to 8.1.0. All six gating prose sweeps clean. ## Decisions diff --git a/man/nested_resamples.Rd b/man/nested_resamples.Rd index 9e27bd0e..433d5e35 100644 --- a/man/nested_resamples.Rd +++ b/man/nested_resamples.Rd @@ -80,16 +80,18 @@ the functions named with it. \itemize{ \item Outer \code{\link[rsample:rolling_origin]{rsample::rolling_origin()}} or a sliding design, with an inner \code{\link[rsample:rolling_origin]{rsample::rolling_origin()}}. \code{\link[=nested_tune_grid]{nested_tune_grid()}}, -\code{\link[=nested_fit_resamples]{nested_fit_resamples()}} and their final fits are tested on all four. -The four other tuners, their final fits and \code{\link[=nested_workflow_map]{nested_workflow_map()}} are -tested on the rolling-origin outer design. The final fit of -\code{\link[=nested_tune_bayes]{nested_tune_bayes()}} is also tested on the sliding-window outer design. -\item Outer \code{\link[rsample:rolling_origin]{rsample::rolling_origin()}} with an inner sliding design. All five -tuners and their final fits are tested on all three. -\code{\link[=nested_workflow_map]{nested_workflow_map()}} is tested on all three with its default \code{fn}, -and with each of the four other tuners on sliding-period alone. +\code{\link[=nested_fit_resamples]{nested_fit_resamples()}} and the final fit of \code{\link[=nested_fit_resamples]{nested_fit_resamples()}} +are tested on all four. The final fit of \code{\link[=nested_tune_grid]{nested_tune_grid()}}, the four +other tuners, their final fits and \code{\link[=nested_workflow_map]{nested_workflow_map()}} are tested on +the rolling-origin outer design. The final fit of \code{\link[=nested_tune_bayes]{nested_tune_bayes()}} +is also tested on the sliding-window outer design. +\item Outer \code{\link[rsample:rolling_origin]{rsample::rolling_origin()}} with an inner sliding design. +\code{\link[=nested_tune_grid]{nested_tune_grid()}} and its final fit are tested on all three. +\code{\link[=nested_workflow_map]{nested_workflow_map()}} is tested on all three with its default \code{fn}. \code{\link[=nested_fit_resamples]{nested_fit_resamples()}} accepts these designs. It checks the inner -design but fits nothing on it. +design but fits nothing on it. The four other tuners are not tested on +these designs. They are tested with an inner sliding design on the +sliding-window pair below. \item A sliding design in both loops, which gives nine pairs. \code{\link[=nested_tune_grid]{nested_tune_grid()}} is tested on all nine. On the sliding-window pair, the four other tuners, \code{\link[=nested_workflow_map]{nested_workflow_map()}} with its default \code{fn}, From 3e0db968b1292ae2ddfad943c19ab10a14b1e2d2 Mon Sep 17 00:00:00 2001 From: Jeffrey Girard Date: Wed, 21 Oct 2026 23:00:17 -0500 Subject: [PATCH 04/13] M151 T4: D-118 and the widened inner-design row --- cairn/DECISIONS.md | 6 ++++++ cairn/ROADMAP.md | 2 +- cairn/milestones/M151-time-series-tuner-cut.md | 3 ++- 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/cairn/DECISIONS.md b/cairn/DECISIONS.md index dd5ea683..0ffb7317 100644 --- a/cairn/DECISIONS.md +++ b/cairn/DECISIONS.md @@ -2111,6 +2111,12 @@ upstream, either of which would let the file return to a shared blob. **Decision:** the `workflow` and `model_spec` methods order those shared arguments as `tune_bayes()` and `tune_sim_anneal()` do. `event_level` stays after `metrics`, as on the grid methods, and `select` stays last. Names and defaults do not change. Rejected: reordering the `model_spec` methods only, which leaves the two methods of one function in different orders where `test-dots-barrier.R` holds them equal apart from `preprocessor`. **Consequences:** no call changes, because R matches an argument after `...` by its full name only. D-069's argument names, defaults and argument set stand. A later change to the order in tune or finetune fails `test-argument-order.R`, which CRAN skips. Falsified by a caller or tool that reads these formals by position. (Narrows D-069.) +### D-118 (2026-10-21): the four other tuners are tested with an inner sliding design on the sliding-window pair alone, and grid results' final fit on the rolling-origin outer design alone. Supersedes clauses of D-080, D-085, D-112 and D-113 + +**Context:** D-085 claimed the three inner sliding designs under an outer `rolling_origin()` for every orchestrator, because the inner design reaches the tuner where the outer design does not. D-112 and D-113 added the four other tuners' final fits on all three inner designs, and the map with each tuner `fn` on sliding-period. The inner design reaches the tuner as an rset that nestedtune hands to tune or finetune. The code that rebuilds and re-points the inner splits is shared by the five tuners, and it stays tested under grid on the three inner designs and on the nine pairs. D-080 kept the grid final fit on all four outer designs, but the final fit reads the data, not the outer splits (D-075). M151's milestone file records what these tests cost. +**Decision:** the four other tuners are tested with an inner sliding design on the sliding-window / sliding-window pair alone, which D-112 and D-113 already claim. Their final fits with an inner sliding design, and the map with a tuner `fn` on an inner sliding design, are not tested. `nested_final_fit()` for grid results is tested on the rolling-origin outer design, on the three inner sliding designs and on the sliding-window pair. For `nested_fit_resamples()` results it stays tested on all four outer designs. The Bayesian final fit on the sliding-window outer design stays, because it is the test that D-075 rests on (D-080). The help of `nested_resamples()` and `NEWS.md` state the tested set, as D-071 requires. Rejected at M151's plan gate: a wider help claim resting on the shared path, because the help claims only what a test runs. +**Consequences:** the combinations this entry leaves untested join the inner-design candidate row. Falsified by a tuner that fails with an inner sliding design under an outer `rolling_origin()` where it passes on the sliding-window pair, or by a grid final fit on a sliding outer design that differs from its reference where the rolling-origin one matches. (Supersedes D-085's Decision clause claiming the inner sliding designs under the four other tuners, the clauses of D-112 and D-113 claiming those tuners' final fits on the three inner designs and the map with each tuner `fn` on sliding-period, and D-080's clause keeping the grid final fit tested on all four outer designs.) +