Evaluation and model selection¶

The cross-validation and selection helpers use the R estimator's predict(data, targets=..., source=...) contract. They are not drop-in GP posterior selection helpers. Splits must protect entire runs; fitted candidates are compared on common valid support. See tutorials for the workflow.

LeaveOneRunOut¶

from multimodalsrm import LeaveOneRunOut · Source

LeaveOneRunOut()

Hold out a named stimulus run across all participants.

This splitter understands the nested multimodal contract, unlike generic sklearn row splitters. At least two distinct run identifiers are required. Participants without observations in a partition are omitted there.

LeaveOneRunOut.split¶

LeaveOneRunOut.split(data)

LeaveOneRunOut.get_n_splits¶

LeaveOneRunOut.get_n_splits(data)

cross_validate¶

from multimodalsrm import cross_validate · Source

cross_validate(
    estimator,
    data,
    *,
    targets,
    sources=('within', 'across', 'both'),
    cv=None,
    affinity=None,
    evaluation_weights=None,
    n_jobs=1,
)

Clone and fit on training runs, then score explicit targets on test runs.

All compared sources use the intersection of their valid prediction entries and the target observation mask. Target observations may be supplied to predict for convenience, but its exclusion boundary removes them before inference. Fit weights are never reused as evaluation weights. With no evaluation weights, summary metrics equally average available score rows. Unsupported folds raise a clear error; they are not silently discarded. n_jobs parallelizes folds in separate processes (1 by default; -1 uses available CPUs). Inner native restart parallelism is suppressed in workers.

compare_models¶

from multimodalsrm import compare_models · Source

compare_models(models, data, *, targets, sources=('within', 'across', 'both'))

Compare fitted models on native observations and common valid support.

The full requested target union is excluded from every conditional solve. A model with any failed source prediction has undefined scores for every source; successful raw predictions are retained for inspection.

select_model¶

from multimodalsrm import select_model · Source

select_model(
    candidates,
    data,
    *,
    targets,
    source="both",
    metric="r2",
    cv=None,
    affinity=None,
    evaluation_weights=None,
    refit=True,
    n_jobs=1,
)

Select independent clones by strict inner run validation, then refit.

Every positive-weight expected row must have a finite selected metric. Exact ties use candidate names. Finite unfinished optimization remains eligible, with warnings and stopping records separate from score validity. status is ok, no_eligible_candidate, or refit_failed. A failed refit retains the selected name but returns no deployable estimator. n_jobs parallelizes independent folds, preserving common support across candidates within each fold. Inner native restart workers are suppressed.

nested_cross_validate¶

from multimodalsrm import nested_cross_validate · Source

nested_cross_validate(
    candidates,
    data,
    *,
    targets,
    source="both",
    metric="r2",
    sources=('within', 'across', 'both'),
    outer_cv=None,
    inner_cv=None,
    affinity=None,
    evaluation_weights=None,
    n_jobs=1,
)

Select only inside each outer training partition and evaluate fresh runs.

Models, folds and selections share outer-fold positions, including failures. Outer scores retain every expected target/source row; an incomplete metric has an undefined summary rather than a favorable available-case average. n_jobs parallelizes outer folds only; each worker selects/refits using serial native inner folds and restarts. All execution defaults to serial.

temporal_isc¶

from multimodalsrm import temporal_isc · Source

temporal_isc(series)

Per-feature temporal Pearson ISC against the leave-one-participant-out mean.

Use each feature's intersection of valid rows across ALL participants. At least three rows and nonconstant query/reference signals are required. Aggregate Fisher-z means are undefined if any participant is undefined; individual outcomes and support counts are always retained. This is temporal ISC, not correlation over features at each time (pattern ISC).

time_segment_matching¶

from multimodalsrm import time_segment_matching · Source

time_segment_matching(series, *, window_size, centering="global")

Match each participant's time windows to the other participants' mean.

Query times must be regularly spaced. Fully valid windows are flattened over time and features, then compared with Pearson correlation. Competing windows that partially overlap the query are excluded; its true match stays. Ties within 1e-12 receive fractional credit. Chance is 1/candidate count per query, not 1/total windows. Constant vectors or fewer than two candidates make a query undefined. An undefined query makes aggregate accuracy None. No windows cross masked rows; do not concatenate independent runs upstream.

centering="feature" is a companion score: subtract each feature's within-window temporal mean before the normalized dot product. It is invariant to a common orthogonal feature rotation (up to roundoff), but not arbitrary feature scaling. The historical scalar-centering metric remains the default; the companion has a distinct metric identifier.