moabb.analysis.meta_analysis.compute_pvals_corrected_ttest#

moabb.analysis.meta_analysis.compute_pvals_corrected_ttest(df, n_train, n_test, order=None)[source]#

Compute the Nadeau & Bengio corrected resampled t-test.

Returns a kxk matrix of one-tailed p-values comparing each pair of pipelines with the corrected resampled t-test of Nadeau & Bengio [1]. Use this test when the rows of df are scores of overlapping cross-validation resamples (e.g. the folds of a within-session (repeated) k-fold evaluation): the folds share training examples, so a standard paired t-test underestimates the variance and is overconfident. The correction inflates the variance by \(n_2 / n_1\), the test/train ratio of each split.

For k-fold cross-validation repeated r times on m examples, use n_test = m // k and n_train = m - n_test and pass the n = r * k per-fold scores as rows.

Parameters:

  • df (pandas.DataFrame) – Scores of each cross-validation resample; samples (resamples) are index, columns are pipelines, and values are scores.

  • n_train (int) – Number of training examples in each resample.

  • n_test (int) – Number of testing examples in each resample.

  • order (list of length (n_pipelines)) – Names corresponding to df columns.

Returns:

pvals – One-tailed p-values; pvals[i, j] is small when pipeline i scores significantly higher than pipeline j.

Return type:

ndarray of shape (n_pipelines, n_pipelines)

References

[1]

Nadeau, C., & Bengio, Y. (1999). Inference for the generalization error. Advances in Neural Information Processing Systems 12; extended version in Machine Learning, 52, 239-281 (2003). https://doi.org/10.1023/A:1024068626366

Examples using moabb.analysis.meta_analysis.compute_pvals_corrected_ttest#

Statistical Analysis and Chance Level Assessment

Statistical Analysis and Chance Level Assessment