moabb.evaluations.CrossSubjectEvaluation#

class moabb.evaluations.CrossSubjectEvaluation(*args, cs_mode=CrossSubjectMode.TRAIN, **kwargs)[source]#

Cross-subject evaluation performance.

Evaluate performance of the pipeline trained on all subjects but one, concatenating sessions.

Parameters:

paradigm (BaseParadigm) – The paradigm to use.
datasets (list of BaseDataset) – The list of dataset to run the evaluation. If none, the list of compatible dataset will be retrieved from the paradigm instance.
random_state (int or None) – If not None, can guarantee same seed for shuffling examples. Defaults to None.
n_jobs (int) – Number of jobs for fitting of pipeline. Defaults to 1.
overwrite (bool) – If true, overwrite the results. Defaults to False.
error_score (str or float) – Value to assign to the score if an error occurs in estimator fitting. If set to 'raise', the error is raised. Defaults to "raise".
suffix (str) – Suffix for the results file.
hdf5_path (str) – Specific path for storing the results and models.
additional_columns (None) – Adding information to results.
return_epochs (bool) – Use MNE epoch to train pipelines. Defaults to False.
return_raws (bool) – Use MNE raw to train pipelines. Defaults to False.
mne_labels (bool) – If returning MNE epoch, use original dataset label if True. Defaults to False.
save_model (bool) – Save model after training, for each fold of cross-validation if needed. Defaults to False.
cache_config (CacheConfig or None) – Configuration for caching of datasets. See moabb.datasets.base.CacheConfig for details. Defaults to None.
n_splits (int or None) – Number of splits for cross-validation. If None, the number of splits is equal to the number of subjects. Defaults to None.
cv_class (type or None) – Cross-validation strategy used to hold out subjects (e.g. LeaveOneGroupOut, GroupShuffleSplit, GroupKFold). Defaults to None (LeaveOneGroupOut, or GroupKFold when n_splits is set).
cv_kwargs (dict) – Keyword arguments for cv_class. calibration_size (float in [0, 1], default 0.0) enables transfer learning: when > 0 each fold becomes (train, calibration, test). The fraction is taken within every held-out subject/session pair so each remains scorable, and the calibration slice is routed (raw) to pipeline steps via set_fit_request. With calibration_labeled=False, only X_target_unlabeled may be routed. With calibration_labeled=True, X_target_labeled and y_target_labeled may be routed. Labeled calibration is only allowed with calibration_size <= 0.5.
cs_mode (CrossSubjectMode or str, default=CrossSubjectMode.TRAIN) – Named cross-subject protocol preset. By default, this is the standard train-only cross-subject evaluation with no target calibration. The TRAIN_TRIALWISE mode additionally enforces one-trial-at-a-time prediction during scoring and supports the built-in "accuracy" and "roc_auc" metrics. Cannot be combined with manual calibration_size or calibration_labeled in cv_kwargs, except for the default TRAIN mode.

Notes

Added in version 1.1.0: Add save_model, cache_config and n_splits parameters

evaluate(dataset: BaseDataset, pipelines: dict, param_grid: dict | None, process_pipeline, postprocess_pipeline=None)[source]#

Evaluate results on a single dataset.

This method return a generator. each results item is a dict with the following conversion:

res = {'time': Duration of the training ,
       'dataset': dataset id,
       'subject': subject id,
       'session': session id,
       'score': score,
       'n_samples': number of training examples,
       'n_channels': number of channel,
       'pipeline': pipeline name}

is_valid(dataset: BaseDataset) → bool[source]#

Verify the dataset is compatible with evaluation.

This method is called to verify dataset given in the constructor are compatible with the evaluation context.

This method should return false if the dataset does not match the evaluation. This is for example the case if the dataset does not contain enough session for a cross-session eval.

Parameters:: dataset (BaseDataset) – The dataset to verify.

Examples using `moabb.evaluations.CrossSubjectEvaluation`#

Cross-Subject SSVEP

Cross-subject transfer with Riemannian alignment

Trialwise cross-subject subject-prototype classification

moabb.evaluations.CrossSubjectEvaluation#

Examples using moabb.evaluations.CrossSubjectEvaluation#

Examples using `moabb.evaluations.CrossSubjectEvaluation`#