Cross-Session on Multiple Datasets

This example shows how to perform a cross-session analysis on two MI datasets using a CSP+LDA pipeline

The cross session evaluation context will evaluate performance using a leave one session out cross-validation. For each session in the dataset, a model is trained on every other session and performance are evaluated on the current session.

# Authors: Sylvain Chevallier <sylvain.chevallier@uvsq.fr>
#
# License: BSD (3-clause)

import warnings

import matplotlib.pyplot as plt
import seaborn as sns
from mne.decoding import CSP
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.pipeline import make_pipeline

import moabb
from moabb.datasets import BNCI2014001, Zhou2016
from moabb.evaluations import CrossSessionEvaluation
from moabb.paradigms import LeftRightImagery


warnings.simplefilter(action="ignore", category=FutureWarning)
warnings.simplefilter(action="ignore", category=RuntimeWarning)
moabb.set_log_level("info")

Out:

/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/seaborn/cm.py:1582: UserWarning: Trying to register the cmap 'rocket' which already exists.
  mpl_cm.register_cmap(_name, _cmap)
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/seaborn/cm.py:1583: UserWarning: Trying to register the cmap 'rocket_r' which already exists.
  mpl_cm.register_cmap(_name + "_r", _cmap_r)
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/seaborn/cm.py:1582: UserWarning: Trying to register the cmap 'mako' which already exists.
  mpl_cm.register_cmap(_name, _cmap)
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/seaborn/cm.py:1583: UserWarning: Trying to register the cmap 'mako_r' which already exists.
  mpl_cm.register_cmap(_name + "_r", _cmap_r)
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/seaborn/cm.py:1582: UserWarning: Trying to register the cmap 'icefire' which already exists.
  mpl_cm.register_cmap(_name, _cmap)
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/seaborn/cm.py:1583: UserWarning: Trying to register the cmap 'icefire_r' which already exists.
  mpl_cm.register_cmap(_name + "_r", _cmap_r)
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/seaborn/cm.py:1582: UserWarning: Trying to register the cmap 'vlag' which already exists.
  mpl_cm.register_cmap(_name, _cmap)
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/seaborn/cm.py:1583: UserWarning: Trying to register the cmap 'vlag_r' which already exists.
  mpl_cm.register_cmap(_name + "_r", _cmap_r)
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/seaborn/cm.py:1582: UserWarning: Trying to register the cmap 'flare' which already exists.
  mpl_cm.register_cmap(_name, _cmap)
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/seaborn/cm.py:1583: UserWarning: Trying to register the cmap 'flare_r' which already exists.
  mpl_cm.register_cmap(_name + "_r", _cmap_r)
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/seaborn/cm.py:1582: UserWarning: Trying to register the cmap 'crest' which already exists.
  mpl_cm.register_cmap(_name, _cmap)
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/seaborn/cm.py:1583: UserWarning: Trying to register the cmap 'crest_r' which already exists.
  mpl_cm.register_cmap(_name + "_r", _cmap_r)

Loading Dataset

Load 2 subjects of BNCI 2014-004 and Zhou2016 datasets, with 2 session each

subj = [1, 2]
datasets = [Zhou2016(), BNCI2014001()]
for d in datasets:
    d.subject_list = subj

Choose Paradigm

We select the paradigm MI, applying a bandpass filter (8-35 Hz) on the data and we will keep only left- and right-hand motor imagery

paradigm = LeftRightImagery(fmin=8, fmax=35)

Create Pipelines

Use the Common Spatial Patterns with 8 components and a Linear Discriminant Analysis classifier.

pipeline = {}
pipeline["CSP+LDA"] = make_pipeline(CSP(n_components=8), LDA())

Get Data (optional)

To get access to the EEG signals downloaded from the dataset, you could use dataset.get_data(subjects=[subject_id]) to obtain the EEG under an MNE format, stored in a dictionary of sessions and runs. Otherwise, paradigm.get_data(dataset=dataset, subjects=[subject_id]) allows to obtain the EEG data in sklearn format, the labels and the meta information. The data are preprocessed according to the paradigm requirements.

# X_all, labels_all, meta_all = [], [], []
# for d in datasets:
#     # sessions = d.get_data(subjects=[2])
#     X, labels, meta = paradigm.get_data(dataset=d, subjects=[2])
#     X_all.append(X)
#     labels_all.append(labels)
#     meta_all.append(meta)

Evaluation

The evaluation will return a DataFrame containing a single AUC score for each subject / session of the dataset, and for each pipeline.

overwrite = True  # set to True if we want to overwrite cached results

evaluation = CrossSessionEvaluation(
    paradigm=paradigm, datasets=datasets, suffix="examples", overwrite=overwrite
)
results = evaluation.process(pipeline)

print(results.head())
results.replace(["session_E", "session_T"], ["session_0", "session_1"], inplace=True)

Out:

Zhou 2016-CrossSession:   0%|          | 0/2 [00:00<?, ?it/s]

  0%|                                               | 0.00/156M [00:00<?, ?B/s]

  0%|                                       | 54.3k/156M [00:00<09:08, 285kB/s]

  0%|                                        | 147k/156M [00:00<06:29, 401kB/s]

  0%|                                       | 446k/156M [00:00<02:11, 1.19MB/s]

  1%|▏                                      | 831k/156M [00:00<01:17, 1.99MB/s]

  1%|▎                                     | 1.11M/156M [00:00<01:09, 2.24MB/s]

  1%|▌                                     | 2.30M/156M [00:00<00:29, 5.13MB/s]

  3%|█                                     | 4.37M/156M [00:00<00:15, 9.82MB/s]

  4%|█▌                                    | 6.49M/156M [00:00<00:11, 13.2MB/s]

  5%|██                                    | 8.57M/156M [00:01<00:09, 15.5MB/s]

  7%|██▌                                   | 10.7M/156M [00:01<00:08, 17.3MB/s]

  8%|███▏                                  | 12.9M/156M [00:01<00:07, 18.8MB/s]

 10%|███▋                                  | 15.2M/156M [00:01<00:07, 19.8MB/s]

 11%|████▏                                 | 17.4M/156M [00:01<00:06, 20.6MB/s]

 13%|████▊                                 | 19.7M/156M [00:01<00:06, 21.2MB/s]

 14%|█████▎                                | 22.1M/156M [00:01<00:06, 22.0MB/s]

 16%|█████▉                                | 24.3M/156M [00:01<00:06, 21.9MB/s]

 17%|██████▍                               | 26.5M/156M [00:01<00:07, 18.5MB/s]

 18%|███████                               | 28.8M/156M [00:02<00:06, 19.8MB/s]

 20%|███████▌                              | 30.9M/156M [00:02<00:06, 20.0MB/s]

 21%|████████                              | 33.2M/156M [00:02<00:05, 20.9MB/s]

 23%|████████▋                             | 35.5M/156M [00:02<00:05, 21.3MB/s]

 25%|█████████▍                            | 38.7M/156M [00:02<00:04, 23.6MB/s]

 27%|██████████▏                           | 41.7M/156M [00:02<00:04, 25.4MB/s]

 29%|██████████▊                           | 44.6M/156M [00:02<00:04, 26.3MB/s]

 30%|███████████▍                          | 47.2M/156M [00:02<00:04, 26.3MB/s]

 32%|████████████▏                         | 50.0M/156M [00:02<00:03, 26.9MB/s]

 34%|████████████▉                         | 53.1M/156M [00:02<00:03, 28.0MB/s]

 36%|█████████████▋                        | 56.1M/156M [00:03<00:03, 28.5MB/s]

 38%|██████████████▎                       | 59.0M/156M [00:03<00:03, 28.7MB/s]

 40%|███████████████                       | 61.9M/156M [00:03<00:03, 28.8MB/s]

 42%|███████████████▊                      | 64.9M/156M [00:03<00:03, 26.5MB/s]

 44%|████████████████▌                     | 68.0M/156M [00:03<00:03, 27.8MB/s]

 45%|█████████████████▏                    | 70.9M/156M [00:03<00:03, 28.1MB/s]

 47%|█████████████████▉                    | 73.9M/156M [00:03<00:02, 28.6MB/s]

 49%|██████████████████▋                   | 76.9M/156M [00:03<00:02, 29.2MB/s]

 51%|███████████████████▍                  | 80.1M/156M [00:03<00:02, 29.2MB/s]

 53%|████████████████████▎                 | 83.2M/156M [00:04<00:02, 29.9MB/s]

 55%|████████████████████▉                 | 86.2M/156M [00:04<00:02, 29.5MB/s]

 57%|█████████████████████▊                | 89.4M/156M [00:04<00:02, 30.1MB/s]

 59%|██████████████████████▌               | 92.5M/156M [00:04<00:02, 30.3MB/s]

 61%|███████████████████████▏              | 95.6M/156M [00:04<00:01, 30.4MB/s]

 63%|████████████████████████              | 98.7M/156M [00:04<00:01, 30.8MB/s]

 65%|█████████████████████████▍             | 102M/156M [00:04<00:01, 30.9MB/s]

 67%|██████████████████████████▏            | 105M/156M [00:04<00:01, 31.0MB/s]

 69%|██████████████████████████▉            | 108M/156M [00:04<00:01, 31.1MB/s]

 71%|███████████████████████████▊           | 111M/156M [00:04<00:01, 31.3MB/s]

 73%|████████████████████████████▌          | 114M/156M [00:05<00:01, 31.0MB/s]

 75%|█████████████████████████████▎         | 117M/156M [00:05<00:01, 31.0MB/s]

 77%|██████████████████████████████         | 121M/156M [00:05<00:01, 30.8MB/s]

 79%|██████████████████████████████▉        | 124M/156M [00:05<00:01, 28.7MB/s]

 81%|███████████████████████████████▋       | 127M/156M [00:05<00:00, 29.6MB/s]

 83%|████████████████████████████████▍      | 130M/156M [00:05<00:00, 29.8MB/s]

 85%|█████████████████████████████████▏     | 133M/156M [00:05<00:00, 29.9MB/s]

 87%|█████████████████████████████████▉     | 136M/156M [00:05<00:00, 30.3MB/s]

 89%|██████████████████████████████████▋    | 139M/156M [00:05<00:00, 29.8MB/s]

 91%|███████████████████████████████████▌   | 142M/156M [00:05<00:00, 30.2MB/s]

 93%|████████████████████████████████████▎  | 145M/156M [00:06<00:00, 30.7MB/s]

 95%|█████████████████████████████████████  | 149M/156M [00:06<00:00, 30.6MB/s]

 97%|█████████████████████████████████████▉ | 152M/156M [00:06<00:00, 30.9MB/s]

 99%|██████████████████████████████████████▋| 155M/156M [00:06<00:00, 30.5MB/s]

  0%|                                               | 0.00/156M [00:00<?, ?B/s]
100%|████████████████████████████████████████| 156M/156M [00:00<00:00, 191GB/s]
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):

Zhou 2016-CrossSession:  50%|#####     | 1/2 [00:14<00:14, 14.78s/it]/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):

Zhou 2016-CrossSession: 100%|##########| 2/2 [00:18<00:00,  8.02s/it]
Zhou 2016-CrossSession: 100%|##########| 2/2 [00:18<00:00,  9.03s/it]

001-2014-CrossSession:   0%|          | 0/2 [00:00<?, ?it/s]/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):

001-2014-CrossSession:  50%|#####     | 1/2 [00:04<00:04,  4.01s/it]/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/mne/utils/check.py:92: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LooseVersion(library.__version__) < LooseVersion(min_version):

001-2014-CrossSession: 100%|##########| 2/2 [00:08<00:00,  4.03s/it]
001-2014-CrossSession: 100%|##########| 2/2 [00:08<00:00,  4.02s/it]
      score      time  samples  ... n_sessions    dataset  pipeline
0  0.956597  0.284326    144.0  ...          2   001-2014   CSP+LDA
1  0.930941  0.414589    144.0  ...          2   001-2014   CSP+LDA
2  0.649691  0.285467    144.0  ...          2   001-2014   CSP+LDA
3  0.526813  0.346710    144.0  ...          2   001-2014   CSP+LDA
4  0.851412  0.342238    200.0  ...          3  Zhou 2016   CSP+LDA

[5 rows x 9 columns]

Plot Results

Here we plot the results, indicating the score for each session and subject

sns.catplot(
    data=results,
    x="session",
    y="score",
    hue="subject",
    col="dataset",
    kind="bar",
    palette="viridis",
)
plt.show()
dataset = 001-2014, dataset = Zhou 2016

Total running time of the script: ( 0 minutes 26.511 seconds)

Gallery generated by Sphinx-Gallery