Examples of how to use MOABB to benchmark pipelines.#

Benchmarking with MOABB#

This example shows how to use MOABB to benchmark a set of pipelines on all available datasets. For this example, we will use only one dataset to keep the computation time low, but this benchmark is designed to easily scale to many datasets.

# Authors: Sylvain Chevallier <sylvain.chevallier@universite-paris-saclay.fr>
#
# License: BSD (3-clause)

import matplotlib.pyplot as plt

from moabb import benchmark, set_log_level
from moabb.analysis.plotting import score_plot
from moabb.paradigms import LeftRightImagery


set_log_level("info")

Loading the pipelines#

The ML pipelines used in benchmark are defined in YAML files, following a simple format. It simplifies sharing and reusing pipelines across benchmarks, reproducing state-of-the-art results.

MOABB comes with complete list of pipelines that cover most of the successful approaches in the literature. You can find them in the pipelines folder. For this example, we will use a folder with only 2 pipelines, to keep the computation time low.

This is an example of a pipeline defined in YAML, defining on which paradigms it can be used, the original publication, and the steps to perform using a scikit-learn API. In this case, a CSP + SVM pipeline, the covariance are estimated to compute a CSP filter and then a linear SVM is trained on the CSP filtered signals.

with open("sample_pipelines/CSP_SVM.yml", "r") as f:
    lines = f.readlines()
    for line in lines:
        print(line, end="")
name: CSP + SVM
paradigms:
  - LeftRightImagery

citations:
  - https://doi.org/10.1007/BF01129656
  - https://doi.org/10.1109/MSP.2008.4408441

pipeline:
  - name: Covariances
    from: pyriemann.estimation
    parameters:
      estimator: oas

  - name: CSP
    from: pyriemann.spatialfilters
    parameters:
      nfilter: 6

  - name: SVC
    from: sklearn.svm
    parameters:
      kernel: "linear"

The sample_pipelines folder contains a second pipeline, a logistic regression performed in the tangent space using Riemannian geometry.

Selecting the datasets (optional)#

If you want to limit your benchmark on a subset of datasets, you can use the include_datasets and exclude_datasets arguments. You will need either to provide the dataset’s object, or a the dataset’s code. To get the list of available dataset’s code for a given paradigm, you can use the following command:

/home/runner/work/moabb/moabb/moabb/datasets/fake.py:93: RuntimeWarning: Setting non-standard config type: "MNE_DATASETS_FAKEDATASET-IMAGERY-10-2--60-60--120-120--FAKE1-FAKE2-FAKE3--C3-CZ-C4_PATH"
  set_config(key, temp_dir)
/home/runner/work/moabb/moabb/moabb/datasets/fake.py:93: RuntimeWarning: Setting non-standard config type: "MNE_DATASETS_FAKEVIRTUALREALITYDATASET-P300-21-1--60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60--120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120--TARGET-NONTARGET--C3-CZ-C4_PATH"
  set_config(key, temp_dir)
BNCI2014-001
BNCI2014-004
Cho2017
GrosseWentrup2009
Lee2019-MI
PhysionetMotorImagery
Schirrmeister2017
Shin2017A
Weibo2014
Zhou2016

In this example, we will use only the last dataset, ‘Zhou 2016’.

Running the benchmark#

The benchmark is run using the benchmark function. You need to specify the folder containing the pipelines to use, the kind of evaluation and the paradigm to use. By default, the benchmark will use all available datasets for all paradigms listed in the pipelines. You could restrict to specific evaluation and paradigm using the evaluations and paradigms arguments.

To save computation time, the results are cached. If you want to re-run the benchmark, you can set the overwrite argument to True.

It is possible to indicate the folder to cache the results and the one to save the analysis & figures. By default, the results are saved in the results folder, and the analysis & figures are saved in the benchmark folder.

results = benchmark(
    pipelines="./sample_pipelines/",
    evaluations=["WithinSession"],
    paradigms=["LeftRightImagery"],
    include_datasets=["Zhou2016"],
    results="./results/",
    overwrite=False,
    plot=False,
    output="./benchmark/",
)
2024-03-18 12:47:56,264 INFO MainThread moabb.evaluations.base Processing dataset: Zhou2016

Zhou2016-WithinSession:   0%|          | 0/4 [00:00<?, ?it/s]MNE_DATA is not already configured. It will be set to default location in the home directory - /home/runner/mne_data
All datasets will be downloaded to this location, if anything is already downloaded, please move manually to this location
/home/runner/work/moabb/moabb/moabb/datasets/download.py:55: RuntimeWarning: Setting non-standard config type: "MNE_DATASETS_ZHOU_PATH"
  set_config(key, get_config("MNE_DATA"))


  0%|                                               | 0.00/156M [00:00<?, ?B/s]

  0%|                                       | 52.2k/156M [00:00<08:37, 302kB/s]

  0%|                                        | 209k/156M [00:00<03:57, 655kB/s]

  1%|▏                                      | 853k/156M [00:00<01:15, 2.05MB/s]

  2%|▋                                     | 2.67M/156M [00:00<00:23, 6.46MB/s]

  3%|█▎                                    | 5.41M/156M [00:00<00:12, 12.3MB/s]

  5%|█▊                                    | 7.61M/156M [00:00<00:11, 13.2MB/s]

  7%|██▌                                   | 10.6M/156M [00:00<00:08, 17.5MB/s]

  9%|███▎                                  | 13.5M/156M [00:01<00:06, 20.9MB/s]

 10%|███▉                                  | 16.0M/156M [00:01<00:07, 19.4MB/s]

 12%|████▋                                 | 19.1M/156M [00:01<00:06, 22.3MB/s]

 14%|█████▍                                | 22.4M/156M [00:01<00:05, 25.4MB/s]

 16%|██████                                | 25.1M/156M [00:01<00:05, 22.6MB/s]

 18%|██████▉                               | 28.4M/156M [00:01<00:05, 25.2MB/s]

 20%|███████▋                              | 31.6M/156M [00:01<00:04, 27.1MB/s]

 22%|████████▍                             | 34.5M/156M [00:01<00:05, 23.9MB/s]

 24%|█████████▏                            | 37.7M/156M [00:02<00:04, 26.2MB/s]

 26%|█████████▉                            | 40.8M/156M [00:02<00:04, 23.8MB/s]

 28%|██████████▋                           | 44.0M/156M [00:02<00:04, 25.9MB/s]

 30%|███████████▎                          | 46.7M/156M [00:02<00:04, 26.0MB/s]

 32%|████████████                          | 49.5M/156M [00:02<00:04, 23.7MB/s]

 34%|████████████▉                         | 52.9M/156M [00:02<00:03, 26.4MB/s]

 36%|█████████████▋                        | 56.1M/156M [00:02<00:03, 27.8MB/s]

 38%|██████████████▎                       | 59.0M/156M [00:02<00:03, 24.7MB/s]

 40%|███████████████▏                      | 62.3M/156M [00:02<00:03, 26.8MB/s]

 42%|███████████████▉                      | 65.6M/156M [00:03<00:03, 28.6MB/s]

 44%|████████████████▋                     | 68.6M/156M [00:03<00:03, 25.4MB/s]

 46%|█████████████████▌                    | 72.0M/156M [00:03<00:03, 27.5MB/s]

 48%|██████████████████▎                   | 75.3M/156M [00:03<00:02, 29.2MB/s]

 50%|███████████████████                   | 78.4M/156M [00:03<00:03, 25.9MB/s]

 52%|███████████████████▊                  | 81.7M/156M [00:03<00:02, 27.8MB/s]

 55%|████████████████████▋                 | 85.1M/156M [00:03<00:02, 25.8MB/s]

 57%|█████████████████████▌                | 88.5M/156M [00:03<00:02, 27.8MB/s]

 59%|██████████████████████▍               | 92.0M/156M [00:04<00:02, 29.6MB/s]

 61%|███████████████████████               | 95.0M/156M [00:04<00:02, 26.3MB/s]

 63%|████████████████████████              | 99.0M/156M [00:04<00:01, 29.7MB/s]

 66%|█████████████████████████▋             | 103M/156M [00:04<00:01, 31.7MB/s]

 68%|██████████████████████████▌            | 106M/156M [00:04<00:01, 33.1MB/s]

 70%|███████████████████████████▍           | 110M/156M [00:04<00:01, 30.3MB/s]

 73%|████████████████████████████▎          | 113M/156M [00:04<00:01, 31.9MB/s]

 75%|█████████████████████████████▎         | 117M/156M [00:04<00:01, 33.4MB/s]

 77%|██████████████████████████████▏        | 121M/156M [00:04<00:01, 33.9MB/s]

 79%|███████████████████████████████        | 124M/156M [00:05<00:00, 32.4MB/s]

 82%|███████████████████████████████▊       | 127M/156M [00:05<00:00, 32.1MB/s]

 84%|████████████████████████████████▊      | 131M/156M [00:05<00:00, 33.8MB/s]

 86%|█████████████████████████████████▋     | 135M/156M [00:05<00:00, 34.2MB/s]

 89%|██████████████████████████████████▌    | 138M/156M [00:05<00:00, 34.5MB/s]

 91%|███████████████████████████████████▍   | 142M/156M [00:05<00:00, 33.3MB/s]

 93%|████████████████████████████████████▎  | 145M/156M [00:05<00:00, 33.1MB/s]

 95%|█████████████████████████████████████▏ | 149M/156M [00:05<00:00, 33.6MB/s]

 98%|██████████████████████████████████████ | 152M/156M [00:05<00:00, 34.2MB/s]

100%|██████████████████████████████████████▉| 156M/156M [00:05<00:00, 34.6MB/s]

  0%|                                               | 0.00/156M [00:00<?, ?B/s]
100%|████████████████████████████████████████| 156M/156M [00:00<00:00, 704GB/s]
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning:   Could not parse meas date from the header. Setting to None.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning:   Could not parse meas date from the header. Setting to None.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning:   Could not parse meas date from the header. Setting to None.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning:   Could not parse meas date from the header. Setting to None.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning:   Could not parse meas date from the header. Setting to None.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning:   Could not parse meas date from the header. Setting to None.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  60 events (all good), 0 – 5 s, baseline off, ~8.0 MB, data loaded,
 'left_hand': 30
 'right_hand': 30>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  59 events (all good), 0 – 5 s, baseline off, ~7.9 MB, data loaded,
 'left_hand': 30
 'right_hand': 29>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
2024-03-18 12:48:08,739 INFO MainThread moabb.evaluations.base Tangent Space LR | Zhou2016 | 1 | 0: Score 0.909
2024-03-18 12:48:09,190 INFO MainThread moabb.evaluations.base CSP + SVM | Zhou2016 | 1 | 0: Score 0.860
2024-03-18 12:48:09,644 INFO MainThread moabb.evaluations.base Tangent Space LR | Zhou2016 | 1 | 1: Score 0.926
2024-03-18 12:48:10,037 INFO MainThread moabb.evaluations.base CSP + SVM | Zhou2016 | 1 | 1: Score 0.924
2024-03-18 12:48:10,488 INFO MainThread moabb.evaluations.base Tangent Space LR | Zhou2016 | 1 | 2: Score 0.964
2024-03-18 12:48:10,885 INFO MainThread moabb.evaluations.base CSP + SVM | Zhou2016 | 1 | 2: Score 0.954

Zhou2016-WithinSession:  25%|##5       | 1/4 [00:14<00:44, 14.75s/it]/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning:   Could not parse meas date from the header. Setting to None.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning:   Could not parse meas date from the header. Setting to None.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning:   Could not parse meas date from the header. Setting to None.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning:   Could not parse meas date from the header. Setting to None.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  40 events (all good), 0 – 5 s, baseline off, ~5.4 MB, data loaded,
 'left_hand': 20
 'right_hand': 20>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
2024-03-18 12:48:13,244 INFO MainThread moabb.evaluations.base Tangent Space LR | Zhou2016 | 2 | 0: Score 0.910
2024-03-18 12:48:13,641 INFO MainThread moabb.evaluations.base CSP + SVM | Zhou2016 | 2 | 0: Score 0.916
2024-03-18 12:48:14,078 INFO MainThread moabb.evaluations.base Tangent Space LR | Zhou2016 | 2 | 1: Score 0.765
2024-03-18 12:48:14,457 INFO MainThread moabb.evaluations.base CSP + SVM | Zhou2016 | 2 | 1: Score 0.753
2024-03-18 12:48:14,898 INFO MainThread moabb.evaluations.base Tangent Space LR | Zhou2016 | 2 | 2: Score 0.904
2024-03-18 12:48:15,297 INFO MainThread moabb.evaluations.base CSP + SVM | Zhou2016 | 2 | 2: Score 0.906

Zhou2016-WithinSession:  50%|#####     | 2/4 [00:19<00:17,  8.67s/it]/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning:   Could not parse meas date from the header. Setting to None.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning:   Could not parse meas date from the header. Setting to None.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
2024-03-18 12:48:17,770 INFO MainThread moabb.evaluations.base Tangent Space LR | Zhou2016 | 3 | 0: Score 0.994
2024-03-18 12:48:18,169 INFO MainThread moabb.evaluations.base CSP + SVM | Zhou2016 | 3 | 0: Score 0.972
2024-03-18 12:48:18,623 INFO MainThread moabb.evaluations.base Tangent Space LR | Zhou2016 | 3 | 1: Score 0.990
2024-03-18 12:48:19,011 INFO MainThread moabb.evaluations.base CSP + SVM | Zhou2016 | 3 | 1: Score 0.980
2024-03-18 12:48:19,454 INFO MainThread moabb.evaluations.base Tangent Space LR | Zhou2016 | 3 | 2: Score 0.992
2024-03-18 12:48:19,843 INFO MainThread moabb.evaluations.base CSP + SVM | Zhou2016 | 3 | 2: Score 0.990

Zhou2016-WithinSession:  75%|#######5  | 3/4 [00:23<00:06,  6.79s/it]/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning:   Could not parse meas date from the header. Setting to None.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning:   Could not parse meas date from the header. Setting to None.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/Zhou2016.py:111: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  raw = read_raw_cnt(fname, preload=True, eog=["VEOU", "VEOL"])
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  40 events (all good), 0 – 5 s, baseline off, ~5.4 MB, data loaded,
 'left_hand': 20
 'right_hand': 20>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  50 events (all good), 0 – 5 s, baseline off, ~6.7 MB, data loaded,
 'left_hand': 25
 'right_hand': 25>
  warn(f"warnEpochs {epochs}")
2024-03-18 12:48:22,175 INFO MainThread moabb.evaluations.base Tangent Space LR | Zhou2016 | 4 | 0: Score 0.995
2024-03-18 12:48:22,557 INFO MainThread moabb.evaluations.base CSP + SVM | Zhou2016 | 4 | 0: Score 0.993
2024-03-18 12:48:23,008 INFO MainThread moabb.evaluations.base Tangent Space LR | Zhou2016 | 4 | 1: Score 0.972
2024-03-18 12:48:23,409 INFO MainThread moabb.evaluations.base CSP + SVM | Zhou2016 | 4 | 1: Score 0.972
2024-03-18 12:48:23,860 INFO MainThread moabb.evaluations.base Tangent Space LR | Zhou2016 | 4 | 2: Score 0.978
2024-03-18 12:48:24,257 INFO MainThread moabb.evaluations.base CSP + SVM | Zhou2016 | 4 | 2: Score 0.968

Zhou2016-WithinSession: 100%|##########| 4/4 [00:28<00:00,  5.85s/it]
Zhou2016-WithinSession: 100%|##########| 4/4 [00:28<00:00,  7.03s/it]
    dataset     evaluation          pipeline  avg score
0  Zhou2016  WithinSession         CSP + SVM   0.932315
1  Zhou2016  WithinSession  Tangent Space LR   0.941601

Benchmark prints a summary of the results. Detailed results are saved in a pandas dataframe, and can be used to generate figures. The analysis & figures are saved in the benchmark folder.

score_plot(results)
plt.show()
Scores per dataset and algorithm
2024-03-18 12:48:24,752 WARNING MainThread moabb.analysis.plotting Dataset names are too similar, turning off name shortening
/home/runner/work/moabb/moabb/moabb/analysis/plotting.py:70: UserWarning: The palette list has more values (6) than needed (2), which may not be intended.
  sea.stripplot(

Total running time of the script: ( 0 minutes 29.390 seconds)

Estimated memory usage: 326 MB

Gallery generated by Sphinx-Gallery