Note
Click here to download the full example code
Benchmarking on MOABB with Tensorflow deep net architectures#
This example shows how to use MOABB to benchmark a set of Deep Learning pipeline (Tensorflow) on all available datasets. For this example, we will use only one dataset to keep the computation time low, but this benchmark is designed to easily scale to many datasets.
# Authors: Igor Carrara <igor.carrara@inria.fr>
#
# License: BSD (3-clause)
import os
import matplotlib.pyplot as plt
import tensorflow as tf
from absl.logging import ERROR, set_verbosity
from tensorflow import keras
from moabb import benchmark, set_log_level
from moabb.analysis.plotting import score_plot
from moabb.datasets import BNCI2014_001
from moabb.utils import setup_seed
set_log_level("info")
# Avoid output Warning
set_verbosity(ERROR)
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
# Print Information Tensorflow
print(f"Tensorflow Version: {tf.__version__}")
print(f"Keras Version: {keras.__version__}")
CPU = len(tf.config.list_physical_devices("CPU")) > 0
print("CPU is", "AVAILABLE" if CPU else "NOT AVAILABLE")
GPU = len(tf.config.list_physical_devices("GPU")) > 0
print("GPU is", "AVAILABLE" if GPU else "NOT AVAILABLE")
Tensorflow Version: 2.12.1
Keras Version: 2.12.0
CPU is AVAILABLE
GPU is NOT AVAILABLE
In this example, we will use only the dataset BNCI2014_001
.
Running the benchmark#
The benchmark is run using the benchmark
function. You need to specify the
folder containing the pipelines to use, the kind of evaluation and the paradigm
to use. By default, the benchmark will use all available datasets for all
paradigms listed in the pipelines. You could restrict to specific evaluation and
paradigm using the evaluations
and paradigms
arguments.
To save computation time, the results are cached. If you want to re-run the
benchmark, you can set the overwrite
argument to True
.
It is possible to indicate the folder to cache the results and the one to save
the analysis & figures. By default, the results are saved in the results
folder, and the analysis & figures are saved in the benchmark
folder.
This code is implemented to run on CPU. If you’re using a GPU, do not use multithreading (i.e. set n_jobs=1)
# Set up reproducibility of Tensorflow
setup_seed(42)
# Restrict this example only on the first two subject of BNCI2014_001
dataset = BNCI2014_001()
dataset.subject_list = dataset.subject_list[:2]
datasets = [dataset]
results = benchmark(
pipelines="./pipelines_DL",
evaluations=["WithinSession"],
paradigms=["LeftRightImagery"],
include_datasets=datasets,
results="./results/",
overwrite=False,
plot=False,
output="./benchmark/",
n_jobs=-1,
)
BNCI2014-001-WithinSession: 0%| | 0/2 [00:00<?, ?it/s]/home/runner/work/moabb/moabb/moabb/datasets/download.py:55: RuntimeWarning: Setting non-standard config type: "MNE_DATASETS_BNCI_PATH"
set_config(key, get_config("MNE_DATA"))
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'lampx.tugraz.at'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
0%| | 0.00/42.8M [00:00<?, ?B/s]
0%| | 32.8k/42.8M [00:00<04:12, 170kB/s]
0%| | 113k/42.8M [00:00<01:38, 434kB/s]
1%|▎ | 290k/42.8M [00:00<00:45, 937kB/s]
1%|▌ | 629k/42.8M [00:00<00:23, 1.79MB/s]
3%|▉ | 1.09M/42.8M [00:00<00:15, 2.73MB/s]
5%|█▊ | 2.06M/42.8M [00:00<00:08, 4.96MB/s]
9%|███▍ | 3.98M/42.8M [00:00<00:04, 9.48MB/s]
15%|█████▍ | 6.32M/42.8M [00:00<00:02, 13.8MB/s]
20%|███████▎ | 8.44M/42.8M [00:00<00:02, 16.1MB/s]
25%|█████████▎ | 10.8M/42.8M [00:01<00:01, 18.3MB/s]
30%|███████████▏ | 13.0M/42.8M [00:01<00:01, 19.4MB/s]
36%|█████████████▎ | 15.4M/42.8M [00:01<00:01, 20.8MB/s]
41%|███████████████▎ | 17.7M/42.8M [00:01<00:01, 21.7MB/s]
47%|█████████████████▎ | 20.1M/42.8M [00:01<00:01, 22.1MB/s]
53%|███████████████████▌ | 22.6M/42.8M [00:01<00:00, 23.0MB/s]
58%|█████████████████████▌ | 24.9M/42.8M [00:01<00:00, 23.1MB/s]
66%|████████████████████████▎ | 28.1M/42.8M [00:01<00:00, 25.7MB/s]
72%|██████████████████████████▊ | 31.0M/42.8M [00:01<00:00, 26.7MB/s]
79%|█████████████████████████████▏ | 33.7M/42.8M [00:01<00:00, 27.0MB/s]
85%|███████████████████████████████▌ | 36.6M/42.8M [00:02<00:00, 27.3MB/s]
92%|█████████████████████████████████▉ | 39.3M/42.8M [00:02<00:00, 27.2MB/s]
99%|████████████████████████████████████▋| 42.4M/42.8M [00:02<00:00, 25.6MB/s]
0%| | 0.00/42.8M [00:00<?, ?B/s]
100%|██████████████████████████████████████| 42.8M/42.8M [00:00<00:00, 185GB/s]
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'lampx.tugraz.at'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
0%| | 0.00/43.8M [00:00<?, ?B/s]
0%| | 32.8k/43.8M [00:00<04:21, 167kB/s]
0%| | 136k/43.8M [00:00<01:21, 532kB/s]
1%|▎ | 298k/43.8M [00:00<00:46, 940kB/s]
1%|▍ | 494k/43.8M [00:00<00:33, 1.29MB/s]
2%|▉ | 1.04M/43.8M [00:00<00:16, 2.67MB/s]
5%|█▋ | 2.00M/43.8M [00:00<00:08, 4.93MB/s]
9%|███▎ | 3.89M/43.8M [00:00<00:04, 9.32MB/s]
15%|█████▍ | 6.44M/43.8M [00:00<00:02, 14.3MB/s]
20%|███████▍ | 8.82M/43.8M [00:00<00:02, 17.2MB/s]
26%|█████████▌ | 11.3M/43.8M [00:01<00:01, 19.7MB/s]
31%|███████████▋ | 13.8M/43.8M [00:01<00:01, 21.0MB/s]
37%|█████████████▊ | 16.3M/43.8M [00:01<00:01, 22.3MB/s]
43%|███████████████▊ | 18.7M/43.8M [00:01<00:01, 22.9MB/s]
49%|█████████████████▉ | 21.3M/43.8M [00:01<00:00, 23.7MB/s]
54%|████████████████████ | 23.7M/43.8M [00:01<00:00, 24.0MB/s]
60%|██████████████████████▏ | 26.3M/43.8M [00:01<00:00, 24.4MB/s]
66%|████████████████████████▎ | 28.8M/43.8M [00:01<00:00, 24.6MB/s]
72%|██████████████████████████▍ | 31.3M/43.8M [00:01<00:00, 24.9MB/s]
77%|████████████████████████████▋ | 33.9M/43.8M [00:01<00:00, 25.2MB/s]
83%|██████████████████████████████▊ | 36.4M/43.8M [00:02<00:00, 25.2MB/s]
89%|████████████████████████████████▉ | 39.0M/43.8M [00:02<00:00, 25.4MB/s]
95%|███████████████████████████████████▏ | 41.6M/43.8M [00:02<00:00, 25.5MB/s]
0%| | 0.00/43.8M [00:00<?, ?B/s]
100%|██████████████████████████████████████| 43.8M/43.8M [00:00<00:00, 200GB/s]
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/paradigms/base.py:355: RuntimeWarning: Concatenation of Annotations within Epochs is not supported yet. All annotations will be dropped.
X = mne.concatenate_epochs(X)
BNCI2014-001-WithinSession: 50%|##### | 1/2 [04:31<04:31, 271.33s/it]/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'lampx.tugraz.at'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
0%| | 0.00/43.1M [00:00<?, ?B/s]
0%| | 32.8k/43.1M [00:00<04:17, 167kB/s]
0%|▏ | 145k/43.1M [00:00<01:15, 572kB/s]
1%|▎ | 306k/43.1M [00:00<00:44, 961kB/s]
2%|▌ | 705k/43.1M [00:00<00:21, 2.02MB/s]
3%|█ | 1.28M/43.1M [00:00<00:12, 3.26MB/s]
5%|█▉ | 2.24M/43.1M [00:00<00:07, 5.31MB/s]
10%|███▌ | 4.17M/43.1M [00:00<00:03, 9.73MB/s]
16%|█████▊ | 6.80M/43.1M [00:00<00:02, 14.9MB/s]
22%|███████▉ | 9.29M/43.1M [00:00<00:01, 17.9MB/s]
28%|██████████▏ | 11.8M/43.1M [00:01<00:01, 20.3MB/s]
34%|████████████▍ | 14.5M/43.1M [00:01<00:01, 22.0MB/s]
40%|██████████████▋ | 17.0M/43.1M [00:01<00:01, 23.2MB/s]
46%|████████████████▉ | 19.7M/43.1M [00:01<00:00, 24.1MB/s]
52%|███████████████████ | 22.3M/43.1M [00:01<00:00, 24.7MB/s]
57%|█████████████████████▏ | 24.7M/43.1M [00:01<00:00, 24.6MB/s]
64%|███████████████████████▌ | 27.4M/43.1M [00:01<00:00, 25.2MB/s]
69%|█████████████████████████▋ | 29.9M/43.1M [00:01<00:00, 25.3MB/s]
75%|███████████████████████████▉ | 32.5M/43.1M [00:01<00:00, 25.5MB/s]
82%|██████████████████████████████▏ | 35.1M/43.1M [00:01<00:00, 25.7MB/s]
88%|████████████████████████████████▌ | 37.9M/43.1M [00:02<00:00, 26.2MB/s]
94%|██████████████████████████████████▉ | 40.6M/43.1M [00:02<00:00, 26.5MB/s]
0%| | 0.00/43.1M [00:00<?, ?B/s]
100%|██████████████████████████████████████| 43.1M/43.1M [00:00<00:00, 143GB/s]
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'lampx.tugraz.at'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
0%| | 0.00/44.2M [00:00<?, ?B/s]
0%| | 32.8k/44.2M [00:00<04:24, 167kB/s]
0%| | 136k/44.2M [00:00<01:22, 532kB/s]
1%|▏ | 275k/44.2M [00:00<00:51, 855kB/s]
1%|▌ | 583k/44.2M [00:00<00:26, 1.63MB/s]
2%|▊ | 1.03M/44.2M [00:00<00:16, 2.56MB/s]
5%|█▋ | 2.00M/44.2M [00:00<00:08, 4.88MB/s]
9%|███▎ | 3.90M/44.2M [00:00<00:04, 9.32MB/s]
16%|█████▉ | 7.06M/44.2M [00:00<00:02, 16.2MB/s]
22%|████████▏ | 9.76M/44.2M [00:00<00:01, 19.6MB/s]
28%|██████████▎ | 12.3M/44.2M [00:01<00:01, 21.2MB/s]
34%|████████████▋ | 15.1M/44.2M [00:01<00:01, 23.5MB/s]
41%|███████████████ | 18.0M/44.2M [00:01<00:01, 25.1MB/s]
47%|█████████████████▏ | 20.6M/44.2M [00:01<00:00, 25.3MB/s]
53%|███████████████████▌ | 23.4M/44.2M [00:01<00:00, 26.1MB/s]
59%|█████████████████████▊ | 26.0M/44.2M [00:01<00:00, 22.9MB/s]
65%|████████████████████████▏ | 28.9M/44.2M [00:01<00:00, 24.6MB/s]
72%|██████████████████████████▊ | 32.0M/44.2M [00:01<00:00, 26.2MB/s]
78%|█████████████████████████████ | 34.7M/44.2M [00:01<00:00, 25.9MB/s]
85%|███████████████████████████████▎ | 37.4M/44.2M [00:02<00:00, 26.4MB/s]
91%|█████████████████████████████████▌ | 40.1M/44.2M [00:02<00:00, 25.7MB/s]
97%|███████████████████████████████████▉ | 42.9M/44.2M [00:02<00:00, 26.4MB/s]
0%| | 0.00/44.2M [00:00<?, ?B/s]
100%|██████████████████████████████████████| 44.2M/44.2M [00:00<00:00, 197GB/s]
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs | 24 events (all good), 2 – 6 s, baseline off, ~4.1 MB, data loaded,
'left_hand': 12
'right_hand': 12>
warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/paradigms/base.py:355: RuntimeWarning: Concatenation of Annotations within Epochs is not supported yet. All annotations will be dropped.
X = mne.concatenate_epochs(X)
BNCI2014-001-WithinSession: 100%|##########| 2/2 [09:07<00:00, 273.91s/it]
BNCI2014-001-WithinSession: 100%|##########| 2/2 [09:07<00:00, 273.52s/it]
dataset evaluation pipeline avg score
0 BNCI2014-001 WithinSession Keras_DeepConvNet 0.522551
1 BNCI2014-001 WithinSession Keras_EEGTCNet 0.462585
2 BNCI2014-001 WithinSession Keras_EEGNeX 0.534371
3 BNCI2014-001 WithinSession Keras_ShallowConvNet 0.612891
4 BNCI2014-001 WithinSession Keras_EEGITNet 0.564524
5 BNCI2014-001 WithinSession Keras_EEGNet_8_2 0.488010
The deep learning architectures implemented in MOABB are: - Shallow Convolutional Network [1] - Deep Convolutional Network [1] - EEGNet [2] - EEGTCNet [3] - EEGNex [4] - EEGITNet [5]
Benchmark prints a summary of the results. Detailed results are saved in a
pandas dataframe, and can be used to generate figures. The analysis & figures
are saved in the benchmark
folder.
score_plot(results)
plt.show()
References#
Total running time of the script: ( 9 minutes 8.660 seconds)
Estimated memory usage: 855 MB