Tutorial 3: Benchmarking multiple pipelines

In this last part, we extend the previous example by assessing the classification score of not one but three classification pipelines.

# Authors: Pedro L. C. Rodrigues, Sylvain Chevallier
#
# https://github.com/plcrodrigues/Workshop-MOABB-BCI-Graz-2019

import warnings

import matplotlib.pyplot as plt
import mne
import seaborn as sns
from mne.decoding import CSP
from pyriemann.classification import MDM
from pyriemann.estimation import Covariances
from pyriemann.tangentspace import TangentSpace
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.pipeline import make_pipeline
from sklearn.svm import SVC

import moabb
from moabb.datasets import BNCI2014_001, Zhou2016
from moabb.evaluations import WithinSessionEvaluation
from moabb.paradigms import LeftRightImagery


mne.set_log_level("CRITICAL")
moabb.set_log_level("info")
warnings.filterwarnings("ignore")

/home/runner/work/moabb/moabb/.venv/lib/python3.10/site-packages/optuna/integration/sklearn.py:14: FutureWarning: `optuna.integration.sklearn` has been deprecated in v4.9.0. This feature will be removed in v6.0.0. See https://github.com/optuna/optuna/releases/tag/v4.9.0. Use `optuna_integration.sklearn` instead.
  optuna_warn(f"{msg} Use `optuna_integration.sklearn` instead.", FutureWarning)

Creating Pipelines

We instantiate the three different classiciation pipelines to be considered in the analysis. The object that gathers each pipeline is a dictionary. The first pipeline is the CSP+LDA that we have seen in the previous parts. The other two pipelines rely on Riemannian geometry, using an SVM classification in the tangent space of the covariance matrices estimated from the EEG or a MDM classifier that works directly on covariance matrices.

pipelines = {}
pipelines["csp+lda"] = make_pipeline(CSP(n_components=8), LDA())
pipelines["tgsp+svm"] = make_pipeline(
    Covariances("oas"), TangentSpace(metric="riemann"), SVC(kernel="linear")
)
pipelines["MDM"] = make_pipeline(Covariances("oas"), MDM(metric="riemann"))

The following lines go exactly as in the previous tutorial, where we end up obtaining a pandas dataframe containing the results of the evaluation.

datasets = [BNCI2014_001(), Zhou2016()]
subj = [1, 2, 3]
for d in datasets:
    d.subject_list = subj
paradigm = LeftRightImagery()
evaluation = WithinSessionEvaluation(
    paradigm=paradigm, datasets=datasets, overwrite=False
)
results = evaluation.process(pipelines)

[codecarbon WARNING @ 11:52:34] Multiple instances of codecarbon are allowed to run at the same time.
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base csp+lda | BNCI2014-001 | 2 | 1test: Score 0.653
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base tgsp+svm | BNCI2014-001 | 2 | 1test: Score 0.662
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base MDM | BNCI2014-001 | 2 | 1test: Score 0.549
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base csp+lda | BNCI2014-001 | 2 | 0train: Score 0.640
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base tgsp+svm | BNCI2014-001 | 2 | 0train: Score 0.667
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base MDM | BNCI2014-001 | 2 | 0train: Score 0.564
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base csp+lda | BNCI2014-001 | 3 | 0train: Score 0.994
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base tgsp+svm | BNCI2014-001 | 3 | 0train: Score 0.989
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base MDM | BNCI2014-001 | 3 | 0train: Score 0.974
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base csp+lda | BNCI2014-001 | 3 | 1test: Score 0.995
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base tgsp+svm | BNCI2014-001 | 3 | 1test: Score 0.998
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base MDM | BNCI2014-001 | 3 | 1test: Score 0.987
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base csp+lda | BNCI2014-001 | 1 | 0train: Score 0.928
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base tgsp+svm | BNCI2014-001 | 1 | 0train: Score 0.946
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base MDM | BNCI2014-001 | 1 | 0train: Score 0.946
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base csp+lda | BNCI2014-001 | 1 | 1test: Score 0.961
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base tgsp+svm | BNCI2014-001 | 1 | 1test: Score 0.971
2026-06-29 11:57:22,898 INFO MainThread moabb.evaluations.base MDM | BNCI2014-001 | 1 | 1test: Score 0.953
2026-06-29 12:02:07,652 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 2 | 2: Score 0.914
2026-06-29 12:02:07,653 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 2 | 2: Score 0.926
2026-06-29 12:02:07,653 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 2 | 0: Score 0.878
2026-06-29 12:02:07,653 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 2 | 0: Score 0.870
2026-06-29 12:02:07,653 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 2 | 1: Score 0.840
2026-06-29 12:02:07,653 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 2 | 1: Score 0.728
2026-06-29 12:02:07,653 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 3 | 1: Score 0.992
2026-06-29 12:02:07,653 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 3 | 1: Score 0.908
2026-06-29 12:02:07,653 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 3 | 0: Score 0.994
2026-06-29 12:02:07,653 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 3 | 0: Score 0.946
2026-06-29 12:02:07,653 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 3 | 2: Score 1.000
2026-06-29 12:02:07,653 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 3 | 2: Score 0.924
2026-06-29 12:02:07,653 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 1 | 1: Score 0.874
2026-06-29 12:02:07,653 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 1 | 1: Score 0.872
2026-06-29 12:02:07,653 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 1 | 0: Score 0.899
2026-06-29 12:02:07,653 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 1 | 0: Score 0.812
2026-06-29 12:02:07,653 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 1 | 2: Score 0.952
2026-06-29 12:02:07,653 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 1 | 2: Score 0.948

As overwrite is set to False, the results from the previous tutorial are reused and only the new pipelines are evaluated. The results from “csp+lda” are not recomputed. The results are saved in ~/mne_data/results if the parameter hdf5_path is not set.

Plotting Results

The following plot shows a comparison of the three classification pipelines for each subject of each dataset.

results["subj"] = [str(resi).zfill(2) for resi in results["subject"]]
g = sns.catplot(
    kind="bar",
    x="score",
    y="subj",
    hue="pipeline",
    col="dataset",
    height=12,
    aspect=0.5,
    data=results,
    orient="h",
    palette="viridis",
)
plt.show()

2026-06-29 12:02:07,793 INFO MainThread matplotlib.category Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
2026-06-29 12:02:07,799 INFO MainThread matplotlib.category Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.