Tutorial 3: Benchmarking multiple pipelines

In this last part, we extend the previous example by assessing the classification score of not one but three classification pipelines.

# Authors: Pedro L. C. Rodrigues, Sylvain Chevallier
#
# https://github.com/plcrodrigues/Workshop-MOABB-BCI-Graz-2019

import warnings

import matplotlib.pyplot as plt
import mne
import seaborn as sns
from mne.decoding import CSP
from pyriemann.classification import MDM
from pyriemann.estimation import Covariances
from pyriemann.tangentspace import TangentSpace
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.pipeline import make_pipeline
from sklearn.svm import SVC

import moabb
from moabb.datasets import BNCI2014_001, Zhou2016
from moabb.evaluations import WithinSessionEvaluation
from moabb.paradigms import LeftRightImagery


mne.set_log_level("CRITICAL")
moabb.set_log_level("info")
warnings.filterwarnings("ignore")

Creating Pipelines

We instantiate the three different classiciation pipelines to be considered in the analysis. The object that gathers each pipeline is a dictionary. The first pipeline is the CSP+LDA that we have seen in the previous parts. The other two pipelines rely on Riemannian geometry, using an SVM classification in the tangent space of the covariance matrices estimated from the EEG or a MDM classifier that works directly on covariance matrices.

pipelines = {}
pipelines["csp+lda"] = make_pipeline(CSP(n_components=8), LDA())
pipelines["tgsp+svm"] = make_pipeline(
    Covariances("oas"), TangentSpace(metric="riemann"), SVC(kernel="linear")
)
pipelines["MDM"] = make_pipeline(Covariances("oas"), MDM(metric="riemann"))

The following lines go exactly as in the previous tutorial, where we end up obtaining a pandas dataframe containing the results of the evaluation.

datasets = [BNCI2014_001(), Zhou2016()]
subj = [1, 2, 3]
for d in datasets:
    d.subject_list = subj
paradigm = LeftRightImagery()
evaluation = WithinSessionEvaluation(
    paradigm=paradigm, datasets=datasets, overwrite=False
)
results = evaluation.process(pipelines)

[codecarbon WARNING @ 19:50:46] Multiple instances of codecarbon are allowed to run at the same time.
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base csp+lda | BNCI2014-001 | 1 | 0train: Score 0.920
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base tgsp+svm | BNCI2014-001 | 1 | 0train: Score 0.917
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base MDM | BNCI2014-001 | 1 | 0train: Score 0.939
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base csp+lda | BNCI2014-001 | 1 | 1test: Score 0.944
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base tgsp+svm | BNCI2014-001 | 1 | 1test: Score 0.956
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base MDM | BNCI2014-001 | 1 | 1test: Score 0.918
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base csp+lda | BNCI2014-001 | 2 | 1test: Score 0.604
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base tgsp+svm | BNCI2014-001 | 2 | 1test: Score 0.609
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base MDM | BNCI2014-001 | 2 | 1test: Score 0.524
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base csp+lda | BNCI2014-001 | 2 | 0train: Score 0.696
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base tgsp+svm | BNCI2014-001 | 2 | 0train: Score 0.734
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base MDM | BNCI2014-001 | 2 | 0train: Score 0.622
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base csp+lda | BNCI2014-001 | 3 | 0train: Score 0.990
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base tgsp+svm | BNCI2014-001 | 3 | 0train: Score 0.986
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base MDM | BNCI2014-001 | 3 | 0train: Score 0.980
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base csp+lda | BNCI2014-001 | 3 | 1test: Score 0.993
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base tgsp+svm | BNCI2014-001 | 3 | 1test: Score 0.997
2026-03-21 19:55:35,394 INFO MainThread moabb.evaluations.base MDM | BNCI2014-001 | 3 | 1test: Score 0.991
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 1 | 2: Score 0.954
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 1 | 2: Score 0.960
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 1 | 0: Score 0.922
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 1 | 0: Score 0.815
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 1 | 1: Score 0.888
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 1 | 1: Score 0.890
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 2 | 2: Score 0.884
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 2 | 2: Score 0.898
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 2 | 0: Score 0.908
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 2 | 0: Score 0.870
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 2 | 1: Score 0.805
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 2 | 1: Score 0.723
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 3 | 2: Score 0.998
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 3 | 2: Score 0.948
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 3 | 0: Score 0.992
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 3 | 0: Score 0.954
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base tgsp+svm | Zhou2016 | 3 | 1: Score 0.998
2026-03-21 20:00:20,482 INFO MainThread moabb.evaluations.base MDM | Zhou2016 | 3 | 1: Score 0.898

As overwrite is set to False, the results from the previous tutorial are reused and only the new pipelines are evaluated. The results from “csp+lda” are not recomputed. The results are saved in ~/mne_data/results if the parameter hdf5_path is not set.

Plotting Results

The following plot shows a comparison of the three classification pipelines for each subject of each dataset.

results["subj"] = [str(resi).zfill(2) for resi in results["subject"]]
g = sns.catplot(
    kind="bar",
    x="score",
    y="subj",
    hue="pipeline",
    col="dataset",
    height=12,
    aspect=0.5,
    data=results,
    orient="h",
    palette="viridis",
)
plt.show()

2026-03-21 20:00:21,183 INFO MainThread matplotlib.category Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
2026-03-21 20:00:21,188 INFO MainThread matplotlib.category Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.