Note

Click here to download the full example code

Cross-Subject SSVEP#

This example shows how to perform a cross-subject analysis on an SSVEP dataset. We will compare four pipelines :

  • Riemannian Geometry

  • CCA

  • TRCA

  • MsetCCA

We will use the SSVEP paradigm, which uses the AUC as metric.

# Authors: Sylvain Chevallier <sylvain.chevallier@uvsq.fr>
#
# License: BSD (3-clause)

import warnings

import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
from pyriemann.estimation import Covariances
from pyriemann.tangentspace import TangentSpace
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

import moabb
from moabb.datasets import Kalunga2016
from moabb.evaluations import CrossSubjectEvaluation
from moabb.paradigms import SSVEP, FilterBankSSVEP
from moabb.pipelines import SSVEP_CCA, SSVEP_TRCA, ExtendedSSVEPSignal, SSVEP_MsetCCA


warnings.simplefilter(action="ignore", category=FutureWarning)
warnings.simplefilter(action="ignore", category=RuntimeWarning)
moabb.set_log_level("info")

Loading Dataset#

We will load the data from the first 2 subjects of the SSVEP_Exo dataset and compare two algorithms on this set. One of the algorithms could only process class associated with a stimulation frequency, we will thus drop the resting class. As the resting class is the last defined class, picking the first three classes (out of four) allows to focus only on the stimulation frequency.

n_subject = 2
dataset = Kalunga2016()
dataset.subject_list = dataset.subject_list[:n_subject]
interval = dataset.interval

Choose Paradigm#

We define the paradigms (SSVEP, SSVEP TRCA, SSVEP MsetCCA, and FilterBankSSVEP) and use the dataset Kalunga2016. All 3 SSVEP paradigms applied a bandpass filter (10-42 Hz) on the data, which include all stimuli frequencies and their first harmonics, while the FilterBankSSVEP paradigm uses as many bandpass filters as there are stimulation frequencies (here 3). For each stimulation frequency the EEG is filtered with a 1 Hz-wide bandpass filter centered on the frequency. This results in n_classes copies of the signal, filtered for each class, as used in the filterbank motor imagery paradigms.

paradigm = SSVEP(fmin=10, fmax=42, n_classes=3)
paradigm_TRCA = SSVEP(fmin=10, fmax=42, n_classes=3)
paradigm_MSET_CCA = SSVEP(fmin=10, fmax=42, n_classes=3)
paradigm_fb = FilterBankSSVEP(filters=None, n_classes=3)

Classes are defined by the frequency of the stimulation, here we use the first two frequencies of the dataset, 13 and 17 Hz. The evaluation function uses a LabelEncoder, transforming them to 0 and 1

freqs = paradigm.used_events(dataset)

Create Pipelines#

Pipelines must be a dict of sklearn pipeline transformer. The first pipeline uses Riemannian geometry, by building an extended covariance matrices from the signal filtered around the considered frequency and applying a logistic regression in the tangent plane. The second pipeline relies on the above defined CCA classifier. The third pipeline relies on the TRCA algorithm, and the fourth uses the MsetCCA algorithm. Both CCA based methods (i.e. CCA and MsetCCA) used 3 CCA components.

pipelines_fb = {}
pipelines_fb["RG+LogReg"] = make_pipeline(
    ExtendedSSVEPSignal(),
    Covariances(estimator="lwf"),
    TangentSpace(),
    LogisticRegression(solver="lbfgs", multi_class="auto"),
)

pipelines = {}
pipelines["CCA"] = make_pipeline(SSVEP_CCA(interval=interval, freqs=freqs, n_harmonics=2))

pipelines_TRCA = {}
pipelines_TRCA["TRCA"] = make_pipeline(SSVEP_TRCA(interval=interval, freqs=freqs))

pipelines_MSET_CCA = {}
pipelines_MSET_CCA["MSET_CCA"] = make_pipeline(SSVEP_MsetCCA(freqs=freqs))

Evaluation#

The evaluation will return a DataFrame containing an accuracy score for each subject / session of the dataset, and for each pipeline.

Results are saved into the database, so that if you add a new pipeline, it will not run again the evaluation unless a parameter has changed. Results can be overwritten if necessary.

overwrite = False  # set to True if we want to overwrite cached results

evaluation = CrossSubjectEvaluation(
    paradigm=paradigm, datasets=dataset, overwrite=overwrite
)
results = evaluation.process(pipelines)

/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'zenodo.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'zenodo.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(

  0%|                                              | 0.00/2.06M [00:00<?, ?B/s]
  2%|▉                                     | 49.2k/2.06M [00:00<00:07, 265kB/s]
  8%|██▉                                    | 156k/2.06M [00:00<00:04, 449kB/s]
 15%|█████▉                                 | 312k/2.06M [00:00<00:02, 630kB/s]
 24%|█████████▎                             | 493k/2.06M [00:00<00:02, 766kB/s]
 36%|█████████████▉                         | 737k/2.06M [00:00<00:01, 959kB/s]
 45%|█████████████████▋                     | 934k/2.06M [00:01<00:01, 991kB/s]
 65%|████████████████████████▏            | 1.34M/2.06M [00:01<00:00, 1.38MB/s]
 80%|█████████████████████████████▍       | 1.64M/2.06M [00:01<00:00, 1.45MB/s]
 94%|██████████████████████████████████▊  | 1.93M/2.06M [00:01<00:00, 1.49MB/s]
  0%|                                              | 0.00/2.06M [00:00<?, ?B/s]
100%|█████████████████████████████████████| 2.06M/2.06M [00:00<00:00, 11.5GB/s]
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'zenodo.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'zenodo.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(

  0%|                                              | 0.00/2.82M [00:00<?, ?B/s]
  1%|▏                                     | 14.3k/2.82M [00:00<00:23, 122kB/s]
  4%|█▍                                     | 108k/2.82M [00:00<00:07, 385kB/s]
 10%|████                                   | 295k/2.82M [00:00<00:03, 694kB/s]
 16%|██████▍                                | 461k/2.82M [00:00<00:03, 777kB/s]
 23%|████████▉                              | 641k/2.82M [00:00<00:02, 848kB/s]
 30%|███████████▊                           | 853k/2.82M [00:01<00:02, 946kB/s]
 37%|█████████████▉                        | 1.03M/2.82M [00:01<00:01, 952kB/s]
 45%|████████████████▌                    | 1.26M/2.82M [00:01<00:01, 1.04MB/s]
 52%|███████████████████▍                 | 1.47M/2.82M [00:01<00:01, 1.07MB/s]
 59%|█████████████████████▋               | 1.65M/2.82M [00:01<00:01, 1.03MB/s]
 66%|████████████████████████▎            | 1.85M/2.82M [00:01<00:00, 1.04MB/s]
 73%|██████████████████████████▉          | 2.05M/2.82M [00:02<00:00, 1.04MB/s]
 80%|█████████████████████████████▋       | 2.26M/2.82M [00:02<00:00, 1.07MB/s]
 87%|████████████████████████████████▎    | 2.46M/2.82M [00:02<00:00, 1.06MB/s]
 94%|██████████████████████████████████▊  | 2.65M/2.82M [00:02<00:00, 1.06MB/s]
  0%|                                              | 0.00/2.82M [00:00<?, ?B/s]
100%|█████████████████████████████████████| 2.82M/2.82M [00:00<00:00, 9.85GB/s]
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'zenodo.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'zenodo.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(

  0%|                                              | 0.00/2.58M [00:00<?, ?B/s]
  1%|▏                                     | 14.3k/2.58M [00:00<00:19, 132kB/s]
  4%|█▌                                     | 100k/2.58M [00:00<00:05, 455kB/s]
  9%|███▎                                   | 223k/2.58M [00:00<00:03, 656kB/s]
 16%|██████▏                                | 411k/2.58M [00:00<00:02, 823kB/s]
 25%|█████████▋                            | 656k/2.58M [00:00<00:01, 1.03MB/s]
 37%|██████████████▏                       | 967k/2.58M [00:00<00:01, 1.25MB/s]
 48%|█████████████████▌                   | 1.23M/2.58M [00:01<00:01, 1.30MB/s]
 58%|█████████████████████▌               | 1.51M/2.58M [00:01<00:00, 1.36MB/s]
 70%|█████████████████████████▊           | 1.80M/2.58M [00:01<00:00, 1.43MB/s]
 79%|█████████████████████████████        | 2.03M/2.58M [00:01<00:00, 1.37MB/s]
 86%|███████████████████████████████▉     | 2.23M/2.58M [00:01<00:00, 1.27MB/s]
 96%|███████████████████████████████████▍ | 2.47M/2.58M [00:02<00:00, 1.28MB/s]
  0%|                                              | 0.00/2.58M [00:00<?, ?B/s]
100%|█████████████████████████████████████| 2.58M/2.58M [00:00<00:00, 15.1GB/s]
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'zenodo.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/home/runner/work/moabb/moabb/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'zenodo.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(

  0%|                                              | 0.00/2.14M [00:00<?, ?B/s]
  1%|▏                                     | 12.3k/2.14M [00:00<00:18, 117kB/s]
  5%|█▉                                     | 104k/2.14M [00:00<00:05, 391kB/s]
 10%|███▊                                   | 208k/2.14M [00:00<00:04, 477kB/s]
 17%|██████▌                                | 358k/2.14M [00:00<00:02, 618kB/s]
 24%|█████████▌                             | 522k/2.14M [00:00<00:02, 719kB/s]
 35%|█████████████▋                         | 751k/2.14M [00:01<00:01, 898kB/s]
 46%|█████████████████▍                    | 980k/2.14M [00:01<00:01, 1.01MB/s]
 55%|████████████████████▎                | 1.18M/2.14M [00:01<00:00, 1.03MB/s]
 65%|███████████████████████▉             | 1.39M/2.14M [00:01<00:00, 1.06MB/s]
 74%|███████████████████████████▎         | 1.59M/2.14M [00:01<00:00, 1.06MB/s]
 85%|███████████████████████████████▌     | 1.83M/2.14M [00:01<00:00, 1.14MB/s]
 95%|██████████████████████████████████▉  | 2.03M/2.14M [00:02<00:00, 1.11MB/s]
  0%|                                              | 0.00/2.14M [00:00<?, ?B/s]
100%|█████████████████████████████████████| 2.14M/2.14M [00:00<00:00, 6.09GB/s]
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")

Kalunga2016-CrossSubject:   0%|          | 0/2 [00:00<?, ?it/s]
Kalunga2016-CrossSubject:  50%|#####     | 1/2 [00:00<00:00,  2.05it/s]
Kalunga2016-CrossSubject: 100%|##########| 2/2 [00:00<00:00,  2.05it/s]
Kalunga2016-CrossSubject: 100%|##########| 2/2 [00:00<00:00,  2.05it/s]

Filter bank processing, determine the filter automatically from the stimulation frequency values of events.

evaluation_fb = CrossSubjectEvaluation(
    paradigm=paradigm_fb, datasets=dataset, overwrite=overwrite
)
results_fb = evaluation_fb.process(pipelines_fb)

/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")

Kalunga2016-CrossSubject:   0%|          | 0/2 [00:00<?, ?it/s]
Kalunga2016-CrossSubject:  50%|#####     | 1/2 [00:00<00:00,  2.25it/s]
Kalunga2016-CrossSubject: 100%|##########| 2/2 [00:00<00:00,  2.17it/s]
Kalunga2016-CrossSubject: 100%|##########| 2/2 [00:00<00:00,  2.18it/s]

TRCA processing also relies on filter bank that is automatically designed.

evaluation_TRCA = CrossSubjectEvaluation(
    paradigm=paradigm_TRCA, datasets=dataset, overwrite=overwrite
)
results_TRCA = evaluation_TRCA.process(pipelines_TRCA)

/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")

Kalunga2016-CrossSubject:   0%|          | 0/2 [00:00<?, ?it/s]
Kalunga2016-CrossSubject:  50%|#####     | 1/2 [00:00<00:00,  1.15it/s]
Kalunga2016-CrossSubject: 100%|##########| 2/2 [00:01<00:00,  1.15it/s]
Kalunga2016-CrossSubject: 100%|##########| 2/2 [00:01<00:00,  1.15it/s]

MsetCCA processing

evaluation_MSET_CCA = CrossSubjectEvaluation(
    paradigm=paradigm_MSET_CCA, datasets=dataset, overwrite=overwrite
)
results_MSET_CCA = evaluation_MSET_CCA.process(pipelines_MSET_CCA)

/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")
/home/runner/work/moabb/moabb/moabb/datasets/preprocessing.py:273: UserWarning: warnEpochs <Epochs |  24 events (all good), 2 – 4 s, baseline off, ~784 kB, data loaded,
 '13': 8
 '17': 8
 '21': 8>
  warn(f"warnEpochs {epochs}")

Kalunga2016-CrossSubject:   0%|          | 0/2 [00:00<?, ?it/s]
Kalunga2016-CrossSubject:  50%|#####     | 1/2 [00:00<00:00,  1.54it/s]
Kalunga2016-CrossSubject: 100%|##########| 2/2 [00:01<00:00,  1.55it/s]
Kalunga2016-CrossSubject: 100%|##########| 2/2 [00:01<00:00,  1.55it/s]

After processing the four, we simply concatenate the results.

results = pd.concat([results, results_fb, results_TRCA, results_MSET_CCA])

Plot Results#

Here we display the results as stripplot, with a pointplot for error bar.

fig, ax = plt.subplots(facecolor="white", figsize=[8, 4])
sns.stripplot(
    data=results,
    y="score",
    x="pipeline",
    ax=ax,
    jitter=True,
    alpha=0.5,
    zorder=1,
    palette="Set1",
)
sns.pointplot(data=results, y="score", x="pipeline", ax=ax, palette="Set1")
ax.set_ylabel("Accuracy")
ax.set_ylim(0.1, 0.6)
plt.show()
plot cross subject ssvep

Total running time of the script: ( 0 minutes 20.830 seconds)

Estimated memory usage: 20 MB

Gallery generated by Sphinx-Gallery