Note
Go to the end to download the full example code.
Within Session P300#
This example shows how to perform a within session analysis on three different P300 datasets.
We will compare two pipelines :
Riemannian geometry
XDAWN with Linear Discriminant Analysis
We will use the P300 paradigm, which uses the AUC as metric.
# Authors: Pedro Rodrigues <pedro.rodrigues01@gmail.com>
#
# License: BSD (3-clause)
import warnings
import matplotlib.pyplot as plt
import seaborn as sns
from mne.decoding import Vectorizer
from pyriemann.estimation import Xdawn, XdawnCovariances
from pyriemann.tangentspace import TangentSpace
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.pipeline import make_pipeline
import moabb
from moabb.datasets import BNCI2014_009
from moabb.evaluations import WithinSessionEvaluation
from moabb.paradigms import P300
getting rid of the warnings about the future
warnings.simplefilter(action="ignore", category=FutureWarning)
warnings.simplefilter(action="ignore", category=RuntimeWarning)
moabb.set_log_level("info")
Create Pipelines#
Pipelines must be a dict of sklearn pipeline transformer.
pipelines = {}
We have to do this because the classes are called ‘Target’ and ‘NonTarget’ but the evaluation function uses a LabelEncoder, transforming them to 0 and 1
labels_dict = {"Target": 1, "NonTarget": 0}
pipelines["RG+LDA"] = make_pipeline(
XdawnCovariances(
nfilter=2, classes=[labels_dict["Target"]], estimator="lwf", xdawn_estimator="scm"
),
TangentSpace(),
LDA(solver="lsqr", shrinkage="auto"),
)
pipelines["Xdw+LDA"] = make_pipeline(
Xdawn(nfilter=2, estimator="scm"), Vectorizer(), LDA(solver="lsqr", shrinkage="auto")
)
Evaluation#
We define the paradigm (P300) and use all three datasets available for it. The evaluation will return a DataFrame containing a single AUC score for each subject / session of the dataset, and for each pipeline.
Results are saved into the database, so that if you add a new pipeline, it will not run again the evaluation unless a parameter has changed. Results can be overwritten if necessary.
paradigm = P300(resample=128)
dataset = BNCI2014_009()
dataset.subject_list = dataset.subject_list[:2]
datasets = [dataset]
overwrite = True # set to True if we want to overwrite cached results
evaluation = WithinSessionEvaluation(
paradigm=paradigm, datasets=datasets, suffix="examples", overwrite=overwrite
)
results = evaluation.process(pipelines)
BNCI2014-009-WithinSession: 0%| | 0/2 [00:00<?, ?it/s]
0%| | 0.00/18.5M [00:00<?, ?B/s]
0%| | 8.19k/18.5M [00:00<04:14, 72.9kB/s]
0%| | 56.3k/18.5M [00:00<01:07, 275kB/s]
1%|▎ | 153k/18.5M [00:00<00:34, 528kB/s]
2%|▋ | 344k/18.5M [00:00<00:18, 968kB/s]
4%|█▍ | 728k/18.5M [00:00<00:09, 1.81MB/s]
7%|██▌ | 1.31M/18.5M [00:00<00:05, 2.90MB/s]
9%|███▏ | 1.61M/18.5M [00:00<00:05, 2.90MB/s]
10%|███▊ | 1.90M/18.5M [00:00<00:05, 2.84MB/s]
12%|████▎ | 2.18M/18.5M [00:01<00:06, 2.71MB/s]
13%|████▉ | 2.45M/18.5M [00:01<00:06, 2.63MB/s]
15%|█████▍ | 2.72M/18.5M [00:01<00:06, 2.56MB/s]
16%|█████▉ | 2.97M/18.5M [00:01<00:06, 2.53MB/s]
17%|██████▍ | 3.23M/18.5M [00:01<00:06, 2.49MB/s]
19%|██████▉ | 3.48M/18.5M [00:01<00:06, 2.47MB/s]
20%|███████▍ | 3.72M/18.5M [00:01<00:06, 2.41MB/s]
21%|███████▉ | 3.96M/18.5M [00:01<00:06, 2.35MB/s]
23%|████████▍ | 4.20M/18.5M [00:01<00:06, 2.23MB/s]
24%|████████▊ | 4.42M/18.5M [00:01<00:06, 2.21MB/s]
25%|█████████▎ | 4.65M/18.5M [00:02<00:06, 2.20MB/s]
26%|█████████▋ | 4.87M/18.5M [00:02<00:06, 2.18MB/s]
27%|██████████▏ | 5.09M/18.5M [00:02<00:06, 2.13MB/s]
29%|██████████▌ | 5.31M/18.5M [00:02<00:06, 2.14MB/s]
30%|███████████ | 5.53M/18.5M [00:02<00:06, 2.11MB/s]
31%|███████████▍ | 5.74M/18.5M [00:02<00:06, 2.06MB/s]
32%|███████████▉ | 5.95M/18.5M [00:02<00:06, 2.04MB/s]
33%|████████████▎ | 6.15M/18.5M [00:02<00:06, 2.00MB/s]
35%|████████████▊ | 6.41M/18.5M [00:02<00:05, 2.12MB/s]
36%|█████████████▏ | 6.63M/18.5M [00:03<00:05, 2.13MB/s]
37%|█████████████▋ | 6.86M/18.5M [00:03<00:05, 2.15MB/s]
38%|██████████████▏ | 7.08M/18.5M [00:03<00:05, 2.15MB/s]
39%|██████████████▌ | 7.30M/18.5M [00:03<00:05, 2.16MB/s]
41%|███████████████ | 7.53M/18.5M [00:03<00:05, 2.18MB/s]
42%|███████████████▍ | 7.75M/18.5M [00:03<00:04, 2.17MB/s]
43%|███████████████▉ | 7.97M/18.5M [00:03<00:04, 2.16MB/s]
44%|████████████████▍ | 8.20M/18.5M [00:03<00:04, 2.18MB/s]
45%|████████████████▊ | 8.42M/18.5M [00:03<00:04, 2.16MB/s]
47%|█████████████████▎ | 8.64M/18.5M [00:03<00:04, 2.11MB/s]
48%|█████████████████▊ | 8.92M/18.5M [00:04<00:04, 2.30MB/s]
50%|██████████████████▎ | 9.18M/18.5M [00:04<00:03, 2.37MB/s]
51%|██████████████████▊ | 9.44M/18.5M [00:04<00:03, 2.42MB/s]
52%|███████████████████▎ | 9.69M/18.5M [00:04<00:03, 2.42MB/s]
54%|███████████████████▊ | 9.93M/18.5M [00:04<00:03, 2.31MB/s]
55%|████████████████████▎ | 10.2M/18.5M [00:04<00:03, 2.19MB/s]
56%|████████████████████▊ | 10.4M/18.5M [00:04<00:03, 2.25MB/s]
57%|█████████████████████▏ | 10.6M/18.5M [00:04<00:03, 2.26MB/s]
59%|█████████████████████▋ | 10.9M/18.5M [00:04<00:03, 2.19MB/s]
60%|██████████████████████▏ | 11.1M/18.5M [00:05<00:03, 2.15MB/s]
61%|██████████████████████▌ | 11.3M/18.5M [00:05<00:03, 2.06MB/s]
62%|███████████████████████ | 11.5M/18.5M [00:05<00:03, 2.09MB/s]
63%|███████████████████████▍ | 11.7M/18.5M [00:05<00:03, 2.11MB/s]
65%|███████████████████████▉ | 12.0M/18.5M [00:05<00:03, 2.13MB/s]
66%|████████████████████████▍ | 12.2M/18.5M [00:05<00:02, 2.23MB/s]
67%|████████████████████████▉ | 12.5M/18.5M [00:05<00:02, 2.32MB/s]
69%|█████████████████████████▍ | 12.7M/18.5M [00:05<00:02, 2.38MB/s]
70%|█████████████████████████▉ | 13.0M/18.5M [00:05<00:02, 2.45MB/s]
72%|██████████████████████████▍ | 13.3M/18.5M [00:05<00:02, 2.48MB/s]
73%|██████████████████████████▉ | 13.5M/18.5M [00:06<00:02, 2.32MB/s]
74%|███████████████████████████▍ | 13.7M/18.5M [00:06<00:02, 2.27MB/s]
75%|███████████████████████████▉ | 14.0M/18.5M [00:06<00:02, 2.19MB/s]
77%|████████████████████████████▎ | 14.2M/18.5M [00:06<00:01, 2.18MB/s]
78%|████████████████████████████▊ | 14.4M/18.5M [00:06<00:01, 2.07MB/s]
79%|█████████████████████████████▏ | 14.6M/18.5M [00:06<00:01, 2.02MB/s]
80%|█████████████████████████████▌ | 14.8M/18.5M [00:06<00:01, 1.99MB/s]
81%|█████████████████████████████▉ | 15.0M/18.5M [00:06<00:01, 1.95MB/s]
82%|██████████████████████████████▍ | 15.2M/18.5M [00:06<00:01, 1.89MB/s]
83%|██████████████████████████████▊ | 15.4M/18.5M [00:07<00:01, 1.87MB/s]
84%|███████████████████████████████▏ | 15.6M/18.5M [00:07<00:01, 1.84MB/s]
85%|███████████████████████████████▌ | 15.8M/18.5M [00:07<00:01, 1.79MB/s]
86%|███████████████████████████████▊ | 16.0M/18.5M [00:07<00:01, 1.76MB/s]
87%|████████████████████████████████▏ | 16.1M/18.5M [00:07<00:01, 1.73MB/s]
88%|████████████████████████████████▌ | 16.3M/18.5M [00:07<00:01, 1.73MB/s]
89%|████████████████████████████████▉ | 16.5M/18.5M [00:07<00:01, 1.71MB/s]
90%|█████████████████████████████████▎ | 16.7M/18.5M [00:07<00:01, 1.70MB/s]
91%|█████████████████████████████████▌ | 16.8M/18.5M [00:07<00:01, 1.66MB/s]
92%|█████████████████████████████████▉ | 17.0M/18.5M [00:08<00:00, 1.67MB/s]
93%|██████████████████████████████████▎ | 17.2M/18.5M [00:08<00:00, 1.64MB/s]
94%|██████████████████████████████████▋ | 17.4M/18.5M [00:08<00:00, 1.72MB/s]
95%|███████████████████████████████████ | 17.5M/18.5M [00:08<00:00, 1.74MB/s]
96%|███████████████████████████████████▍ | 17.7M/18.5M [00:08<00:00, 1.79MB/s]
97%|███████████████████████████████████▊ | 17.9M/18.5M [00:08<00:00, 1.82MB/s]
98%|████████████████████████████████████▏| 18.1M/18.5M [00:08<00:00, 1.83MB/s]
99%|████████████████████████████████████▌| 18.3M/18.5M [00:08<00:00, 1.83MB/s]
100%|████████████████████████████████████▉| 18.5M/18.5M [00:08<00:00, 1.85MB/s]
0%| | 0.00/18.5M [00:00<?, ?B/s]
100%|█████████████████████████████████████| 18.5M/18.5M [00:00<00:00, 67.0GB/s]
BNCI2014-009-WithinSession: 50%|█████ | 1/2 [00:18<00:18, 18.08s/it]
0%| | 0.00/18.5M [00:00<?, ?B/s]
0%| | 32.8k/18.5M [00:00<01:05, 282kB/s]
1%|▏ | 104k/18.5M [00:00<00:38, 478kB/s]
1%|▌ | 249k/18.5M [00:00<00:22, 825kB/s]
3%|█ | 528k/18.5M [00:00<00:12, 1.44MB/s]
5%|██ | 984k/18.5M [00:00<00:07, 2.43MB/s]
7%|██▍ | 1.24M/18.5M [00:00<00:07, 2.32MB/s]
8%|██▉ | 1.47M/18.5M [00:00<00:08, 2.10MB/s]
9%|███▍ | 1.69M/18.5M [00:00<00:08, 1.96MB/s]
10%|███▊ | 1.89M/18.5M [00:01<00:08, 1.95MB/s]
11%|████▏ | 2.09M/18.5M [00:01<00:08, 1.95MB/s]
12%|████▌ | 2.29M/18.5M [00:01<00:08, 1.93MB/s]
14%|█████ | 2.51M/18.5M [00:01<00:08, 1.98MB/s]
15%|█████▍ | 2.71M/18.5M [00:01<00:08, 1.95MB/s]
16%|█████▊ | 2.91M/18.5M [00:01<00:08, 1.81MB/s]
17%|██████▏ | 3.09M/18.5M [00:01<00:09, 1.59MB/s]
18%|██████▌ | 3.26M/18.5M [00:01<00:09, 1.56MB/s]
19%|██████▉ | 3.47M/18.5M [00:01<00:08, 1.69MB/s]
20%|███████▍ | 3.70M/18.5M [00:02<00:08, 1.82MB/s]
21%|███████▊ | 3.89M/18.5M [00:02<00:07, 1.85MB/s]
22%|████████▏ | 4.08M/18.5M [00:02<00:07, 1.82MB/s]
23%|████████▌ | 4.26M/18.5M [00:02<00:07, 1.82MB/s]
24%|█████████ | 4.54M/18.5M [00:02<00:06, 2.09MB/s]
26%|█████████▌ | 4.81M/18.5M [00:02<00:06, 2.26MB/s]
28%|██████████▏ | 5.10M/18.5M [00:02<00:05, 2.45MB/s]
29%|██████████▊ | 5.40M/18.5M [00:02<00:05, 2.55MB/s]
31%|███████████▎ | 5.66M/18.5M [00:02<00:05, 2.55MB/s]
32%|███████████▊ | 5.92M/18.5M [00:03<00:04, 2.56MB/s]
34%|████████████▍ | 6.21M/18.5M [00:03<00:04, 2.64MB/s]
35%|████████████▉ | 6.49M/18.5M [00:03<00:04, 2.67MB/s]
37%|█████████████▌ | 6.78M/18.5M [00:03<00:04, 2.72MB/s]
38%|██████████████ | 7.06M/18.5M [00:03<00:04, 2.74MB/s]
40%|██████████████▋ | 7.35M/18.5M [00:03<00:04, 2.76MB/s]
41%|███████████████▏ | 7.63M/18.5M [00:03<00:03, 2.75MB/s]
43%|███████████████▊ | 7.90M/18.5M [00:03<00:03, 2.74MB/s]
44%|████████████████▎ | 8.18M/18.5M [00:03<00:03, 2.59MB/s]
46%|████████████████▉ | 8.50M/18.5M [00:03<00:03, 2.75MB/s]
47%|█████████████████▌ | 8.78M/18.5M [00:04<00:03, 2.74MB/s]
49%|██████████████████ | 9.06M/18.5M [00:04<00:03, 2.73MB/s]
50%|██████████████████▋ | 9.35M/18.5M [00:04<00:03, 2.78MB/s]
52%|███████████████████▎ | 9.67M/18.5M [00:04<00:03, 2.88MB/s]
54%|████████████████████ | 10.0M/18.5M [00:04<00:02, 3.05MB/s]
56%|████████████████████▋ | 10.3M/18.5M [00:04<00:02, 3.05MB/s]
58%|█████████████████████▎ | 10.7M/18.5M [00:04<00:02, 3.11MB/s]
59%|██████████████████████ | 11.0M/18.5M [00:04<00:02, 3.15MB/s]
61%|██████████████████████▋ | 11.4M/18.5M [00:04<00:02, 3.18MB/s]
63%|███████████████████████▎ | 11.7M/18.5M [00:04<00:02, 3.22MB/s]
65%|████████████████████████ | 12.0M/18.5M [00:05<00:02, 3.22MB/s]
67%|████████████████████████▋ | 12.4M/18.5M [00:05<00:01, 3.23MB/s]
69%|█████████████████████████▎ | 12.7M/18.5M [00:05<00:01, 3.22MB/s]
70%|██████████████████████████ | 13.0M/18.5M [00:05<00:01, 3.20MB/s]
72%|██████████████████████████▋ | 13.4M/18.5M [00:05<00:01, 3.20MB/s]
74%|███████████████████████████▍ | 13.7M/18.5M [00:05<00:01, 3.23MB/s]
76%|████████████████████████████ | 14.0M/18.5M [00:05<00:01, 3.22MB/s]
83%|██████████████████████████████▋ | 15.4M/18.5M [00:05<00:00, 6.19MB/s]
92%|██████████████████████████████████▏ | 17.1M/18.5M [00:05<00:00, 9.43MB/s]
0%| | 0.00/18.5M [00:00<?, ?B/s]
100%|█████████████████████████████████████| 18.5M/18.5M [00:00<00:00, 58.8GB/s]
BNCI2014-009-WithinSession: 100%|██████████| 2/2 [00:33<00:00, 16.68s/it]
BNCI2014-009-WithinSession: 100%|██████████| 2/2 [00:33<00:00, 16.89s/it]
Plot Results#
Here we plot the results to compare the two pipelines
fig, ax = plt.subplots(facecolor="white", figsize=[8, 4])
sns.stripplot(
data=results,
y="score",
x="pipeline",
ax=ax,
jitter=True,
alpha=0.5,
zorder=1,
palette="Set1",
)
sns.pointplot(data=results, y="score", x="pipeline", ax=ax, palette="Set1")
ax.set_ylabel("ROC AUC")
ax.set_ylim(0.5, 1)
plt.show()
Total running time of the script: (0 minutes 36.546 seconds)
Estimated memory usage: 314 MB