Is this model really better?

STEP 1

Warm-up: no models yet

◆ Why start with this question? So that you catch the idea the whole task rests on before you learn a single technical word. Get this one right and the remaining nine steps are just the mathematics of the same idea.

Your taskAnswer the question below. Getting it wrong is fine. Nobody is grading you here; you are measuring your own intuition.

Two people played 10 rounds of rock-paper-scissors. One won 6, the other won 4.
Is the first one the better player?

STEP 2

Set up the task: what are you comparing with what?

◆ Why this step exists You cannot interpret the result without knowing what the two things actually do. A p-value never tells you “why”, only “yes or no”. The “why” is on you.

Your taskPick a dataset and two different models. Under each choice you get a plain-English description of what it is.

Dataset

Model A

Model B

STEP 3

Predict, before you see the result

◆ Why we ask you to predict first Two reasons. One: predicting and then seeing produces durable learning in a way that only seeing does not; your brain binds the information while it is trying to generate the answer. Two: what this site really measures is not your knowledge but how reliable your intuition is. Your wrong predictions are your most valuable data.

Your taskPick an option and lock it in. You will see the result at step 8, and you cannot change it before then.

STEP 4

First evidence: what everybody does

◆ Why we do this Because this is exactly what gets done in the field, in papers, in blog posts and in Kaggle notebooks: split the data 70% train / 30% test, fit both models, look at the accuracies, take the higher one. We will do that first, and then show you why it is not enough.

Your taskJust watch. Look at how each model carves up the data and how many points it misses.

—

class 0class 1class 2 a point the model got wrong

STEP 5

The shake: we rattle the number you trusted

◆ Why we do this The split at step 4 was random. A random choice produced a single number and you were about to decide on the strength of it. Now we take the same data and the same models and change nothing but the random split. The models do not change. The data does not change. Only who is in training and who is in test.

Your taskPress “Split again” at least 5 times. Watch where the accuracies go on each press.

0 splits

STEP 6

So how many times should you repeat it?

◆ Why not just repeat it 100 times at random? Because the repeats are not independent of each other. They all reuse the same 400 rows over and over, so the same points keep appearing in both training and test. A hundred repeats do not give you a hundred independent pieces of evidence; they count the same evidence a hundred times and lead you to believe you are certain when you are not. This is why the structure of the repeats matters.

Ethem Alpaydin's answer is 5x2cv: cut the data in half, train on the first half and test on the second, then swap the roles. That is one repeat. Do it 5 times with different halves. Ten fits in total, and every row is balanced within itself.

✦ The key point On every fold both models see the same split. That is what “paired” means: the luck of the split hits both of them at once, so taking the difference cancels most of that luck. What we are measuring is no longer “how good is A” but “how consistent is the gap between A and B”.

STEP 7

Run it: ten fits, real code

◆ Why we show you the code Because this is not an animation. The code below runs on your machine, with real scikit-learn. You can copy it into your own project. There is no black box.

# 5 repeats x 2 folds. In 2-fold CV the second fold is already the swap of the first:
# each half is training once and test once.
for i in range(5):
    cv = StratifiedKFold(n_splits=2, shuffle=True, random_state=i)
    for j, (tr, te) in enumerate(cv.split(X, y)):
        errA = 1 - A.fit(X[tr], y[tr]).score(X[te], y[te])
        errB = 1 - B.fit(X[tr], y[tr]).score(X[te], y[te])
        p[i][j] = errA - errB          # paired difference, same split

Your taskPress the button and run the ten fits. Then look at the difference column: is the sign always the same, or does it flip?

STEP 8

Pick the right test

◆ Why “the right test” and not just “a test”? Every statistical test comes with a design assumption. Apply the wrong one and you still get a number, a p-value still gets written down, and the reviewer may well not notice, but that number is lying. This step sits at the heart of the most common and most silent mistake in ML.

What you haveTen paired differences · the models were refitted on every fold · the splits overlap, so they are not independent.

STEP 9

What is the p-value saying?

◆ Why this gets its own step Because the p-value is the most misunderstood number in the history of science. Choosing the right test and then writing the result up in the wrong sentence is no better than never testing at all.

Your taskPick what the p-value above actually means. Three of the options are wrong sentences that get written all the time.

STEP 10

The report: sign your name to it

◆ Why the last step is writing Because the output of the work is not a p-value, it is a sentence. The organisation will read that sentence, the reviewer will read it, the team will read it. One badly built sentence throws away an entire correct analysis.