Decision support, not a verdict. Risk flags indicate that a student may benefit from timely support and should be reviewed by an educator or advisor. All data shown is synthetic.
Two questions you should ask of any system like this before trusting it.
What this page is for
“Should I trust this thing, and does it treat everyone the same?”
Every number on this page was measured on students the model was never allowed to learn from — that's the only honest way to test it. Nothing here is hard-coded; it is all recomputed each time the model is retrained, including the results that are unflattering.
How to read this: each point is the system's accuracy if you stopped and judged students at that week, using nothing after it. The lines rise because more of the course has happened. What matters is that they start high enough to be useful — a system that only works at the end is useless, because the student has already gone.
ROC-AUC
0.845
Show it one student who dropped out and one who didn't: it correctly picks the riskier of the two about 84% of the time. A coin flip would be 0.50.
In plain English
Show it one student who dropped out and one who didn't. How often does it correctly rank the dropout as the riskier of the two?
How it's worked out
Compare every possible pairing of a dropout and a non-dropout, and count how often the ranking is right.
What it tells you
0.84 means it gets that pairing right 84% of the time.
Why it's here at all
It measures whether the ranking is useful, without depending on where you draw the flag line.
What counts as good
0.5 is a coin flip. 1.0 is perfect. Above 0.8 is genuinely useful.
Recall
0.816
Of the students who really did drop out, it caught about 82% of them — and missed the rest.
In plain English
Of the students who really dropped out, how many the system managed to catch.
How it's worked out
Correct flags divided by the number of students who actually dropped out.
What it tells you
0.82 means it caught roughly 8 in every 10 students who needed help, and missed 2.
Why it's here at all
This is the number that matters most here — a missed student is a real harm.
Precision
0.602
Of the students it flagged, about 60% really did drop out. The others got an unnecessary check-in.
In plain English
Of the students it flagged, how many really did drop out.
How it's worked out
Correct flags divided by total flags.
What it tells you
0.60 means 6 in every 10 flagged students really were at risk — the other 4 got an unnecessary check-in.
Why it's here at all
It tells you how much of a teacher's time the system wastes.
The honest range on that first number
0.75 – 0.92
We tested on only 82 students, so the true score sits somewhere in this range rather than exactly at 0.845. The width is part of the result.
In plain English
The honest range the true value probably sits in.
How it's worked out
Re-run the scoring thousands of times on random re-samples of the test students, and take the middle 95% of the answers.
What it tells you
“0.84 (0.75–0.92)” means the real number is probably somewhere in that range — we tested on only 82 students.
Why it's here at all
A single number implies a precision we don't have. The width of the range is part of the result.
Why precision is lower than recall
On purpose
We tuned it to catch more struggling students, accepting more false alarms. Missing someone who needed help is a real harm; an unnecessary friendly email costs five minutes.
The same numbers, week by week
A chart is easy to skim but hard to check. Each row is a prediction made using only the weeks up to that point.
| Week | ROC-AUC | F1 | Precision | Recall | Flagged |
|---|---|---|---|---|---|
| Week 2 | 0.789 | 0.639 | 0.523 | 0.821 | 54% |
| Week 3 | 0.804 | 0.627 | 0.538 | 0.750 | 48% |
| Week 4 | 0.831 | 0.687 | 0.590 | 0.821 | 48% |
| Week 5 | 0.847 | 0.696 | 0.585 | 0.857 | 50% |
| Week 6 |
Measured at the final course week so each student is counted exactly once.
Biggest flag-rate gap between groups
0.200
One group gets flagged about 20 percentage points more often than another, in "Academic background". Compare it with the real dropout rates below before calling it unfair.
In plain English
Does the system flag one group far more often than another?
How it's worked out
Work out the share flagged within each group, then take the biggest gap between two groups.
What it tells you
0.13 means one group is flagged 13 percentage points more often than another.
Why it's here at all
A gap isn't automatically unfair — groups can genuinely differ — which is why the real dropout rate is shown next to it.
A single probability tells an advisor what the model thinks but not how firmly. When repeated stochastic passes disagree, the system says so instead of presenting a confident-looking number it cannot support.
Average confidence
90.3%
Each score is produced 30 times with small random changes. On average those runs agree this closely.
In plain English
How much the model agrees with itself.
How it's worked out
The model is run 30 times with small random changes switched on. If all 30 runs land close together, confidence is high; if they scatter, it's low.
What it tells you
High confidence means the score is stable. Low confidence means the model is genuinely unsure about this particular student.
Why it's here at all
A score with no confidence attached invites more trust than it deserves.
What counts as good
Higher is steadier — but low confidence is honesty, not failure.
Calibration error
In plain English
We chose on purpose to catch more struggling students, accepting more false alarms.
How it's worked out
The flag line is set to favour recall — and capped so the system can never flag more than 40% of a cohort.
What it tells you
More unnecessary check-ins, fewer missed students.
Why it's here at all
Missing someone who needed help is worse than sending a friendly email that turned out to be unnecessary. The cap exists because a list of 300 names is not a signal.
| 0.862 |
| 0.719 |
| 0.639 |
| 0.821 |
| 44% |
| Week 7 | 0.875 | 0.730 | 0.657 | 0.821 | 43% |
| Week 8 | 0.876 | 0.767 | 0.719 | 0.821 | 39% |
“Flagged” is the share of the cohort that would appear on a teacher's list that week. It grows as the course runs because more students have had time to fall behind.
What counts as good
Closer to 0 is more even.
Biggest gap in who it gets wrong
0.333
How unevenly the system's mistakes fall across groups. You can flag every group equally often and still make all your errors on one of them — this is the number that catches that.
In plain English
When the system gets it wrong, is it wrong equally often for every group?
How it's worked out
Compare, across groups, how often it correctly catches at-risk students and how often it falsely flags safe ones.
What it tells you
A large gap means one group bears more of the system's mistakes.
Why it's here at all
You can flag every group equally often and still make all your errors on one of them.
What counts as good
Differences near 0, ratios near 1.
Differences are best at 0.0; ratios are best at 1.0. Both scales are shown because the paper states equalised odds as a ratio.
Gaps computed over reliable groups only — 2 of 4 groups have at least 15 students. Faded bars are groups too small to be statistically reliable.
The ground truth the model is trying to match. A flag-rate gap that mirrors a base-rate gap is a different situation from one that does not.
| Group | n | Flag rate | Actual rate | TPR | FPR | F1 |
|---|---|---|---|---|---|---|
| female | 34 | 35.3% | 35.3% | 0.83 | 0.09 | 0.83 |
| male | 43 | 39.5% | 27.9% | 0.83 | 0.23 | 0.69 |
| nonbinarytoo small to be reliable | 1 | 100.0% | 100.0% | 1.00 | — | 1.00 |
| prefer not to saytoo small to be reliable | 4 | 50.0% | 75.0% | 0.67 | 0.00 | 0.80 |
The paper's third criterion, alongside demographic parity and equalised odds: similar students should be treated similarly.
88.2%
Consistency
How closely each student's score matches their 5 nearest neighbours in behaviour space — demographics are deliberately excluded from the distance, so two students who behave alike but differ demographically are neighbours by construction.
0.0e+0
Counterfactual change · satisfied
Largest change in any student's score when a demographic field is rewritten to every possible value. Zero by construction when sensitive attributes are excluded from model inputs. Measured rather than assumed, so the exclusion is verified end to end.
Reading these numbers
How fairness is handled
0.038
When the system says "60%", the true figure is off by about 4 percentage points on average — close enough that the percentages can be read literally.
In plain English
Whether “60%” actually means 60%.
How it's worked out
Group all predictions by their stated percentage, then check what fraction of each group really dropped out, and measure the gap.
What it tells you
0.04 means the stated percentages are off by about 4 points on average — close enough to read literally.
Why it's here at all
Without it, the model's numbers rank students correctly but the percentages are meaningless.
What counts as good
Below 0.05 is good.
Handed to a human instead
28.4%
For this share of predictions the system declines to stand behind its own answer — 9% because its runs disagreed, 24% because the score sat on the borderline.
In plain English
The system saying: don't act on me alone for this student.
How it's worked out
Triggered when the 30 runs disagree too much, or when the score sits right on the borderline where a nudge either way would flip the decision.
What it tells you
A prompt to look at the student's situation yourself, not a conclusion.
Why it's here at all
A system that never admits doubt gets trusted in exactly the cases where it shouldn't be.
If the uncertainty estimate is doing its job, the bars should rise from left to right and track the dashed line.
Monotonic, so ROC-AUC and all ranking metrics are unchanged. It only puts the probabilities on a scale where 0.30 means roughly a 30% chance.