A model at 0.91 accuracy and a model at 0.93 accuracy can have completely different failure modes, and only one of them may be shippable. The aggregate number is a summary of a distribution you have not looked at. Error analysis is the work of looking at it.
We run this step on nearly every review engagement, usually before touching the architecture. It is cheap — a few hours — and it routinely redirects a team that was about to spend three weeks on a bigger backbone toward a labeling fix or a threshold change that lands in two days.
Why before how
There are only a handful of things you can actually do to a model that is underperforming:
- Fix or add data (more examples, better labels, different sampling).
- Change preprocessing or augmentation so the training distribution matches serving.
- Change the objective — loss, class weights, decision threshold.
- Change capacity or architecture.
- Accept the error and handle it downstream (routing, human review, abstention).
Error analysis exists to tell you which of those five is worth doing. Without it, teams default to option 4 because it is the most interesting one. In our experience it is the right answer maybe a fifth of the time.
Step 1: build a prediction table, not a metric
Everything below works off one artifact: a row per validation example with the fields you need to group by. Build it once, keep it as a file, version it with the run record.
import numpy as np, pandas as pd
probs = model.predict(val_ds, verbose=0)
pred = probs.argmax(axis=1)
conf = probs.max(axis=1)
df = pd.DataFrame({
"id": val_ids, # stable identifier, not row order
"y_true": y_val,
"y_pred": pred,
"conf": conf,
"loss": -np.log(probs[np.arange(len(y_val)), y_val] + 1e-12),
})
df = df.merge(metadata, on="id", how="left") # site, device, capture date, operator, resolution
Two requirements that people skip and regret. First, make the dataset deterministic and unshuffled when you predict — val_ds must yield examples in a known order, or the IDs will not line up with the predictions. Second, join real metadata. The grouping columns are where the value is: capture device, factory line, clinic, season, image resolution, time of day, annotator. Models fail along the seams of the data collection process, and those seams are in the metadata, not in the pixels.
Step 2: per-slice metrics, ranked by recoverable loss
Aggregate accuracy hides slices. Compute the metric per group and rank by how much total error lives there, not by how bad the rate is — a 40% error rate on eleven examples is noise, while a 12% error rate on 4,000 examples is your whole gap.
slice_report = (df.groupby("capture_device")
.agg(n=("id", "size"),
acc=("y_true", lambda s: (df.loc[s.index, "y_pred"] == s).mean()),
mean_loss=("loss", "mean"))
.assign(errors=lambda t: (t.n * (1 - t.acc)).round())
.sort_values("errors", ascending=False))
Read it with two guards. Slices under ~100 examples get wide confidence intervals; attach a binomial interval before you believe a gap. And a slice that is underperforming and underrepresented in training is a sampling problem, which is a data fix, not a model fix.
The decision rule we use: if the worst slices contain more than about a third of total error and they share an identifiable cause, stop the modeling work and fix the data or the sampling. Rebalancing the training set toward a starved slice is usually a one-day change with a measurable effect; the bigger backbone is three weeks and often moves the aggregate while leaving the slice exactly where it was.
Step 3: the ranked confusion inventory
For classification, the confusion matrix is a starting point, but a flat ranked list of (true, predicted, count) pairs is easier to act on, especially past ten classes.
pairs = (df[df.y_true != df.y_pred]
.groupby(["y_true", "y_pred"]).size()
.sort_values(ascending=False).head(15))
The top few pairs almost always fall into one of three buckets, and the bucket determines the fix:
- Genuinely ambiguous classes. Two labels that domain experts disagree on. Merging them, or adding a coarse-then-fine two-stage head, beats any amount of training.
- Label definition drift. The guideline changed partway through annotation. Fix the labels; the model is correctly learning an inconsistent target.
- Real visual or signal similarity. This is the one case where more capacity, higher input resolution, or targeted hard-negative mining genuinely helps.
Sort each top pair's examples by loss and look at the twenty worst. Twenty images, ten minutes. We have never run this and learned nothing.
Step 4: confidence-stratified errors
Split errors by confidence, because the two halves mean different things.
- High-confidence errors (
conf > 0.9, wrong): suspect the label first. These are the strongest signal of annotation noise and near-duplicate leakage between splits, and they are worth routing straight into a label audit. - Low-confidence errors: the model is honestly uncertain. These are candidates for an abstention path — return "unknown" below a threshold and route to human review — rather than for retraining.
If a meaningful share of production traffic can be deferred, an abstention threshold often buys more effective accuracy than any architecture change. Calibrate first, though; raw softmax scores are not probabilities and the threshold you pick on uncalibrated outputs will not hold.
Step 5: Grad-CAM as a hypothesis test, not a gallery
Saliency maps are over-produced and under-used. The failure mode is generating a hundred pretty heatmaps and concluding nothing. Use them to test one specific hypothesis: is the model keying on the artifact rather than the object?
import keras, numpy as np
grad_model = keras.Model(model.inputs,
[model.get_layer(last_conv).output, model.output])
def gradcam(x, class_idx):
import tensorflow as tf
with tf.GradientTape() as tape:
conv_out, preds = grad_model(x[None, ...])
loss = preds[:, class_idx]
grads = tape.gradient(loss, conv_out)[0]
weights = tf.reduce_mean(grads, axis=(0, 1))
cam = tf.nn.relu(tf.tensordot(conv_out[0], weights, axes=1))
return (cam / (tf.reduce_max(cam) + 1e-8)).numpy()
Run it on the twenty worst examples from one confusion pair, with a written prediction of what you expect to see. If attention sits on a timestamp overlay, a ruler, a bed rail, or a watermark, you have a shortcut-learning problem — and the fix is cropping, augmentation, or collection, not depth. If attention sits correctly on the object and the model is still wrong, the signal is genuinely hard and capacity or resolution is back on the table.
One caution: Grad-CAM is coarse and backbone-dependent. Treat it as evidence for a hypothesis you already formed from the slice table, never as the origin of one.
Non-classification tasks
The structure holds; the columns change.
- Time series. Slice by horizon, by series, and by regime. Rank series by contribution to total weighted error. Most forecasting "model problems" are three high-volume series behaving differently from the other nine hundred.
- Detection and segmentation. Decompose into localization versus classification versus duplicate/missed detections, and slice by object size. Small-object recall is usually its own distinct failure, with its own distinct fix (input resolution, anchor or stride settings, tiling).
- Regression. Plot residuals against each metadata column and against the target itself. Heteroscedasticity by slice is the same finding as a bad slice accuracy.
Make it a fixed step, not a heroic one
Write the prediction table and the slice report as a script that runs after every training run and writes to the run directory next to the metrics. Two things follow. You get a diff: which slices moved, and whether the aggregate improvement came from the slice you were targeting or from somewhere unrelated. And the team stops arguing about what the model is bad at, because the table says so.
The cost is a few hours once and a minute per run after that. The payoff is spending your engineering weeks on the right one of those five options.
If you have a Keras model that is close but failing in ways nobody can characterize, a model and pipeline review starts exactly here — the slice table first, the architecture afterward. Get in touch with the problem description and we will tell you what we would look at first.