Transfer Learning That Holds Up: A Fine-Tuning Recipe for Small Datasets

Most applied vision problems outside big tech come with a few hundred to a few tens of thousands of labeled images — enough for transfer learning, nowhere near enough for training from scratch, and small enough that sloppy methodology produces results that evaporate in production. This is the fine-tuning recipe we use in Keras, with the reasoning behind each step.

Step 0: an honest evaluation setup

Before any training: fix the split, and check it for leakage. On small datasets, near-duplicate images landing on both sides of a random split are the norm, not the exception — burst shots, frames from one video, the same part photographed twice on one bench. Split by the natural grouping unit (session, patient, machine, field, day), not by image. A model that "hit 96%" on a leaky split has told you nothing except that you'll be having an awkward conversation after deployment.

Pick the metric that matches the operating point — on imbalanced problems that is usually per-class recall at a chosen precision, or PR-AUC. Plain accuracy on a 95/5 class split is a vanity number.

Step 1: a linear probe as the baseline

Load a pretrained backbone, freeze it entirely, and train only a small head:

base = keras.applications.EfficientNetV2B0(
    include_top=False, weights="imagenet", pooling="avg"
)
base.trainable = False

inputs = keras.Input(shape=(224, 224, 3))
x = base(inputs, training=False)
x = keras.layers.Dropout(0.2)(x)
outputs = keras.layers.Dense(num_classes, activation="softmax")(x)
model = keras.Model(inputs, outputs)

This "linear probe" trains in minutes, cannot catastrophically overfit the backbone, and sets the floor every later experiment must beat. It is also diagnostic: if a frozen backbone already performs well, the features transfer, and careful fine-tuning will likely add a few points. If the probe performs at chance, the domain gap is large and no learning-rate schedule will rescue it — the conversation should turn to data before architecture.

Note the training=False in the call. It runs the backbone in inference mode, which matters in the next step.

Step 2: unfreeze gradually, with a much smaller learning rate

Fine-tuning is where small datasets get destroyed by large gradients. Three rules:

1. Unfreeze from the top. Later layers encode task-specific features and adapt usefully; early layers encode edges and textures that transfer as-is. Unfreeze the top block or two first, and extend downward only if validation improves.
2. Drop the learning rate by 10–100× relative to head training — around 1e-5 with Adam is a sensible start. The pretrained weights are a good solution already; the job is to nudge them, not relearn them.
3. Keep BatchNorm frozen. With batch sizes small datasets force on you, letting BatchNorm update its statistics mid-fine-tune shifts the very distribution the pretrained weights expect, and accuracy quietly degrades. Calling the base model with training=False keeps BN in inference mode even after base.trainable = True — this is the single most common fine-tuning bug we find in review.

base.trainable = True
for layer in base.layers[:-30]:
    layer.trainable = False

model.compile(
    optimizer=keras.optimizers.Adam(1e-5),
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"],
)

Step 3: augmentation that respects the domain

Augmentation is regularization, and on small datasets it does more work than any architectural choice. But it must preserve label semantics for your domain: horizontal flips are free for natural photos and wrong for text or lateral X-rays; aggressive hue shifts are harmless for shape-driven classes and destructive when color carries the label. Start from a modest set — flips where valid, small rotations and translations, mild brightness/contrast jitter — as preprocessing layers inside the model, so they run only in training mode and ship inside the artifact:

augment = keras.Sequential([
    keras.layers.RandomFlip("horizontal"),
    keras.layers.RandomRotation(0.05),
    keras.layers.RandomContrast(0.1),
])

Add stronger schemes (RandAugment, MixUp) only after the simple set is measured, one change at a time.

Step 4: early stopping, and restraint

Small-data fine-tuning overfits in few epochs. EarlyStopping(patience=5, restore_best_weights=True) on the validation metric, plus ReduceLROnPlateau, covers most cases. Resist the urge to iterate against the validation set dozens of times — with a few hundred validation examples, you will eventually overfit to the split through your own choices. Keep a small untouched test set for the final number, and report it once.

Step 5: change one thing at a time

The recipe above has maybe six knobs that matter: how much to unfreeze, learning rate, augmentation strength, dropout, class weighting, input resolution. Run them as controlled comparisons against the linear-probe baseline with fixed seeds and a fixed split, and record every run. On small datasets, run-to-run variance is large enough that a single comparison can mislead; when a decision matters, repeat it across seeds and compare means.

None of this is glamorous, and that is rather the point. On small datasets, disciplined methodology — honest splits, a baseline, frozen BatchNorm, one change at a time — is routinely worth more than any exotic architecture, and it is the part that survives contact with production.