Reproducible Keras Training: Seeds, Determinism, and the Run Record

"We got 94.1 last month" is not a result you can act on if nobody can produce 94.1 again. On review engagements, irreproducible training is the finding that blocks everything downstream: you cannot attribute an improvement, you cannot bisect a regression, and you cannot defend a model to a regulator or a customer. The good news is that reproducibility is almost entirely a tooling and discipline problem, and a day of work usually settles it.

This post covers what to pin in a Keras 3 project, what determinism costs, and the minimum run record worth keeping.

Decide which kind of reproducibility you need

There are three levels, and they cost very different amounts:

1. Reproducible conclusions. Rerun the experiment with new seeds and the decision holds — variant B still beats variant A. This is what matters for modeling decisions, and it is the level most teams actually need.
2. Reproducible runs. Same seed, same code, same data, same hardware gives the same metrics to a few decimal places. This is what you need to bisect a regression.
3. Bit-identical runs. Every float matches. Needed for audit trails, some regulated settings, and debugging numerical divergence between two machines.

Level 1 comes from methodology: fixed splits, repeated seeds, and comparing means rather than single runs. Levels 2 and 3 come from the mechanics below. Pick deliberately — full determinism on GPU costs throughput, and paying for it on every experiment when you only need level 1 is waste.

Seed everything, once, before anything else

Keras 3 gives you one call that covers the Python, NumPy, and active-backend RNGs:

import keras
keras.utils.set_random_seed(1337)

Put it at the top of the entry point, before model construction and before any dataset object is created. Seeding after a layer is built means the initializers already drew from an unseeded stream.

That call does not cover everything:

• tf.data shuffling. Pass an explicit seed= to Dataset.shuffle(), and set reshuffle_each_iteration deliberately — True is right for training, but it means epoch order differs unless the seed is fixed.
• Augmentation layers. Keras preprocessing layers take a seed argument. Set it on each one.
• Backend-specific randomness. Under JAX, explicit PRNG keys are the model; pass them through rather than relying on globals. Under PyTorch with a DataLoader, set worker_init_fn and a generator, or worker processes reseed themselves independently.
• Python hash ordering. PYTHONHASHSEED affects set and dict iteration order, which can affect file ordering. Set it in the environment, not in the script — by the time Python runs your code it is too late.

GPU determinism, and what it costs

Even with every seed fixed, GPU training usually is not bit-identical between runs. The reason is not the RNG; it is nondeterministic kernels. Atomic-add reductions, some cuDNN convolution algorithms, and autotuned kernel selection accumulate floating-point results in a different order each run, and floating-point addition is not associative. The differences are tiny per step and compound over training.

On the TensorFlow backend:

import os
os.environ["TF_DETERMINISTIC_OPS"] = "1"
os.environ["TF_CUDNN_DETERMINISTIC"] = "1"
import tensorflow as tf
tf.config.experimental.enable_op_determinism()

On PyTorch, torch.use_deterministic_algorithms(True) with torch.backends.cudnn.deterministic = True and benchmark = False. Under JAX, XLA is largely deterministic for a fixed compilation, but a changed device count or input shape changes the compiled program — so pin both.

Expect a real throughput cost: in our measurements it commonly lands somewhere between 5% and 30% on convolutional workloads, depending on which kernels lose their fast path. Deterministic algorithms also raise errors for ops that have no deterministic implementation, which is a useful way to find out what your model is actually doing. Our default: determinism on for the final, reported runs and for any bisect; off for exploratory sweeps, where the seed and the run record are enough.

Pin the things that are not code

Seeds are the part teams remember. These are the parts that break reproducibility six months later:

• Data version. A dataset directory that people add files to is not a version. Snapshot it — content-hashed manifests, a dataset registry, or at minimum a frozen file list with checksums recorded per run. "We retrained on the current data" is not a reproducible statement.
• The split. Store the actual split assignment as an artifact, not the seed that generated it. Splitting logic changes; the file does not.
• Environment. Pin Keras, the backend, CUDA/cuDNN, and driver versions in a lockfile or container image. A cuDNN minor version bump can shift the fourth decimal place, and you will spend a day on it.
• Preprocessing. If normalization statistics are computed from the training set, save them with the checkpoint. Recomputing them from a slightly different dataset is a silent, untraceable change.

The run record

Every training run should write one record, automatically, with no human in the loop. Minimum fields:

• Git commit of the training code, and whether the tree was dirty.
• Full resolved config (all hyperparameters, after defaults are applied — not the CLI flags you typed).
• Seeds, determinism flags, and backend name.
• Dataset identifier or manifest hash, and the split artifact ID.
• Environment: framework and CUDA versions, device type and count.
• Metrics per epoch, final metrics, and the checkpoint path.

MLflow, Weights & Biases, or a JSON file per run in object storage all work. The tool matters far less than the rule: a run that did not write a record did not happen, and its number does not get quoted in a meeting.

A sensible sequence

1. Add keras.utils.set_random_seed() and explicit seeds on shuffle and augmentation layers.
2. Freeze the split to an artifact; hash the dataset.
3. Add the run record, and make the training script fail if the git tree is dirty in CI.
4. Rerun a known job twice with the same seed. If metrics differ beyond noise, enable determinism and find the nondeterministic op.
5. Measure the throughput cost of determinism once, then decide where you keep it on.

None of this improves accuracy by a point. It changes what every later measurement is worth — which, over a project, is the larger number.