Every team we review has a hyperparameter story, and most of them are the same story: someone ran a search, the search reported a great validation number, and the model that shipped did not reproduce it. The search was not wrong about the number. It was wrong about what the number meant.
Hyperparameter tuning is the cheapest place in a project to fool yourself, because the tool is explicitly optimizing a metric you chose on a split you chose. Give it a leaky split or a noisy metric and it will find the leak and the noise faster than it finds a better model. This post is the methodology we use with KerasTuner: what to search, what not to search, how to keep the result honest, and when to stop.
Baseline first, and write the number down
Do not start a search without a reference point. Train one model with sensible defaults — Adam at 1e-3, a standard architecture for the problem, the augmentation you already trust — and record its metric with a confidence interval, not a single value.
That number does two jobs. It tells you whether the search found anything (a 0.3-point gain over a run-to-run spread of 0.8 points is not a finding), and it tells you when to stop paying for compute. On the tabular problems we see, the honest comparison is often not against another network at all but against gradient boosting; that argument is in Tabular Data in Keras and it applies here. A search that beats your neural baseline but loses to LightGBM has told you something useful, just not what you hoped.
Fix your seeds and your data order before you search, too. If two identical configurations disagree by more than the gap you are chasing, you are tuning noise. Our notes on that are in Reproducible Keras Training.
What is worth searching
Search budget is finite, and the parameters do not matter equally. Roughly in order of payoff:
- Learning rate. Almost always the highest-variance knob. Search it log-uniform over two or three decades.
- Regularization strength. Weight decay, dropout rate, label smoothing — whatever your overfitting story calls for.
- Capacity. Width and depth, as a small number of coupled choices rather than a per-layer free-for-all.
- Batch size, only jointly with learning rate, since the two trade off.
What is usually not worth searching: optimizer identity (pick AdamW and move on), activation function, initializer, and anything where the literature already has a defensible default. Every dimension you add multiplies the space, and a search over twelve hyperparameters with a budget of forty trials is a random sample, not an optimization.
Be equally suspicious of architecture search. Tuning the number of units in three dense layers is fine. Searching over block types, kernel sizes, and normalization placement at the same time is a research project with a compute bill, and it rarely beats a known-good backbone plus transfer learning.
A search space that stays readable
KerasTuner's model-builder form keeps the space next to the model, which makes it reviewable:
import keras_tuner as kt
import keras
def build(hp):
lr = hp.Float("lr", 1e-4, 1e-2, sampling="log")
width = hp.Choice("width", [64, 128, 256])
depth = hp.Int("depth", 1, 3)
dropout = hp.Float("dropout", 0.0, 0.5, step=0.1)
inputs = keras.Input(shape=(N_FEATURES,))
x = inputs
for _ in range(depth):
x = keras.layers.Dense(width, activation="relu")(x)
x = keras.layers.Dropout(dropout)(x)
outputs = keras.layers.Dense(1)(x)
model = keras.Model(inputs, outputs)
model.compile(
optimizer=keras.optimizers.AdamW(lr),
loss=keras.losses.BinaryCrossentropy(from_logits=True),
metrics=[keras.metrics.AUC(name="pr_auc", curve="PR")],
)
return model
Four hyperparameters, each with a justification. sampling="log" on the learning rate matters: with linear sampling, most of your trials land in the top decade and you never see 3e-4.
Note the metric. objective should be the thing the product cares about, evaluated the way production evaluates it — PR-AUC on an imbalanced problem, not accuracy. If the decision downstream is a hard yes/no, tuning on a ranking metric and then picking a threshold afterwards is the right split of concerns; see Calibration and Thresholds.
Pick a tuner, and pick early stopping deliberately
tuner = kt.Hyperband(
build,
objective=kt.Objective("val_pr_auc", direction="max"),
max_epochs=40,
factor=3,
hyperband_iterations=1,
executions_per_trial=2,
directory="tuning",
project_name="risk_v3",
seed=1234,
)
tuner.search(
train_ds,
validation_data=val_ds,
callbacks=[keras.callbacks.EarlyStopping("val_pr_auc", patience=5, mode="max")],
)
Three practical choices in there.
Hyperband over random search when epochs are expensive. It gives short budgets to many configurations, then promotes survivors. The catch is that it assumes early performance predicts final performance, which is false for configurations that need a long warmup — a low learning rate with a schedule can look terrible at epoch 3 and win at epoch 40. If your training curves cross late, use RandomSearch or BayesianOptimization with a full budget instead. Random search remains a strong default and is much easier to reason about.
executions_per_trial=2 or more. This is the line most teams skip, and it is the one that stops you from shipping noise. Each configuration is trained more than once and the results averaged. It doubles cost and roughly halves the number of spurious winners. If you cannot afford it, at least retrain the top five configurations three times each before choosing between them.
seed on the tuner, and a fresh project_name per experiment. KerasTuner resumes from directory/project_name, which is a convenience right up until it silently mixes results from an older, different search space into today's leaderboard. Treat the project name as an experiment ID.
The split is where searches go wrong
A tuner with a few hundred trials is a very effective overfitter of your validation set. Two defenses, both cheap:
Three splits, not two. Train on train, select with validation, and report on a test set that the tuner never saw. The gap between the winning trial's validation score and its test score is the amount of validation overfitting you just did. We expect to see some; we get suspicious when it exceeds the baseline's run-to-run spread.
A split that matches deployment. Random splits leak on grouped data (multiple rows per patient, per store, per device) and on anything temporal. Group-aware or time-ordered splits, described in Time-Series Forecasting in Keras, are not optional here: a leaky split does not merely inflate one number, it steers the entire search toward configurations that exploit the leak.
If the dataset is small — a few thousand examples — a single validation split is too noisy to rank forty configurations. Use k-fold inside the objective (average the folds per trial) and accept the k× cost, or accept that you can only distinguish large differences.
Read the results, do not just take the top row
best_hps = tuner.get_best_hyperparameters(num_trials=10)
for hp in best_hps:
print(hp.values)
Look at the top ten, not the top one. What you want to see is a region: nine of ten winners with learning rates between 6e-4 and 2e-3 is a robust setting. If the top ten are scattered across the whole space, your metric is dominated by noise and the leaderboard ordering is close to arbitrary — no amount of extra trials fixes that, but executions_per_trial and a larger validation set might.
Also check the edges. If the best learning rate sits on the boundary of the range you allowed, the search wanted to go further and you clipped it. Widen the range and rerun rather than accepting a corner solution.
Then retrain the chosen configuration from scratch on train plus validation, with the epoch count the tuner found, and evaluate once on test. The tuner's internal best model was trained under early stopping on the split you are now folding in; a clean retrain is the artifact you actually ship.
What to record
A tuning run is an experiment, and the result is worthless six months later without its context. We keep, per search: the search space definition (the build function, in version control), the objective and split definition, tuner type and budget, executions_per_trial, the seed, library versions, and the full trial table exported as CSV — not just the winner. When someone asks next quarter whether dropout mattered, the trial table answers it in a minute and a rerun costs a day.
Where this usually lands
In most engagements, tuning is worth a bounded budget and no more. Data quality, the loss function, and the split definition move the metric more than the learning rate does, and they move it in ways that hold up in production. A disciplined forty-trial random search over four hyperparameters, run after the baseline and the split are trustworthy, captures most of the available gain.
If a search is the only thing standing between your model and its target, the honest read is usually that the target needs a different model or better data — not another hundred trials.