Most of the deployment writing about Keras stops at the edge: convert to LiteRT, quantize, ship to a device. Plenty of applied work goes the other way — the model stays on a server behind an HTTP or gRPC endpoint, called by an existing backend service, with a latency budget written into somebody's SLA. TensorFlow Serving is still the shortest path from a Keras checkpoint to that endpoint, and it is the part of the pipeline teams most often improvise.
This post covers the four things that decide whether a served model is operable: the export signature, the version directory layout, batching, and how you measure latency before you commit to a number.
Export the inference artifact, not the training one
Keras 3 separates the two deliberately. model.save("model.keras") is the training artifact — architecture, weights, optimizer state, resumable. TF Serving does not read it. What it reads is a SavedModel directory with a serving signature:
model.export("/models/defect_classifier/3")
That writes a SavedModel whose default signature takes the model's input spec and returns its output. Two decisions are worth making explicitly rather than inheriting:
What the signature accepts. If your service sends JPEG bytes, a signature that accepts a float32 tensor of shape (None, 224, 224, 3) pushes decode, resize, and normalization into the calling service. That code will drift from your training preprocessing — not maybe, eventually. Wrap the preprocessing into the exported graph when the input is cheap to describe:
import tensorflow as tf
from keras.export import ExportArchive
@tf.function(input_signature=[tf.TensorSpec([None], tf.string, name="image_bytes")])
def serve_jpeg(image_bytes):
def one(b):
x = tf.io.decode_jpeg(b, channels=3)
x = tf.image.resize(x, [224, 224])
return tf.cast(x, tf.float32) / 255.0
x = tf.map_fn(one, image_bytes, fn_output_signature=tf.float32)
return {"probs": model(x, training=False)}
archive = ExportArchive()
archive.track(model)
archive.add_endpoint(name="serving_default", fn=serve_jpeg)
archive.write_out("/models/defect_classifier/3")
What the signature returns. Return named outputs, and return what the caller actually needs — probabilities plus an argmax label, say, rather than raw logits the client has to softmax correctly. Naming the outputs means you can add a second head later without breaking positional parsing on the client.
Check the result before it leaves your machine:
saved_model_cli show --dir /models/defect_classifier/3 --tag_set serve \
--signature_def serving_default
If the dtypes and shapes there are not exactly what your client sends, fix it now. Signature mismatches surface as an opaque gRPC error at integration time.
A note on backends: TF Serving consumes SavedModel, so the TensorFlow backend is the clean path. If your training runs on the JAX backend, you can still export to SavedModel via model.export() with the TF interop path, but verify it on the target — or serve through a runtime that matches your backend instead. Choose the serving runtime before you choose the training backend, not after.
Version directories are your rollback mechanism
TF Serving's model directory convention is a numbered subdirectory per version:
/models/defect_classifier/
1/ saved_model.pb variables/
2/ saved_model.pb variables/
3/ saved_model.pb variables/
By default the server polls the base path and loads the highest integer it finds. Write the new version to a temporary path and move it in atomically — a half-copied directory that appears mid-poll will be loaded and will fail.
The default "latest" policy is fine for a single-model dev box and wrong for production, because rollback means deleting a directory. Use an explicit model config instead:
model_config_list {
config {
name: "defect_classifier"
base_path: "/models/defect_classifier"
model_platform: "tensorflow"
model_version_policy { specific { versions: 2 versions: 3 } }
version_labels { key: "stable" value: 2 }
version_labels { key: "canary" value: 3 }
}
}
Now both versions are resident, clients can request a label rather than a number, and rollback is a config change instead of a file deletion. Two loaded versions cost roughly two copies of the weights in RAM (and in GPU memory, if you serve on GPU) — budget for it, and keep the retained set small.
Tie the version number to your run record. Version 3 should be traceable to a git commit, a dataset hash, and an evaluation report. A served model whose provenance is "whatever Dana copied up in March" is the same problem as an irreproducible training run, one layer further downstream.
Batching is the throughput knob, and it costs latency
A single inference request usually leaves an accelerator badly underused. Server-side batching lets TF Serving group concurrent requests into one graph execution. Enable it with --enable_batching and a config file:
max_batch_size { value: 32 }
batch_timeout_micros { value: 5000 }
max_enqueued_batches { value: 100 }
num_batch_threads { value: 4 }
The only parameter that really requires thought is batch_timeout_micros: how long the server waits for a batch to fill before running a short one. It is a direct trade of tail latency for throughput. At 5 ms you have given away up to 5 ms of every request's budget; in exchange, under load you may run 4–8× fewer graph executions.
How to set it, concretely:
- Measure your actual arrival rate at peak. If requests arrive at 20/s and your timeout is 5 ms, batches will average about one request — you paid the latency and got nothing.
- Set
max_batch_sizefrom a throughput sweep on the target hardware: batch 1, 4, 8, 16, 32, and find where throughput stops improving. On many CNNs on a single GPU this flattens well before 64. - Set the timeout to the smallest value that fills a useful batch at your peak rate, and no larger than about 10% of your latency budget.
- Re-measure end to end. Batching interacts with
num_batch_threadsand with intra-op parallelism; the model-level sweep is an estimate, not the answer.
If traffic is genuinely low and latency is the whole requirement, leave batching off. It is not free, and "on by default" is not a reason.
Measure the latency you are going to promise
The number that matters is p99 end to end, measured from the calling service, under production-like concurrency. Means are not useful here: a model with a 20 ms mean and a 400 ms p99 will page somebody.
A workable measurement protocol:
- Load-test with a fixed request rate, not a closed loop of "as fast as possible" — open-loop load reveals queueing that closed-loop hides.
- Report p50, p95, p99 separately for the client's total time and the server's own
:predicthandling time. The gap is serialization, network, and queueing, and it is frequently larger than the model. - Include payload size in the test. Sending raw
float32tensors as JSON over REST is the most common avoidable latency problem we see; gRPC with binary tensors, or passing compressed image bytes into an in-graph decode, routinely cuts request time by a large fraction on image models. - Test at the concurrency you expect, plus 2×. Latency curves are flat until they are not.
TF Serving exposes Prometheus metrics via --monitoring_config_file; export request counts, latency histograms, and batch size distribution from day one. The batch-size histogram is the single most informative graph for tuning the settings above, and nobody adds it until they need it.
A serving checklist
- Signature verified with
saved_model_cli, preprocessing decided and documented. - Version directories written atomically, explicit version policy, labels for
stableandcanary. - Version number traceable to commit, data hash, and evaluation report.
- Batching configured from a measured sweep, or deliberately off.
- p50/p95/p99 measured open-loop at expected peak, with payload format decided on evidence.
- Prometheus metrics scraped, including batch size.
None of this changes model accuracy. It decides whether the model is operable — whether you can roll back in a minute, explain a latency spike, and say what version produced a given prediction. That is usually the difference between a model that stays in production and one that gets quietly switched off.
If you have a Keras model that works in a notebook and needs a serving path with a latency budget attached, that is the kind of problem our Deployment & MLOps work starts from — send us the details through the contact form and a senior engineer will reply.