Server-side inference cost is usually a memory-bandwidth problem, not a FLOPs problem. A model whose weights are 4 GB of float32 spends most of its per-token or per-request time moving those weights from HBM into compute, and halving or quartering that number is the single largest lever most teams have. That is the case for quantization, and it is why we now reach for it earlier in engagements than we used to.
What changed is the tooling. Quantizing a Keras model used to mean leaving Keras: export a SavedModel, hand it to the LiteRT converter or a vendor toolchain, then re-verify everything on the far side. We wrote about that path in Exporting Keras Models to LiteRT and ONNX, and it is still the right path for edge targets. But Keras 3 also ships an in-process route:
model.quantize("int8")
model.save("model_int8.keras")
One call, model mutated in place, still a Keras model, still callable with model.predict. This post is about what that call actually does, what it does not do, and how to decide whether the accuracy you pay is worth the memory you save.
What model.quantize() touches
The important thing to understand up front: this is post-training weight quantization, applied selectively, not a whole-graph conversion.
Keras walks the layer tree and quantizes the layer types that implement a quantized path — in practice the big matrix-multiply layers: Dense, EinsumDense, and Embedding, which is where the parameter count lives in transformer-style models. Weights are stored as int8 (or int4) with per-channel scales; at call time the layer dequantizes or runs an integer kernel depending on backend support. Layers without a quantized implementation — normalization, activations, most convolutions at the time of writing — are left in their original dtype and keep working. Nothing errors; they are simply skipped.
This has two consequences worth internalizing before you benchmark.
First, the win is concentrated and predictable on transformer-ish architectures and much smaller on a ConvNet, where most parameters sit in layers the quantizer currently passes over. If you quantize a ResNet and see a 4% memory reduction, the API is not broken — your parameters are simply not where it looks.
Second, model size drops more reliably than latency improves. Memory is a direct function of the stored weights. Speed depends on whether your backend and hardware have a fast integer kernel for that layer shape; if the runtime dequantizes to compute, you pay an unpacking cost and may land slower than where you started at small batch sizes. Measure both, separately.
You can also be selective. Quantize a submodel or an individual layer rather than the whole thing:
backbone.quantize("int8") # leave the task head alone
model.get_layer("decoder_dense").quantize("int8")
That is the usual escape hatch when one sensitive block accounts for most of the accuracy loss.
int8, int4, float8: picking a mode
Treat these as a ladder and stop at the first rung that meets your budget.
int8 is the default answer. Roughly 4× smaller weights against float32, 2× against bfloat16, and on well-behaved models the task-metric loss is often under a point. Start here.
int4 roughly halves memory again and costs noticeably more accuracy. It earns its place when a model has to fit in a specific amount of VRAM — one GPU instead of two changes your bill more than a fractional metric change changes your product. Verify support in your Keras version before planning around it; int4 arrived later than int8 and layer coverage is narrower.
float8 is a different animal: it targets training and inference throughput on hardware with native FP8 support (H100-class and newer), not primarily memory savings on commodity GPUs. If you are not on that hardware, it is not your rung.
And there is a mode people forget to consider: none. Casting to bfloat16 is half the memory of float32 for free, with essentially no accuracy question to answer. If bfloat16 gets you inside budget, take it and go do something else. Quantization is a trade; do not pay for it if you are not short of memory.
The measurement that decides it
Post-training quantization changes your model's outputs on purpose. That makes elementwise closeness the wrong acceptance test — the same conclusion we reach on the export path. Compare task metrics on a held-out labeled set, at your production operating point.
The harness we use is small and worth scripting once:
import copy, keras
baseline = keras.saving.load_model("model.keras")
base_metrics = evaluate(baseline, test_ds) # your real metric, not accuracy-by-default
base_bytes = model_file_size("model.keras")
quant = keras.saving.load_model("model.keras")
quant.quantize("int8")
quant.save("model_int8.keras")
print(base_metrics, evaluate(quant, test_ds))
print(base_bytes, model_file_size("model_int8.keras"))
print(latency_p50_p95(baseline, batch), latency_p50_p95(quant, batch))
Four numbers, always reported together: task metric, file size, p50 latency, p95 latency. A decision made on any one of them alone tends to get revisited in production.
Two details that change the verdict more often than people expect:
- Benchmark at your real batch size. Quantization helps most when the workload is bandwidth-bound — small batches, large weights. Large-batch, compute-bound serving may see little or nothing.
- Look at the tail of the metric, not the mean. Aggregate accuracy can hold steady while a rare-but-important class loses several points of recall. Break results down by class or segment before signing off. On imbalanced problems this is where quantization regressions hide.
When plain PTQ is not enough
If int8 costs more than you can accept, you have three moves, in increasing order of effort.
Quantize less. Leave the output head, the embedding table, or the first and last blocks in higher precision. Mixed-precision-by-layer recovers most of the loss for a modest memory give-back, and it is a ten-minute experiment.
Use a calibrated method. GPTQ-style quantization runs a small calibration corpus through the model and adjusts weights layer by layer to minimize output error, rather than rounding each weight independently. For LLM-scale models in KerasHub this is frequently the difference between usable int4 and unusable int4. It costs a calibration pass — minutes to hours — and the calibration data must look like production traffic, exactly as with a representative dataset for LiteRT.
Quantization-aware training. Simulate quantization during fine-tuning so the weights adapt to it. It recovers the most accuracy and costs the most: a training run, a data pipeline, and a reproducibility story. Reach for it when quantization is load-bearing for the product and PTQ has already been shown to fall short — not before.
The operational part
A quantized model is a distinct artifact and deserves to be treated as one. Version it separately from its float parent and record which parent it came from, the mode used, the Keras version, and the calibration data if any. Keep the float checkpoint: it is your reference for parity runs, and the starting point when you requantize for different hardware next year.
Then wire the four-number comparison into CI, so every retrained model re-proves its quantized variant before release. Quantization is not a one-time optimization you perform in a notebook and forget; it is a step in the release pipeline, and a model that drifts into a shape the quantizer handles badly should fail a build rather than surprise you in a dashboard.
Start with bfloat16. If you still need room, try model.quantize("int8") and measure four numbers. Escalate only when the numbers say you must. Most teams we work with stop at the second step, which is rather the point of having the ladder.