A trained Keras model is not a deliverable; a verified inference artifact is. The gap between the two — export, conversion, quantization, and parity testing — is where deployment schedules quietly slip. This post covers the two export paths we use most for edge and cross-runtime deployment, LiteRT and ONNX, and the verification discipline that applies to both.
First, a naming note: LiteRT is the current name for what was TensorFlow Lite — same runtime and file format (.tflite), rebranded as it broadened beyond TensorFlow. If your docs and dependencies say TFLite, you are in the same ecosystem.
Separate the training artifact from the inference artifact
Keras 3 draws a line worth respecting. model.save("model.keras") produces the training artifact: full architecture, weights, optimizer state, resumable. model.export("export_dir") produces the inference artifact: a SavedModel with a serving signature and no training machinery. Converters consume the exported inference graph:
model.export("export/1")
converter = tf.lite.TFLiteConverter.from_saved_model("export/1")
tflite_model = converter.convert()
Exporting means committing to an input contract. Decide now, not at integration time: input dtype (float32 vs uint8), layout, resolution, and whether preprocessing (resize, scaling, normalization) lives inside the graph or in the calling application. Every preprocessing step you leave outside the artifact is a step some client will eventually implement slightly differently — bake preprocessing into the exported graph when the runtime allows it.
The LiteRT path: ops, then quantization
Conversion failures are almost always op coverage. LiteRT implements a subset of TensorFlow ops in its builtin set; models using ops outside it either fail to convert or convert with Flex delegate fallback (SELECT_TF_OPS), which links a much larger runtime into your binary — often unacceptable on mobile or embedded targets. Convert early in the project, with the real architecture and a dummy checkpoint, so an unsupported op is a design finding rather than a launch blocker. Custom layers written with standard keras.ops arithmetic generally convert cleanly; exotic ops, dynamic shapes, and control flow are where trouble lives.
Quantization is a decision ladder, cheapest first:
- Dynamic-range quantization — one flag, no data needed, weights in int8, roughly 4× smaller and faster on CPU. Try it first.
- Full integer quantization — weights and activations in int8; required for many NPUs, DSPs, and microcontrollers. Needs a representative dataset: a few hundred real, preprocessed inputs the converter runs to calibrate activation ranges.
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = rep_data_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
The representative dataset must reflect production inputs — same preprocessing, same distribution. Calibrating int8 ranges on data that doesn't match production is a subtle way to ship a worse model that passed all your tests.
The ONNX path
ONNX is the practical choice when the target runtime is ONNX Runtime, TensorRT, or a vendor toolchain that speaks it. From a TF-backend Keras model, tf2onnx converts the exported SavedModel:
python -m tf2onnx.convert --saved-model export/1 --output model.onnx --opset 17
Keras 3's multi-backend nature offers a second road: run the model under the PyTorch backend and use torch.onnx.export, which is occasionally the cleaner path for architectures whose ops map awkwardly through tf2onnx. Either way, pin the opset version to what your deployment runtime supports, and record the converter versions in the artifact's metadata — "which tool produced this file" is exactly the fact you need during an incident, and exactly the fact nobody wrote down.
Parity testing: the non-negotiable step
Every conversion is a re-implementation of your model by other code, and quantization changes the numbers on purpose. So verify, mechanically:
- Fix a battery of real inputs (hundreds, covering all classes and edge conditions — not one random tensor).
- Run them through the original Keras model and the converted artifact.
- For float conversions, assert closeness (
atolaround1e-5is typical; justify anything looser). For quantized models, compare task metrics on a labeled evaluation set instead — elementwise closeness is the wrong test once int8 is involved; what matters is accuracy at the operating point. - Wire this into CI so every retrained model re-proves parity before release.
Finally, benchmark on the actual target hardware — the device in the field, not your workstation. Quantized speedups vary enormously across CPUs, NPUs, and delegates (XNNPACK, GPU, vendor NPU), and a configuration that flies on one phone crawls on another.
The shape of a good export pipeline
Export → convert → quantize → parity-check → benchmark, scripted end-to-end and run from CI on every training run that produces a candidate model. Teams that treat export as a one-time manual step redo this work, slightly differently, every release — and discover regressions on devices. Teams that script it ship model updates the way they ship code. The second group has measurably fewer bad weeks.