Real, measured latency, throughput, and model-size numbers across three ONNX Runtime configurations — not estimates. This is what a systems-level validation report looks like before a model ships to a latency- and memory-constrained device.
Measured on a CPU-only sandboxed container — CPUExecutionProvider only, no GPU available in the environment this was benchmarked in.
| Config | p50 (ms) | p95 (ms) | Throughput (fps) | Peak mem (MB) | Model size (MB) |
|---|
—
Two things worth being upfront about. First, weights are random-initialized, not trained — legitimate
for this benchmark specifically, since latency/memory/throughput depend on architecture and numeric
precision, not weight values. Second, INT8 quantization here is scoped to the dense (MatMul/Gemm)
layers only; the convolutional layers stayed FP32 because this ONNX Runtime build's CPU kernel
registry doesn't implement ConvInteger, which is what dynamic quantization emits for
Conv ops (there's no fused dynamic QLinearConv in the ONNX spec — only static/QAT quantization
produces that). That's also a realistic real-world deployment choice, not just a workaround.
python/build_model.py — constructs the ONNX graph directly (no PyTorch, no weight download) python/benchmark.py — runs all 3 configs, writes demo_output.json (this page's data source)