The 150 ms budget: latency engineering for live sketch recognition

DrawLa’s core promise is that the AI guesses while you draw, not after. That promise has a number attached: if the round-trip from pen stroke to updated prediction takes much more than 150 milliseconds, the magic collapses into “laggy quiz app”. This post is about defending that number in production.

Why 150 ms, and why it’s really 120

Human-computer interaction research has known for decades that ~100 ms is the threshold under which feedback feels instantaneous, and that beyond a few hundred milliseconds users perceive a system as sluggish. For a drawing game the sweet spot sits in between: predictions may lag a stroke slightly, but they must visibly react to this stroke, not the previous one. We set the client’s inference cadence to 120 ms – roughly 8 prediction updates per second – and gave the full pipeline a soft budget of 150 ms end-to-end.

Measure first: the three-part number

You cannot defend a budget you don’t measure, and averages lie. Every prediction response carries server-side timings, and the client compares them against its own send timestamp. In debug builds we render the result directly in the prediction panel:

E2E 87.4 ms · inf 11.2 · ws 3.1

Three numbers, three very different problems:

• inf – pure model inference time on the server. The part everyone assumes dominates.
• ws – server-side WebSocket handling overhead: parsing the stroke batch, rasterizing, queueing.
• E2E – what the player actually feels: everything above plus the network both ways and client-side processing.

The uncomfortable early lesson: inference was never the bottleneck. Our 345-class sketch classifier, exported to ONNX and served in-process with ONNX Runtime, answers in ~10–15 ms on Cloud Run’s CPUs. The budget was being eaten elsewhere – by the network path and, in bad weeks, by our own serialization sloppiness.

The four optimizations that mattered

1. Kill the cold starts

Cloud Run’s scale-to-zero is wonderful for cost and fatal for latency: a cold start means loading Python, the framework and a neural network before answering – multiple seconds where the budget allows 150 milliseconds. We pin min-instances: 1. One always-warm instance is the single cheapest latency optimization we ever bought.

2. Stay on one connection

Early prototypes sent strokes over the WebSocket but fetched predictions over HTTP. Moving everything onto the one WebSocket removed a TLS handshake tax and – more importantly on mobile networks – removed a second congestion-controlled connection that could stall independently. One socket, multiplexed message types, predictable behavior.

3. Batch points, not strokes

A finger on a 120 Hz phone screen produces far more input events than anyone needs. The client coalesces pen points and ships them at the inference cadence instead of per-event. Fewer, slightly larger messages beat many tiny ones on every network we tested – and the server does strictly less parsing work per second.

4. Serve the model in-process

There is no model server, no gRPC hop, no sidecar: the game process owns the ONNX session and calls it like a function. A dedicated inference service earns its complexity when you need independent scaling or GPU pooling; at one warm CPU instance it would only add a network hop to every one of those eight-per-second calls.

What we deliberately did not do

The tempting “obvious” move is running the model in the browser – WebAssembly or WebGPU inference would cut the network out of the loop entirely. We prototyped it and said no, for a game-integrity reason: the referee must not live on the player’s machine. A client-side verdict is a cheatable verdict, and in a competitive game the AI’s confidence is the score. We pay the network tax to keep the whistle on the server. (The broader topic of on-device versus server-side inference is exactly what our sister site drawla.app covers in depth – this post stays on the game-engineering side of that line.)

The numbers today

On a typical European broadband connection to our us-central deployment, players see end-to-end times of 80–120 ms: ~10–15 ms inference, single-digit WebSocket overhead, the rest network. On mobile LTE it degrades gracefully into the 150–250 ms range – noticeable if you look for it, playable if you don’t. The prediction cadence hides a lot: because updates arrive rhythmically eight times a second, a slightly late verdict reads as “the AI is thinking”, not “the game is broken”. Perceived latency is a design resource, not just a measurement.

Previous post: the multiplayer architecture. Next: designing the chaos effects – where we intentionally make drawing harder.