LiveLens: Two Languages, One Browser Tab, Zero Servers — Now with In-Browser AI Chat

A real-time trading blotter and a webcam object detector have more in common than they look like they should. Both are event-driven: a stream of updates arrives continuously, something needs to render the current state without falling behind, and something else needs to watch that stream for patterns worth flagging – a position that’s moved too far, a count that’s spiked. I’ve spent most of the last decade building the first kind of system for capital markets. LiveLens is what happens when you point the same architectural instincts at a webcam instead of a market feed, and run the whole thing client-side, in a single browser tab, with no backend at all.

Since the first version the app has grown quite a bit. What started as an object detection demo with a Python analytics layer now also includes a full AI chat interface that runs either entirely in the browser — no API key, no server — or connects to Claude, OpenAI, or Gemini via their streaming APIs. All in the same tab.

What it does

Grant camera access and LiveLens detects objects in the live feed, tracks each one’s identity across frames, and continuously computes rolling statistics – per-class counts, average dwell time – flagging anything that looks like a genuine anomaly rather than normal frame-to-frame noise. None of the video, and none of the analysis, ever leaves the tab.

Flip to the Chat tab and you get a conversational AI interface that lets you run language models completely offline in the browser, or switch over to a cloud provider if you need more horsepower. The two modes share the same UI, same settings panel, same streaming experience — it’s not two separate features bolted together, its one interface with a source selector.

Three layers, three tools, on purpose

It would be tempting to reach for one language and use it for everything. I split LiveLens into three layers instead, each running the tool that’s actually good at that job:

• Perception – TensorFlow.js runs a MobileNet-based object detector (coco-ssd) directly in the browser, WebGL-accelerated.
• Identity – a small TypeScript tracker turns a stream of unlabelled per-frame detections into persistent tracked objects.
• Statistics – a genuine Python module, running in-browser via Pyodide, does the counting, aggregation, and anomaly detection with pandas and numpy.
• Chat – Transformers.js v3 runs ONNX-quantised language models directly in the browser via WebGPU or WASM, no API key required. Optionally routes to Claude, OpenAI or Gemini instead.

Why the neural net runs in JavaScript, not Python

This is worth being upfront about, because “Python in the browser” is the headline feature and it would be easy to assume the model runs there too. It doesn’t, deliberately.

In-browser neural-net inference is a mature, well-supported path in the JavaScript ecosystem – TensorFlow.js and transformers.js both have WebGL/WebGPU-accelerated runtimes, wide model support, and years of production hardening. Pyodide, which compiles the CPython interpreter and the scientific Python stack to WebAssembly, doesn’t have an equivalent neural-net execution engine yet. Asking it to run the detector would mean either a much slower pure-Python inference path or shipping a second runtime just to reinvent what TensorFlow.js already does well.

So the detector runs where the ecosystem is strongest, and Python gets used for what it’s actually strong at: data manipulation and statistics. That division of labour is the whole point of the architecture, not an incidental detail.

The tracker: turning detections into objects

coco-ssd gives you a fresh, unlabelled set of bounding boxes on every frame. It has no concept of “the same coffee cup as last frame” – as far as the model is concerned, every detection is a new observation with no history.

To compute dwell time, or to feed any kind of coherent time series into the analytics layer, that identity has to come from somewhere. LiveLens uses a lightweight IoU-based tracker: on each frame, every new detection is matched to the closest existing track of the same class by bounding-box overlap, using greedy matching against an intersection-over-union threshold.

function intersectionOverUnion(a, b) {
  const [ax, ay, aw, ah] = a;
  const [bx, by, bw, bh] = b;
  const x1 = Math.max(ax, bx);
  const y1 = Math.max(ay, by);
  const x2 = Math.min(ax + aw, bx + bw);
  const y2 = Math.min(ay + ah, by + bh);
  const interArea = Math.max(0, x2 - x1) * Math.max(0, y2 - y1);
  const unionArea = aw * ah + bw * bh - interArea;
  return unionArea <= 0 ? 0 : interArea / unionArea;
}

Tracks that go unmatched for more than 1.5 seconds are dropped – long enough to bridge a missed frame or brief occlusion without inventing a new identity for the same physical object, short enough that a track doesn’t linger indefinitely after something actually leaves the frame. It’s a simplified relative of the greedy-matching idea behind SORT-style trackers, without a Kalman filter – accurate enough for a browser demo, not accurate enough for a production tracking system.

Pyodide: real pandas, real numpy, zero server

This is the part that made the project worth writing up. Pyodide loads the full CPython interpreter, plus numpy and pandas, compiled to WebAssembly, and runs it in the same tab as the React app:

def analyze(window_json: str) -> str:
    objects = json.loads(window_json)
    df = pd.DataFrame(objects)

    counts = df.groupby("class")["id"].nunique().to_dict()
    df["dwell"] = df["last_seen"] - df["first_seen"]
    dwell_ms = df.groupby("class")["dwell"].mean().to_dict()

    anomalies = []
    for cls, count in counts.items():
        hist = _history[cls]
        if len(hist) >= 5:
            mean, std = np.mean(hist), np.std(hist)
            z = (count - mean) / std if std > 0 else 0
            if abs(z) >= 2.5:
                anomalies.append({"class": cls, "count": count, "z": round(z, 2)})
        hist.append(count)

    return json.dumps({"counts": counts, "dwell_ms": dwell_ms, "anomalies": anomalies})

_history is module-level state inside the Pyodide runtime, so it persists across calls for the life of the tab without React having to manage a rolling buffer itself – React just calls analyze() on an interval with the current tracked-object snapshot and gets back a JSON summary, exactly as if it were talking to a REST endpoint. The difference is there’s no network hop, no serialization over the wire beyond a function call boundary, and no server to deploy or pay for.

The anomaly check itself is deliberately simple: a rolling per-class count history, a z-score against the mean and standard deviation of the last ~60 windows, and a threshold. It won’t catch anything subtle, but it’s the same basic shape as the alerting logic behind far more sophisticated systems – computed live, over a real stream, with real statistics.

The AI Chat layer: running language models in the browser

The newer addition to LiveLens is a chat tab that lets you talk to a language model without leaving the browser. There’s no proxy, no serverless function, nothing phoning home. The model weights download once, gets cached by the browser, and runs locally on WebGPU or falls back to WASM if WebGPU isn’t available.

Under the hood its Transformers.js v3 running ONNX-quantised models in a Web Worker so the inference doesn’t block the UI thread. The worker loads the pipeline, receives messages, streams tokens back, and the main thread just renders whatever it receives. Its a clean separation and it means the camera detection and Python analytics keep running even while the language model is thinking.

Which models are available offline

The model picker offers two groups — general purpose and coding — so you can pick based on what you’re actually doing:

• SmolLM2 135M / 360M / 1.7B — HuggingFace’s small but surprisingly capable series, good for quick questions and demos. The 135M model loads in seconds even on slower connections.
• Qwen 2.5 0.5B / 1.5B — strong multilingual reasoning from Alibaba, fits comfortably in browser memory.
• Llama 3.2 1B — Meta’s smallest Llama, good general-purpose model, 2GB download.
• Phi 3.5 Mini — Microsoft’s model that punches above it’s weight for reasoning and code, 2.2GB.
• Qwen2.5-Coder 0.5B / 1.5B / 3B — dedicated code models that are genuinely useful for explaining snippets, debugging, and answering tech questions without sending your code to a third-party server.

Switching to a coding model automatically swaps the system prompt to something more appropriate — it’s a small thing but it makes a noticeable difference in the quality of responses for technical questions. The system prompt is fully editable anyway, so you can tune it however you want.

The streaming bug that took a while to track down

Getting real token streaming working with Transformers.js v3 and chat-style input was not as straightforward as I expected. The first attempt used TextStreamer with skip_prompt: true, which is the obvious approach — stream tokens as they’re generated and skip the input tokens so you only show the response. Except the model kept echoing the user’s question back at the start of every response.

The issue is that when you pass a messages array to the pipeline, Transformers.js applies the chat template internally to produce the full formatted prompt string, then tokenizes it. The TextStreamer needs to know how many tokens to skip, but the token count it calculates doesn’t line up with what you’d expect when the input is an array of message objects rather then a plain string.

The fix was to apply the chat template manually first:

const promptText = await tokenizer.apply_chat_template(
  messages,
  { tokenize: false, add_generation_prompt: true }
);

const streamer = new TextStreamer(tokenizer, {
  skip_prompt: true,
  skip_special_tokens: true,
  callback_function: (text) => self.postMessage({ type: 'token', text }),
});

await pipe(promptText, { max_new_tokens: 256, streamer, return_full_text: false });

By passing a string instead of an array, skip_prompt calculates the boundary correctly and the tokens stream in real-time as they’re generated — no post-processing, no fake typing effect, no waiting for the full response before anything appears on screen.

Cloud API fallback: Claude, OpenAI, Gemini

The offline models are genuinely useful but they have obvious limits — a 1B parameter model is not going to match GPT-4o on complex reasoning tasks. So the same chat interface also supports switching to a cloud provider, using the exact same streaming UX.

All three providers stream tokens over server-sent events. There’s a readSSE async generator that handles the data: line parsing, [DONE] termination, and abort signals, and each provider has its own thin wrapper on top that handles the auth header and extracts the right field from the delta object:

• Claude — API key stored in localStorage, anthropic-dangerous-direct-browser-access: true header to allow direct browser calls, extracts content_block_delta.text_delta from the SSE stream.
• OpenAI — API key in localStorage, standard chat/completions stream, extracts choices[0].delta.content.
• Gemini — uses OAuth 2.0 via Google Identity Services rather than an API key, because Gemini supports it and its a nicer UX than asking users to generate and paste service account credentials. You provide an OAuth client ID, click Sign in with Google, and it handles the token flow. Roles get mapped to Gemini’s convention (assistant becomes model) before the request goes out.

API keys and the client ID are stored in localStorage only — they never leave the browser except in the Authorization header of the request to the respective API. No telemetry, no backend logging, nothing persisted anywhere else.

Settings UX: floating panels, not a modal

The settings live behind two icon buttons in the chat topbar — a gear icon for model and provider config, and a pencil icon for the system prompt. Clicking either one opens a floating panel that overlays the messages area without pushing any content around or blocking interaction with the rest of the page. The panel is non-modal, so you can still scroll through the conversation and type in the input while its open.

Keeping them seperate felt right. Model selection is something you do once per session. The system prompt is something you might actually want to tweak mid-conversation, so giving it its own dedicated panel makes it feel less buried.

Interrupt and send

One thing that bothered me about a lot of chat demos is that the Send button disappears while the model is responding and you can’t send anything new until its done. If you realise mid-response that your question was badly phrased, or you want to follow up immediately, you have to wait.

In LiveLens you can send a new message at any point. If generation is in progress, the current response gets aborted, the partial text gets saved into the conversation history, and the new request starts immediately. The button label changes to “Interrupt & Send” during generation so its clear what’s going to happen. There’s also a separate Stop button if you just want to halt the current response without sending anything new.

For offline models the abort signal goes to the Web Worker which terminates the generation loop. For API sources it cancels the fetch via AbortController. Either way the partial text is already in state when the abort fires so nothing gets lost.

Context management

A small token usage indicator in the input bar shows roughly what percentage of the model’s context window is consumed. It turns amber at 80% — at which point you probably want to either clear the context or be a bit more concise. “Clear context” is one click and its right there next to the usage pill, not buried in a settings menu somewhere.

The app also remembers the last selected tab, model, and provider across reloads. The offline model restarts loading automatically on page load if that was what you had selected — it’s already cached locally after the first download so its fast.

Why this matters beyond a toy demo

The pattern here – a continuous event stream, an identity/state layer sitting between raw events and anything downstream, and a statistics layer watching for deviation from a rolling baseline – is the same shape as a real-time trading blotter watching market data for anomalous price moves, or an operations dashboard flagging unusual order flow. The domain changed from financial instruments to webcam detections; the architecture didn’t. Being able to build that shape entirely client-side, with a real statistical computing stack and no backend at all, says something about how far browser runtimes have come.

The chat layer adds something different: a language model that runs in the same tab as the rest of the app, that can be pointed at what the detection pipeline is seeing, and that doesn’t require an account or API key to use. Whether that’s useful in production depends on the use case, but the fact that its a viable option at all is worth noting. A 1B parameter model that runs locally in a browser tab and streams tokens in real time without a server would have been a strange claim to make even two or three years ago.

Practical implications

Detection quality depends on lighting and the model’s speed/accuracy trade-off. lite_mobilenet_v2 is tuned to keep up with live video, not to win accuracy benchmarks – expect it to miss small or partially occluded objects.

WebGL matters. TensorFlow.js prefers a WebGL backend; without it the app falls back to WASM, which is noticeably slower. Most modern browsers and GPUs handle this fine, but it’s worth knowing if performance looks off on an older machine.

Pyodide’s first load has a real cost. Fetching the CPython interpreter plus numpy and pandas as WASM packages is a few megabytes of one-time download, cached by the browser afterwards. The analytics panel shows “Loading Python…” for exactly this reason – it’s not a bug, it’s an honest reflection of what’s happening.

The tracker is intentionally simple. Greedy IoU matching with no motion prediction means fast-moving objects or heavy occlusion can split one physical object into two tracked IDs. A production system would add a motion model; a demo doesn’t need one to make the point.

Offline model quality scales with size. The 135M SmolLM2 is genuinely fast but don’t expect it to write production code or reason through anything complicated. The 3B Qwen-Coder is a lot more capable but its a 3GB download, so pick based on what you actually need. For anything serious the API providers are the more pragmatic choice anyway — the offline path is there for when you want zero data leaving the machine.

Summary

LiveLens now covers more ground than it did at launch. The detection and analytics pipeline is unchanged: TensorFlow.js for perception, a TypeScript IoU tracker for identity, and real pandas/numpy via Pyodide for statistics — all client-side, all in one tab. On top of that there’s a chat interface that runs ONNX language models locally via Transformers.js and WebGPU, with a fallback to cloud APIs (Claude, OpenAI, Gemini) when you need more capability. All four providers stream tokens using the same UX. You can interrupt a response mid-stream and send a new message without losing the partial output. Settings live in non-modal floating panels rather then taking up permanent screen space.

None of these choices were made to show off a single trick. Each layer is there because its the right tool for that specific job — the same principle that governs how I’d architect any production system, just applied here to something you can try in a browser tab in under a minute.