SenseGraph: Can a Precomputed Graph Replace Some Attention Compute?

An idea path for cutting GPU cost by deriving a graph from an existing model once, plus a first working demo that you can run in a browser tab.

The idea

Running a language model is expensive because every query pays for attention at query time. Attention compares every token with every other token, which costs O(n^2 x d) per layer, and models stack 22 to 96 of those layers. That is why serving them leans so heavily on GPUs.

But a trained model already stores a great deal of what it learned in its weights. The embedding table alone encodes which tokens mean similar things: sort, order, comparator and key sit close together. So here is the question behind SenseGraph: could some of the work that attention does at query time be replaced by structure computed once from an existing model, such as a graph over what it has already learned? No retraining and no new model, just a one-time derivation from the one you already have. If that works for even part of the workload, the per-query cost, and the GPU needed to pay it, drops.

This is an idea path, not a finished result. The demo below is the first step on it.

The first step: a working demo

The demo takes the narrowest useful case, nearest-token and phrase retrieval over a model’s embeddings, and compares two ways of answering it on the same model, the same tokenizer and the same input. Both drop stop words such as “on”, “a” and “the” from a search query before building it, so they answer exactly the same question.

• Baseline: a simulated single attention pass followed by a scan of the whole vocabulary.
• SenseGraph: a nearest-neighbor graph precomputed once from the embedding table. A single token is one lookup. A phrase scores a few hundred candidates, taken from its tokens’ neighbor lists and then widened by one refinement round, in a single batched call.

Note on the diagram: the baseline in the app simulates a single attention pass followed by a full vocabulary scan. It does not run a model’s real 22 to 96 layers, so a real forward pass would cost more than the baseline measured here.

How the demo works

1. Pick a model from a dropdown. The app reads the safetensors header from Hugging Face and downloads only the embedding rows it needs, using HTTP range requests. For a 0.5B parameter model that is roughly 90 MB, not the whole 1 GB file.
2. Tokenize the query with the model’s own tokenizer. I implemented byte level BPE (Qwen, SmolLM) and SentencePiece style BPE (TinyLlama, Phi) in TypeScript so the input matches what the real model would see. For search queries, stop words are dropped in both engines and shown struck through in the token list.
3. Build the neighbor graph once: for every token in the vocabulary, find its 20 nearest neighbors by cosine similarity. This is the expensive step. It runs in parallel web workers (up to 8) with WebAssembly SIMD and is cached in the browser, so it happens once per model. The build is tiled: each worker owns a range of rows and receives the rest of the table one small tile at a time, so no worker holds the whole table. That keeps memory low enough for phones, and gives the same graph as an all-pairs search.
4. Answer queries from the graph. A single token is one neighbor lookup. A phrase starts from the union of its tokens’ neighbor lists as candidates, then runs one refinement round: the ten best candidates contribute their own neighbors as extra candidates.
5. Score all candidates in one call. A Rust function builds the weighted query vector once and scores every candidate with a single dot product, so a phrase query is one WASM call, not one call per candidate. The score is the cosine to the mean of the normalized token vectors.

Why the graph needs a refinement round

A phrase is not a single token, and the best answer for it may not appear in any one token’s neighbor list. In an early run of “ancient castle on a hill above the village”, the exact scan ranked “city” first, but “city” was not in the top 20 neighbors of any query token, so a graph that only used those lists could not return it. The top 10 overlap was 70% and the top result differed. Stop words made it worse, because “on”, “a” and “the” pulled the query toward generic words.

The fix has two parts. Both engines drop stop words, so they answer the same question. And the graph widens its candidates through the neighbors of its best first-round results, which are themselves well connected. The app still reports the overlap with the exact scan for every query, so you can see how close it gets rather than taking it on trust.

What I measured

Real numbers from the current build of the live page, using Qwen2.5-Coder 1.5B (50,000 tokens indexed, 1,536 dimensions), with 8 worker threads. The baseline is the simulated attention pass plus full scan described above. Its latency is about twice as high in the continuous run as in the suite, probably from contention in the tab, so the 177x figure is likely flattering and the suite’s 133x is the steadier number. A run on the smaller Qwen 2.5 0.5B gave 73x on the same suite. Single runs are noisy, so run it yourself.

The speedup comes from precomputation, not from the browser. The browser is just where the demo runs. The cost is paid up front: for Qwen2.5-Coder 1.5B the graph is about 7.6 MB and took 311 seconds to build on 8 threads, then it is cached. The smaller SmolLM2 135M took 114 seconds. The same approach would run in Node, natively or on a server, and would likely be faster there.

What the demo shows, and what it does not yet

It shows that, for retrieval over static embeddings, a precomputed graph can return nearly the same top 10 as an exact scan at a fraction of the work. The overlap compares the graph with an exact scan of the same vectors, so it shows the graph is a faithful shortcut, and as the castle example shows, not always a perfect one.

It does not yet show that this matches a real model’s outputs, handles context (the word “bank” is the same vector in “river bank” and “bank account”), understands word order (“the cat sat on the mat” scores like “the mat sat on the cat”), or reduces GPU use in practice. The baseline is a cost model, not a real forward pass. Those are the open questions, and the next section is how I intend to answer them.

Limitations, and which ones are addressable

1. No context. Each token has one static vector, so “bank” is the same in “river bank” and “bank account”. The repository has a rule-based sense resolver, but its senses are seeded by hand and it is not used in the web benchmark. Addressable: cluster a real model’s contextual hidden states into sense nodes, or rerank the graph’s candidates with a small transformer.
2. No generation. This is retrieval and ranking, not a language model. One related experiment is to restrict the output layer’s vocabulary projection to graph candidates.
3. No word order. A phrase is scored against the mean of its token vectors, so “the cat sat on the mat” and “the mat sat on the cat” score the same. Addressable: n-gram nodes and positional weighting.
4. One-time build cost and a vocabulary cap. The exact all-pairs build took 114 seconds for 49,152 tokens at 576 dimensions and 311 seconds for 50,000 tokens at 1,536 dimensions, and grows roughly with the square of the vocabulary, so only the first 50,000 tokens of larger vocabularies are indexed. Addressable: an approximate nearest-neighbor index or WebGPU.
5. Memory on phones. An early build crashed the browser on a phone, because each worker held its own copy of the embedding table. The build now streams the table in tiles, which should fix that, but I have only tested it on a desktop so far. The table is still held once in memory, so streaming the download straight to browser storage is the next step.
6. Phrase queries are approximate. Averaged vectors lose compositional meaning (“hot dog”), and the graph only scores candidates it reaches. The app reports the overlap with the exact scan for every query.
7. Not yet compared with a real model. The baseline is a cost model and the overlap is against the same embedding table. Next: compare with a real model’s output in the browser and on a standard similarity benchmark, then measure real GPU cost.

The path forward

• Check against real model outputs. Compare against final layer embeddings or a standard retrieval benchmark, not only against an exact scan of the same table.
• Handle context and order. Test queries where context or word order changes meaning, which is where attention earns its cost, and see how far a graph can go or where it needs a hybrid, for example a small transformer that reranks the graph’s candidates.
• Measure real GPU cost. Run the Qwen2.5-Coder models and larger vocabularies, and compare against a real forward pass on a GPU.
• Go beyond the embedding table. Explore graphs derived from deeper layers of an existing model, not only the input embeddings, and speed up the build with WebGPU or an approximate nearest-neighbor index.

Each step is a way to enrich the demo toward a full scale version of the idea.

Under the hood

• Rust compiled to WebAssembly for vector storage, SIMD cosine similarity, the neighbor graph and the batched candidate scoring function.
• TypeScript, React and Web Workers for the app, the tokenizers, the parallel graph build and the two engines.
• An Nx monorepo with the Rust core, a Node command line REPL and the web app, with CI that runs the Rust tests, builds both WASM targets, type checks everything and deploys the demo.
• Code models first. The default model is Qwen2.5-Coder 0.5B, with SmolLM2, Qwen, TinyLlama and Phi available from the same dropdown.

Try it

The demo is live at pixytech.github.io/SenseGraph. Load a model, run a phrase, compare the two panels, then run the complex task suite to see the input tokens, both engines’ top results, the overlap and the speedup for each query. The source is at github.com/pixytech/SenseGraph.

SenseGraph is an open source experiment. The numbers above are from my own runs, so please run it yourself and tell me what you get.