Will memory prices keep dropping? KVCache reduced by 99%, single‑GPU throughput up 6×, HKUST, CMU…

Will memory prices drop again? KVCache cuts 99%, single‑card throughput jumps 6×, HKUST, CMU…

The paper mentions that KV Cache can compress by at least 6×.

It should be noted that large model inference is increasingly memory‑hungry, and KV Cache accounts for a significant portion of that cost.

Now Google suddenly slashed it dramatically, causing the stocks of memory‑chip giants like Micron to fall.

Only a few months later, someone dealt an even harsher blow to KV Cache.

This time even compression isn’t enough.

The context length grew from 4K to 256K, yet the KV Cache usage stayed the same.

At a 256K context, it remains fixed at 72 MiB. Compared with the original model, the KV Cache is reduced by99.1%.

Even more striking, a single 80 GB GPU can run 512 parallel long inferences, achieving an output throughput of 12,124 tokens/s, which is 6.42 times that of the original model.

Moreover, its reasoning ability has basically been preserved, with the AIME 2024‑2026 average pass rate reaching 82.9%,while the evaluated bounded-state models, even at 14B, top out at only 18.8%.

This is YanShan Technology‘s YanXin Matrix, jointly released and open‑sourced with Hong Kong University of Science and Technology, CMU and other institutions, the EdgeRock‑4B.

One change it brings to long inference is:

No matter how long the inference, memory does not increase.

The model weights and inference code are now open source, and also support text and image inference.

Why does longer inference cost more?

Current models often have to process tens of thousands of tokens; agents can even work continuously for dozens of rounds, constantly stuffing search results, code, and tool call logs into the context.

The more the model works, the more eager it gets, but the GPU memory can’t keep up.

The problem lies in the KV cache.

Nowadays, most models store inference history in the KV Cache, adding more each time a token is generated.

Take Qwen3.5-4B as an example: each token adds 32 KiB of KV Cache. At 64 K tokens, a single sequence consumes 2 GiB; at 256 K, it’s 8 GiB.

This is especially deadly in the era of inference models.

Models are increasingly prone to “deep thinking,” outputting tens of thousands of tokens at a time; an agent can run for dozens of tool calls, with the context growing longer and longer.

The longer the inference, the more VRAM it consumes, and the fewer requests a single GPU can handle simultaneously.

On the edge side, the KV cache also has to compete with perception and control modules for the already limited memory.

There are roughly two categories of industry approaches.

One type “slims down” each token:

Multi‑head latent attention (MLA) compresses KV, quantization methods like KIVI reduce precision, and Google’s TurboQuant also belongs to this line.

But they only lower the cost per individual token; the cache still grows with inference.

Another category is bounded-state models:

Loop models, state‑space models, sliding‑window models—memory is inherently fixed.

The problem is that they often cannot retain the distant details needed for subsequent reasoning, so their reasoning ability is severely compromised.

Thus the real challenge becomes:

Can we keep the critical details needed for inference within fixed memory?

EdgeRock’s answer is:Yes.

One card, 512 inference streams running simultaneously

First, let’s look at the most intuitive throughput.

The test configuration is 32K input, 32K output, and all models run on a single Hopper 80GB GPU, each using the empirically fastest batch size.

As a result, EdgeRock-4B was scaled up to 512 concurrent streams at once, achieving an output throughput of 12,124 tokens/s.

The base model Qwen3.5-4B’s fastest setting is batch size 64, yielding only 1,889 tokens/s; Gemma 4 E4B, MiniCPM5-2B, and LFM2.5-2.6B also fall between 1,700 and 2,200 tokens/s.

Why is the gap so large? The accounting is actually easy.

After processing 32K input plus 32K output, the KV Cache for each sequence of Qwen3.5-4B is 2 GiB.

A single 80 GB GPU will be filled up with just a few dozen concurrent requests, and additional requests must queue.

When EdgeRock‑4B runs in batch, the total historical state that each sequence needs to store is fixed at 97.76 MiB, regardless of length.

For example, if Qwen3.5‑4B were to process the same 512 sequences, the KV cache alone at generation completion would require about 1 TiB, roughly equivalent to the VRAM of 13 80 GB GPUs.

EdgeRock fits on a single card, peaking at 61.26 GiB, and still includes the model weights.

Now let’s look at scalability.

Across the 16 workloads formed by pairing input and output sizes from 4K to 32K, the batch size was increased from 32 up to 512, and the weighted decoding throughput rose from 4,128 to 12,497 tokens/s, while the peak VRAM usage never reached the single‑GPU limit.

A single request is also not slow.

Models with KV Cache that grows with inference have a common problem: they get slower the more they write.

Because for each newly generated token, it has to look back at the increasingly large KV Cache.

EdgeRock’s computational cost per token does not increase with context length.

After a 128K input, generating another 128K tokens maintains a steady speed of 233.7 tokens/s from start to finish;

Qwen3.5-4B drops from around 169 down to 151.3.

The longer the context, the more you save

Throughput increases severalfold, which, from another perspective, means cost drops by the same factor.

The same card, the same load, generates the same number of tokens:

• With a 4K input, EdgeRock only requires 24.9% of GPU time for Qwen3.5-4B;
• With a 32K input, it only needs 15.6%.

If billing is based on GPU time, the compute cost per token also decreases proportionally.

Averaged over four input lengths, EdgeRock’s throughput is 4.94 times that of Qwen3.5-4B and 7.18 times that of LFM2.5-2.6B.

More importantly, the trend: the longer the context, the larger the gap.

The KV cache of Qwen3.5-4B grows larger as the length increases, but EdgeRock’s does not.

Besides “saving,” fixed memory also brings an often‑overlooked benefit: predictability.

You can calculate exactly how much memory each request will need before inference starts.

• Inference service can plan capacity by request count, without guessing how long each request will run;
• Edge devices allocate once, and no matter how long the inference runs, memory won’t be exhausted;
• Agents’ historical memory no longer grows with the number of tool calls;
• Dedicated chips can even store all historical states on-chip.

In an era where memory is becoming increasingly precious, this calculation only becomes more valuable.

Save memory without compromising intelligence

Seeing this, knowledgeable readers might ask: constant memory, RNN, Mamba have already achieved this—what’s so special?

The novelty lies in saving memory while retaining inference capability.

The team compared EdgeRock-4B with 16 bounded-state models, covering routes such as GDN, RWKV, Mamba, among which three are at the 14B level.

On AIME 2024–2026, EdgeRock-4B achieved an average single-pass rate of 82.9%; the strongest in the control group, RWKV-7 G1j 13.3B, only reached 18.8%, and AHN-GDN Qwen2.5 14B was 11.4%.

Across 29 text benchmarks, EdgeRock-4B ranks first in the bounded-state model on 24 of them; its equal-weight average across the 29 benchmarks is 23.8 percentage points higher than the strongest baseline, AHN‑GDN Qwen2.5 14B.

Compared with the source model, the capabilities are essentially retained:

• The average AIME pass rate stays at 96.3% of Qwen3.5-4B, reaching 97.5% on MATH‑500;
• Among the 29 text benchmarks, it surpasses Qwen3.5-4B on eight, including MMLU‑Pro, SuperGPQA, HumanEval, and MBPP;
• On six vision benchmarks, it achieves an average score of 84.9%, preserving 95.3% of Qwen3.5-4B’s performance.

This performance is already on par with leading small‑size models whose memory grows with inference.

Secret: Anchor the Memory with “Rock Anchors”

So, how does EdgeRock achieve this?

The core is a new memory mechanism, Rock Memory Anchors (RMA).

Its design is inspired by rock anchors: fixing the key information needed for reasoning.

Just as a climber drives a rock anchor into the wall, providing a reliable foothold for every subsequent move, RMA “nails” the truly important details of the reasoning process into a fixed‑capacity anchor library that can be read directly when needed.

Specifically, EdgeRock stores the history of each sequence in three parts, each serving its own role:

• Loop StateSummarize the continuously evolving context, capture the gist;
• Local WindowCover the most recent 1024 tokens, focus on the present;
• Anchor LibraryRetain the most valuable details from earlier history, remember the key points.

The selection process for anchors is as follows:

The writer generates a candidate record for each token, and the scoring network assigns it a score;

For every accumulated 256 candidates, they are compared with the existing records in the anchor library, keeping only the top 1,024 highest‑scoring entries and discarding the rest.

The four KV groups operate independently.

The retained anchors are independent of each other and their content remains unchanged. So even if their source has long slid out of the local window, later reasoning can still read them directly.

In short, the model doesn’t learn to “remember everything,” but to “remember only what’s worth remembering.”

Local window with anchor point library fixes the attention KV of each sequence at 72 MiB. This is the source of the memory staying constant.

Architecturally, EdgeRock-4B is built on Qwen3.5-4B: it retains 24 Gated DeltaNet layers, replaces 8 full‑attention layers with a sliding‑window attention of size 1024, plus an independent RMA module.

The entire model is derived from Qwen3.5-4B through staged conversion,without any pre‑training from scratch.

So do the anchor points actually work?

The team ran an experiment: during inference they turned off all 8 RMA modules, while the recurrent layer and local attention operated as usual.

On long contexts of 256 segments ranging from 16K to 130K tokens, each segment was asked four questions covering retrieval, state updates, cross‑document information composition, and variable binding:

• With RMA on, 964 out of 1,024 questions were answered correctly, an accuracy of 94.1%;
• With RMA off, only 4 were answered correctly, an accuracy of 0.4%.

Long-distance recall is almost zero. This shows that EdgeRock indeed relies on anchors to retrieve information from afar to answer.

Moreover, each anchor records its source location, allowing you to directly see what the model chose to remember.

Will memory prices keep falling?

EdgeRock is developed by YanShan Technology’s YanXin Matrix in collaboration with Hong Kong University of Science and Technology, CMU, and other universities. The model is now open‑source and available for commercial use.

The team revealed that larger‑scale versions of EdgeRock are on the way.

For robots that need to run for extended periods, this mechanism also offers intriguing possibilities.

After all, a robot perceives the environment while planning actions, and tasks can run for several hours or even longer; the historical cache can’t keep expanding indefinitely.

If we can calculate the memory usage of the inference process in advance, edge deployment will also save a lot of hassle.

Whether memory will still drop is left to the market to answer.

But at least one thing has changed: no matter how long the model thinks, its memory won’t bloat anymore.

Paper URL:https://rockmatrix.ai/documents/edgerock-report-zh.pdf
GitHub: https://github.com/rockmatrix-ai/EdgeRock
Hugging Face: https://huggingface.co/rockmatrix-ai/EdgeRock

One-click triple action“Like” “Share” “Heart”

Feel free to leave your thoughts in the comments!

— End —

🌟 Light up the star 🌟

Daily updates on cutting‑edge technology