Insufficient VRAM and Expensive RAM: KVCache Keeps Growing—Huawei Proposes a New Solution
Insufficient VRAM, memory too expensive, KVCache keeps growing: Huawei offers its own new solution
Content Summary
Author ｜ Chu Xingjuan. KV Cache, previously treated as a “temporary cache,” is now becoming an important data asset that needs to be stored, scheduled, and reused long‑term. Model contexts are getting longer, and Agent task durations are increasingly
What used to be treated as a “temporary cache,” the KV Cache, is now becoming an important data asset that needs to be stored, scheduled, and reused over the long term.
As model contexts grow longer, Agent tasks run for longer periods, and scenarios like AI coding even require project context to be retained across days or weeks, the three‑tier cache hierarchy that relies on GPU memory, RAM, and local SSDs is hitting limits in capacity, cost, and lifespan. Many vendors in the industry are adding a fourth layer—specialized storage for large‑scale inference—including Huawei.
Recently, Huawei officially launched the OceanStor M900 AI Memory Storage. The OceanStor M900 is not just “adding another disk” to AI servers; it aims to turn the KV Cache from a temporary resource on compute nodes into a “memory” that can be shared, persisted, and reused across the cluster.
This also reflects Huawei’s assessment of the current stage of the AI industry: as model capabilities and compute power gradually converge, the variables that determine the efficiency of enterprise AI deployment are increasingly whether there is high-quality data, sufficient contextual memory, and whether these memories can be reused repeatedly at lower cost.
In traditional large‑model conversations, KV Cache is usually just short‑lived temporary data. After the user ends a session, this cache can be released quickly. However, agents and AI coding change this logic.
Typical Q&A sessions are usually short‑term and single‑turn, with strong contextual relevance, so the KV cache may only need to be retained for a few hours and can be cleared once the conversation ends. However, in AI coding scenarios, code repositories, project frameworks, previously generated results, and project context are continuously reused, so the KV cache may need to be kept for days or even weeks.
In other words, different applications are beginning to exhibit completely different “memory cycles.” Enterprises need to define, based on specific business needs, which KV pairs are worth preserving, how long to keep them, and when to evict them, rather than applying a single fixed strategy.
This is also why Huawei directly calls the OceanStor M900 “AI memory storage.” Yang Wendao, President of Huawei’s Distributed Storage division, said, “The ‘M’ stands for Memory; its core goal is to extend contextual memory beyond GPU memory and system RAM, allowing KV pairs associated with more tokens to be stored long‑term and hit repeatedly.”
Internal testing at Huawei and customer POCs show that using a dedicated KV cache storage can roughly double the token hit rate while reducing the first-token latency (TTFT). For enterprises, this not only impacts user experience but also directly affects inference cost: if historical context hits the cache, there is no need to re‑execute the full prefilling each time.
Currently, the most common inference cache hierarchy is: L1 in GPU memory, L2 in system memory, and L3 on local SSD. GPU and memory access latencies are in the nanosecond range but have limited capacity and high cost; local SSD offers larger capacity but its latency has already entered the millisecond range. As long‑context and large‑scale agent tasks increase, these three layers alone are becoming increasingly insufficient to accommodate the ever‑growing KV cache.
Yang Wendao explained that the OceanStor M900 AI memory storage focuses on the L3.5 tier. L3.5 is not a concept invented solely by Huawei; it naturally emerged across the industry as AI inference scaled up and faced capacity and cost bottlenecks. Huawei chose to launch a dedicated product at this tier because the traditional three‑level cache architecture can no longer support the future scale of KV caches.
According to Huawei’s actual calculations with customers, if a large amount of context is truly retained as required by production, the needed storage is no longer at the TB level but at the PB level. OceanStor M900 extends the KV Cache in supernodes from GPU memory and system memory further to SSDs, providing up to 64 PB per cluster, and the KV Cache capacity usable by a single NPU is also increased from the GB level to the TB level.
From Huawei’s perspective, professional KV context memory storage is not simply adding another product category; it could become a new fixed tier in large‑scale inference infrastructure.
Moving the KV Cache from GPU memory and RAM down to SSD seems to solve capacity and cost issues, but it also introduces a new problem: latency.
Access latency for GPU memory and RAM is on the nanosecond scale, whereas SSDs are already in the millisecond range. If the model has to fetch data from SSD each time it needs a token, the slower data access could theoretically offset the benefits of cache reuse. Therefore, what truly determines whether L3.5 can work is how the KV is scheduled.
Yang Wendao told InfoQ that Huawei also encountered this issue in internal tests and customer POCs. The industry currently commonly uses the Mooncake inference framework, and Huawei is building on this framework to further address KV scheduling and prefetching. The system needs to predict which KV the model is likely to use next and perform hierarchical prefetching based on the performance characteristics of GPU memory, RAM, and SSD, so that when access is actually required, the data is already in the faster upper layer.
At the same time, traditional approaches that use distributed file systems or parallel file systems to store KV often require converting KV into files first, then locating the files through directories, reading the files, and finally re‑parsing them back into KV for the model. This process involves multiple format and protocol conversions.
OceanStor M900 AI memory storage aims to shorten this process to a “single‑hop direct pass.” KV no longer needs to be converted into traditional file objects; it can be accessed directly according to KV semantics, and data exchange is completed via the high‑speed communication network within the Ascend hyper‑node. Essentially, this allows the NPU to devote as much compute power as possible to inference itself, rather than consuming resources on data conversion and movement.
Currently, the OceanStor M900 adopts a “three‑core‑in‑one” architecture that integrates networking, CPU, and disk‑control capabilities, and works in tandem with the Lingqu network and Ascend hyper‑node. Huawei ultimately aims to treat computing, storage, and networking as a single entity, rather than the traditional notion of “compute nodes with external storage.”
The disclosed data shows that in traditional PCIe+RoCE architectures, GPU or NPU access to SSD pools may require five data movements, whereas the OceanStor M900 can reduce this to one, cutting I/O latency and first-token latency by more than 50%. Its access latency is about 10–15 microseconds, and access bandwidth can reach 40 GB/s.
However, this raises another critical question: does the OceanStor M900 only operate within Huawei’s own ecosystem?
Yang Wendao explained that the OceanStor M900 is currently positioned to pair with Ascend super‑nodes, using the UB protocol to handle internal data exchange within the super‑node. For upper‑layer applications, there is no need to be aware of these underlying communication methods. For non‑Ascend, non‑super‑node, or heterogeneous computing environments, Huawei already offers the OceanStor M800, which adapts to different compute platforms via standard networks and hardware.
The OceanStor M900 and OceanStor M800 are not a “high‑low” tier relationship but rather distinct product forms aimed at different infrastructures. The OceanStor M800 was launched before the M900 and has already been tested with multiple customers; the OceanStor M900 further targets Ascend super‑nodes, achieving deeper integration and collaboration among compute, storage, and networking.
Although primarily aimed at super‑nodes, the OceanStor M900 is valuable well before reaching a 100,000‑card scale. Yang Wendao noted that its value grows as the cluster size expands. In theory, once an enterprise moves to large‑scale AI inference and begins encountering KV cache capacity, cost, and sharing issues, a need for specialized memory storage may arise.
Another often overlooked issue with professional memory storage is SSD lifespan.
Typical enterprise‑grade SSDs are measured by DWPD (Drive Writes Per Day), usually around 1–1.5 writes per day. For traditional storage workloads this level generally meets lifecycle requirements, but the write pattern of KV Cache is completely different.
If a large number of agents and long-context tasks continuously create, update, and discard KV, the SSD may need to be written at an extremely high frequency. Continuing with the traditional model would only allow us to dilute the write pressure by constantly adding more SSDs, which would cause costs to rise rapidly.
Yang Wendao gave an example: if an SSD can only be fully written once per day, then preserving a large amount of KV would require a very large physical capacity. However, if the DWPD is increased to 24, which is equivalent to allowing a full-disk write every hour, then a massive amount of KV that only needs to be kept for an hour can be quickly written, discarded, and the same media reused.
Therefore, one of the key values of the OceanStor M900 is extending media lifespan. Huawei disclosed that by jointly optimizing the media, chips, and control software, it can avoid rapid SSD wear while maintaining high-frequency read/write performance. In tests, the relevant lifespan capability can be improved by nearly 16 times.
This point is actually the key factor that determines whether “SSD for AI memory” is commercially viable. If the lifespan issue cannot be solved, KV Cache will have to continue relying on expensive RAM or obtain capacity through massive disk stacking, and the cost advantage of specialized memory storage will be diminished.
On the other hand, not all KV entries need to be stored permanently. Huawei believes that the truly reasonable approach is to set the cache lifecycle dynamically based on business needs.
Ordinary conversations may be retained only for a few hours, AI coding can be kept for several days or even weeks; different models and projects will adopt different strategies. Enterprises usually know which business contexts are worth preserving long‑term, so the system can dynamically evict data according to business rules.
It can be seen that future AI memory management will essentially become more like traditional computer multi‑level caches: instead of simply aiming for “store as much as possible,” it will seek a dynamic balance among hit rate, capacity, cost, lifespan, and access latency.
Huawei also places KV Cache into a larger “AI Data Platform” ecosystem. Wu Junjie, Vice President of Huawei’s Data Storage product line, explained that the AI Data Platform defined by Huawei currently includes four core capabilities: the KV Cache library, a knowledge base, and a memory store, all managed through UCM (Unified Cache Manager).
Among them, the KV Cache library addresses the issue of repeated context computation during inference; the knowledge base converts traditional documents into knowledge that is easier for AI to retrieve and understand, improving answer accuracy in specialized scenarios; the memory store further preserves the historical experience, past responses, and task outcomes generated by agents during long‑term task execution, allowing the system to gradually accumulate “experience.” The three correspond to “short‑term context,” “domain knowledge,” and “long‑term experience,” together forming a new data layer for enterprise AI.
Compared with the OceanStor M900, which mainly targets large-scale data centers and supernode inference, the AI Data Platform is more oriented toward on‑premises enterprise deployment, aiming to become a universal data foundation for enterprises running AI applications in the future.
“Huawei definitely doesn’t want to be a lone hero,” Yang Wendao said.
Currently, Huawei is continuing to leverage its vertical integration advantage in Ascend supernodes on one hand, while on the other hand trying to integrate upper‑level KV scheduling and inference capabilities into mainstream open‑source ecosystems.
Mooncake is one of the key points. Yang Wendao said that Huawei is already cooperating with the Mooncake team and plans to contribute some inference‑related code to the Mooncake community in the future. Huawei hopes that professional memory storage will ultimately become a foundational capability for the entire industry, rather than a closed technology monopolized by a single vendor.
In his view, the L3.5 market is still in its early stage, and the first issue to resolve is the industry question of “whether this market can truly take off,” rather than locking customers into a closed ecosystem prematurely. Only when inference frameworks, models, compute vendors, and storage vendors participate together can external KV Cache and professional memory storage achieve large‑scale deployment.
Facing competition from other storage vendors, Huawei believes that AI memory storage is essentially not just adding an interface to traditional storage, but a new architecture driven by AI inference demands. Huawei has conducted experimental tests with leading domestic internet companies, and current results show that while traditional file storage and parallel file systems can hold KV, they still incur significant overhead in access paths, data transformation, media lifespan, and inference coordination.
Therefore, the key to future competition is who can truly solve KV’s capacity, latency, scheduling, lifespan, and ecosystem coordination challenges.
Conference Recommendations
QCon Global Software Development Conference 2026 (Shanghai) will be held on October 22–24. This year’s conference focuses on engineering practices in the Harness AI era, covering topics from “building AI” to “harnessing AI”, including hot technical directions such as AI‑Native architecture, Agent Runtime, AI Infra, Agent security and observability, Loop Engineering, embodied intelligence and world models. We invite the global tech community and frontline industry practitioners to share the most valuable engineering experiences of the AI‑Native era. For more details, scan the QR code or contact the ticket manager at 18514549229 for inquiries.
Domestic GPUs are increasing, so why are they actually harder to use?
Are you also “watching”? 👇
Get MirrorSpeed