Tensormesh and AMD have announced a collaboration aimed at letting companies run more AI models on fewer GPUs by managing memory beyond on-chip high-bandwidth memory.
The setup combines Tensormesh's KV cache software with AMD's virtual memory tools. It was tested on Dell servers equipped with eight AMD MI355 GPUs and Dell storage. The system lets AI workloads move KV cache data across DRAM, SSDs and remote storage instead of keeping it only in scarce GPU memory.
KV cache stores data from earlier stages of an AI model's processing, so the model does not need to recompute the same information repeatedly. That is especially important for inference, where businesses want to serve models quickly and at lower cost across large document collections and other data-heavy workloads.
Tensormesh argues that the industry's main response to memory limits has been to add more GPUs or build chips with more memory, increasing both supply pressure and cost. Its approach instead aims to make fuller use of memory already available across a server node by extending the memory a GPU can access.
In tests on a 300GB document set using Kimi-K2.6, the companies said time-to-first-token fell from 3.4 seconds to under half a second when cached KV data in DRAM and network file storage was reused rather than recomputed. They also said output throughput held at about 48 tokens per second as workload size increased, while unoptimised inference throughput dropped by nearly 40% under the same conditions.
On the same hardware, model density doubled, suggesting more models could be served without adding GPUs. For corporate AI users, that could influence infrastructure spending as they try to meet rising inference demand without expanding hardware fleets at the same pace.
Memory pressure
High-bandwidth memory has become a central constraint in AI infrastructure because many modern models require large amounts of memory for model weights and cached context. As a result, access to GPU memory is a major factor in how many models an organisation can run on a given set of machines.
Tensormesh said users of LMCache can also repurpose cache chunks stored for short-term memory virtualisation for later prefix and non-prefix KV cache matching. That allows the same stored data to serve more than one purpose in the inference stack, with the aim of improving overall efficiency.
The collaboration follows an investment by AMD Ventures in Tensormesh as part of a USD $20 million funding round. Tensormesh said it has raised USD $24.5 million in total from backers including Valley Capital Partners, NVentures, AMD Ventures, CoreWeave and Laude Ventures.
Tensormesh was founded by faculty members, researchers and alumni from the University of Chicago, UC Berkeley and Carnegie Mellon. The company focuses on inference optimisation and cache management for enterprise AI systems, an area drawing more attention as businesses move from training models to running them in production.
The partnership also gives AMD another example of software and systems work designed to make its accelerators more practical in large-scale AI deployments. Hardware vendors are increasingly addressing bottlenecks in memory access, data movement and serving efficiency through software as well as chip design.
For Dell, the test environment highlights the role of server and storage suppliers in AI system design beyond simply providing compute hardware. AI deployments often depend on how effectively storage, system memory and networking work with accelerators, especially when workloads outgrow local GPU memory.
Junchen Jiang, Chief Executive Officer and Co-founder of Tensormesh and creator of LMCache, described the work as a way to expand the memory available to inference systems. "This powerful new collaboration builds on AMD's recent strategic investment in Tensormesh and expands the capabilities of AMD GPUs," said Jiang.
"Together, we're greatly enhancing the memory that inference engines can access for model weights and KV cache, using all of the memory resources on each node," Jiang said.
AMD described the arrangement as a combination of its virtual GPU memory management and Tensormesh's cache software. "We recognize Tensormesh and LMCache as KV cache management leaders," said Anush Elangovan, Vice President of AI Software at AMD.
"And we're thrilled to announce AMD's breakthrough in virtual GPU memory management, amplified by LMCache and Tensormesh," Elangovan said.