Cost per task emerges as next test for AI memory boom

Lee Soo-hwa

The author is a research professor of Big Data Innovation Convergence College at Seoul National University and the head of the AI Center at DLG Law Corporation.

Lee Dong-soo, CEO of A2SYS, recently offered an explanation for surging memory demand: It is not simply that AI agents consume enormous amounts of memory, but that nobody knows how much they will need.

Citing research showing that token consumption can vary 30-fold even when the same model tackles the same problem, with the model unable to predict usage beforehand, Lee argues that demand includes an “uncertainty premium” as companies stockpile capacity.

How agents operate also determines costs. Experiments by a team led by KAIST distinguished professor Yoo Min-soo using Meta’s Llama 3.1 70B model found that an agent repeatedly reviewing and revising its answers consumed 136.5 times the GPU energy of a conventional chatbot response. Exploring multiple solutions simultaneously required less than half that energy while raising accuracy from 67 to 82 percent.

Yet equating exploding token usage with equally explosive demand for high bandwidth memory, or HBM, tells only half the story. Model weights remain fixed. Tokens retained as context occupy capacity in the key-value cache, while newly generated tokens consume bandwidth through repeated access to weights and cached information. During token-by-token generation, bandwidth can become a greater constraint than computing power.

Nor can today's shortages be attributed solely to uncertainty. Micron says HBM requires roughly three times as many wafers as conventional DRAM to produce the same number of bits, approaching four times for newer generations. It has contracted most of next year's HBM supply and expects memory markets to tighten over the following two years.

That makes measuring what Lee calls the “memory elasticity of intelligence” essential: the marginal productivity of memory. How much does adding HBM improve concurrent user capacity and response speed? Without answers, accumulated inventory could become an expensive idle asset when new factories increase supply.

Efficiency does not necessarily reduce demand. In “The Coal Question” (1865), William Stanley Jevons observed that improvements in steam-engine efficiency ultimately increased coal consumption. Likewise, caching, compression and routing could reduce memory costs per task while encouraging wider AI adoption and greater overall HBM demand. But this paradox cannot tell investors which facilities will benefit.

Nvidia calls data centers "AI factories" producing tokens. Customers, however, purchase completed tasks, not tokens. The decisive metric should be total memory, electricity and GPU costs per successfully completed task, extending Lee's proposed measure to include power and time. As uncertainty surrounding AI investment recedes, corporate valuations will depend less on memory volumes than on the cost of work that memory actually completes.