The hidden costs of local LLM inference

The allure of running large language models (LLMs) locally is understandable—impressive hardware, full control, and no reliance on third-party services. But beneath the surface lies a more complex reality: the cost of local inference often outweighs the perceived benefits. Enter Featherless.ai, a service designed to make LLM inference effortless, cost-effective, and accessible.

Most users running LLMs locally adopt batch sizes of 1 (BS=1), often aiming for the best-case scenario of minimal latency. Yet, this approach doesn’t account for hidden costs, particularly energy consumption. In fact, our analysis shows that for typical BS=1 usage, energy expenses alone can exceed the price of our $25/month premium tier—a fraction of the cost of maintaining high-end hardware.

In this post, we dive into benchmarks of popular models across a range of hardware setups, showcasing the complexities and costs of local inference. From GPU splits and layer offloading to the challenges of multi-threaded CPU performance, we break down the numbers to reveal how Featherless.ai eliminates these headaches with a single, predictable pricing model.

Benchmarks: How Local Inference Stacks Up

Using llama.cpp’s llama-bench, we evaluated some of the most popular models in different quantizations using both high-end and enthusiast-grade hardware:

• 2× NVIDIA RTX 4090, Ryzen 9 7950x3D, 64GB DDR5 (~$6,000)

• 14” M3 Max MacBook Pro with 36GB of unified memory ($2,899)

To replicate the 2×4090 results, we use the following command with arguments:

• -p 0 - number of tokens already present in the prompt. we chose 0 to get best case results for local inference

• -n 128 - number of tokens to generate during a single run.

• -ngl 999 - number of layers to be offloaded to the gpu(s) we chose 999 arbitrarily as it gets clamped to the actual number of layers in the model

• -ts “50/50” - the distribution of layers between multiple GPUs, 50/50 meaning both GPUs process the same number of layers

• -mg 0 - ID of the main GPU

• -fa 1 - we chose to use flash attention, as it gave us a noticeable improvement on NVIDIA GPUs.

When measuring the 1×4090 performance, the command is simpler as we don’t have to specify the split ratio and the main gpu.

Things get a bit more complicated for 4090 with CPU offloading case as we have to manually specify the number of layers that are processed on the GPU and because we’re using different quantizations, this number is different on a case by case basis.

For our CPU-only runs, the situation is a bit different. Here we explicitly tell llama.cpp to use 12 threads using the -t 12 flag. We’ve found that increasing the number further resulted in decreased performance. Most likely option is the multi-CCD nature of our chosen CPU - meaning that only 16 out of the 32 available threads have access to the increased L3 Cache.

Compiled into a single table:

* Not optimal (run with 2 GPUs while the model fits on 1)

** Special quant optimized for AVX512 capable CPUs or non-Apple ARM CPUs

The Featherless.ai Advantage

Featherless.ai simplifies LLM inference, giving you:

• Access to any Hugging Face model with serverless convenience.

• No need for expensive hardware, energy bills, or complex configurations.

Our fixed-rate pricing eliminates the guesswork of per-token billing, empowering developers to build innovative applications without worrying about costs spiraling out of control.

Conclusion

Running LLMs locally might seem appealing at first, but the reality is far from simple. Between high hardware costs, wasted GPU resources, and hidden energy expenses, it’s clear that local inference is often inefficient and impractical. For most developers, Featherless.ai offers a far better alternative.

With Featherless.ai, you can focus on building while we handle the heavy lifting. Start your journey today and unlock the full potential of LLMs—hassle-free and cost-effective.