When I first installed a local LLM, I expected to use it the same way that I'd been using ChatGPT.It soon became apparent that on my modest hardware, this wasn't going to work.By changing the way I use my local LLMs, they've become much more useful.
A local LLM on my hardware will never match ChatGPT My expectations were all wrong Services such as ChatGPT, Claude, and Gemini provide access to leading models running on powerful cloud infrastructure.Companies behind leading closed models such as ChatGPT, Claude, and Gemini no longer disclose their parameter counts, but DeepSeek-V3 was published in 2024 with 671 billion total parameters.It’s a pretty safe bet that current frontier models are likely to be at least as big, if not larger.
In comparison, the largest models I can practically run on my mini PC are 8B models, and that's at a push.An 8B model on my hardware can't match the overall capability, speed, or reliability of a leading cloud-based model.The problem was that when I first tried a local LLM, I treated it just like ChatGPT.
I asked it a broad, open-ended question, which ChatGPT would have answered almost instantly, but my local LLM took an age to even start responding, and even then, the response crawled.I was looking at my local LLM in the completely wrong way because it's unreasonable to expect it to work the same as a cloud-based LLM.Where small LLMs struggle The tasks that exposed the gap fastest Close Leading cloud-based LLMs are generally much better than small local models at answering broad questions.
While a local LLM can attempt to answer these questions, the results are usually disappointing.I asked ChatGPT why IPv6 adoption has been slow despite being available for decades.This is a question that requires combining technical knowledge, tech history, economics, and more into a coherent argument.
ChatGPT instantly generated a useful response covering multiple ideas and forming a convincing argument.I asked the same of Llama 3.1 8B running in Ollama on my mini PC.First, the response took an age to generate; I sat and watched the reply get written one word at a time for a while, and then it became too painful, so I got on with other things.
It took over 11 minutes to get a complete response, and that response missed key factors, such as how Network Address Translation (NAT) delayed the impact of IPv4 running out of addresses by allowing multiple devices to share a public address.On my hardware, a small local LLM can't match a leading cloud-based model when tackling broad, complex questions that require lengthy answers.That doesn't mean that a local LLM is useless; I just needed to start using it in the right ways.
Related Google's Gemma AI runs locally on my $300 mini PC, and it replaced ChatGPT for more than I expected I ran Google's Gemma AI locally on a cheap mini PC, and it handled more of my everyday ChatGPT tasks than I expected.Posts 7 By Rich Hein I give my LLM small, simple tasks Being specific makes a big difference The problem I had was that with a powerful cloud-based LLM, you can often afford to be vague.Even with a vague, unfocused prompt, a powerful cloud-based LLM may be able to infer your intent and produce a useful response.
A small local LLM struggles with this.That's why I switched to giving my local LLM narrowly defined jobs with specific instructions that meant it didn't have to waste time figuring out what the prompt actually meant and could get on with the job.For example, I use a local LLM to take news from RSS feeds and summarize it into a news report.
This is a narrow and specific task: take this information and summarize it in natural language.The local LLM knows what it has to do and can produce useful results, even if it takes time to generate them.Another way I use a local LLM is to turn information pulled from Home Assistant, including calendar events and current weather, into a spoken morning briefing.
Once again, the LLM has a clearly defined task: take this collection of data and turn it into a natural language briefing.Tasks like these have clear boundaries, limited context, and predictable outputs.With these constraints, my small local LLM can do a much better job than when I ask it big questions.
It's not about having a highly detailed prompt as much as a narrow and clearly defined one; the smaller the problem the LLM has to solve, the more consistent the results.Choosing the tasks to give to a local LLM Some things aren't worth the effort I've reached the point where I know which tasks my local LLM will be able to handle and which are beyond its capabilities.There are some simple questions that can help.
First, I ask myself whether I need to use a local LLM at all.Local AI has many benefits, not least of which is privacy, but not every job needs to stay local.I also consider how broad the task is; if it's not a small, narrowly defined job, then it's not something my local LLM will handle well.
There are plenty of tasks that I still give to ChatGPT or Claude because they either don't need to stay local or are too big for my local LLM to handle.Now that I know the right questions to ask, there's a lot that my local LLM can do.Don't expect the world from a local LLM If you're lucky enough to own a powerful AI rig with a huge amount of VRAM, then you can run powerful local models that can do a lot of what cloud-based LLMs can do.
For the rest of us, local LLMs can still play a role, as long as we set our expectations accordingly.
Read More
