People usually assume that local AI requires a powerful GPU.To some degree that is true—many things do require one.However, there is a huge range of models that are capable of doing useful things.
If you have an old PC or laptop made in the last 5-or-so years and at least 8GB of RAM, you should expect no more than around 10 tokens per second.That doesn't sound like much, but in many applications, the raw tokens per second don't actually matter very much.Fire up Ollama One command to install, one to run, and an API for everything else Close Ollama has become the go-to solution for most people self-hosting AI services.
It provides a built-in library and an OpenAI-compatible interface that runs a server locally.That makes it an ideal choice if you want to build an AI using an open weight model that actually performs a function rather than just chatting with it.The big advantage is convenience.
Ollama handles many of the tedious details, like hardware detection and quantization selection, for you.Keep an eye on quantization settings: it is easy to select the wrong format, and two bits of quantization can make a huge difference in system performance.Once you find a model you like, run ollama pull (modelname), then ollama run (modelname).
Experiment with llama.cpp Bare bones but fast if you need every bit of performance Llama.cpp is an inference engine designed to run GGUF-format models.It lacks interface overhead and uses the most aggressive CPU optimizations available.That makes it a good choice if you need every bit of performance, but it isn't without drawbacks.
The trade-off is that it is command-line heavy (though there is a web UI); you have to manage your own model files and understand quantization formats.To get started, download a release binary or build it yourself, download a GGUF model from Hugging Face, and run llama-cli or llama-server.Related I switched from LM Studio to llama.cpp, and I'm never going back to a bloated wrapper Don't be deadset on using LM Studio when there is a great alternative, llama.cpp, that does everything you need without the extra bloat.
Posts 2 By Jorge A.Aguilar One critical setting to watch: cap your context length rather than accepting the defaults.The KV cache can eat up your RAM very quickly, and setting it too high is a quick way to bog down your system.
Try Gemma 4's edge models E2B and E4B were built for low-power hardware The two smallest members of the Gemma 4 family—E2B and E4B—were released in Spring 2026 specifically for "edge" hardware.The "E" stands for effective parameters, which provide depth without requiring the massive memory a dense model would need.For example, E4B runs on about 5GB at 4-bit quantization.
These models are great at a whole host of basic activities, including summarizing, drafting, extracting structured data, and translation.So, if you need something to parse PDFs to pull out billing amounts, they're a good choice.They can even reason a bit, though they'll struggle compared to larger thinking models.
When using them via Ollama, you should match the variant to your available RAM—use E4B if you have 8GB, and E2B if you have less.If you have to adjust your context, don't go too high—that eats into your RAM too.Automatic image description Captioning a photo library is the ideal slow-laptop project One of my favorite local AI projects is a small program that automatically captions my screenshots.
My specific setup runs on a GPU, but there is no reason you couldn't run it on a CPU, especially if it doesn't need to be done in real time.Batch-generating captions or alt text for a folder of images is a great project for a slow laptop specifically for that reason.If a model produces three tokens per second while you're at work or asleep, who cares if it is a bit slow? There are several options that can do this: Moondream2 (or Moondream3) (1.6B parameters)—Built for edge use and easily fits in 3GB with the Q4 quantization SmolVLM 2B—An open-weight model that handles batch jobs well.
Gemma 4 E4B—Every version has vision capabilities included.If you need to churn through a decade of photos and make them searchable, this can be incredibly useful.I'd recommend starting with a set of about a dozen very different images to see how your prompt and model work committing to a 10,000 photo archive.
Private document search You can intelligently search your documents Close Most search built into PCs is basically a "dumb" search.It looks for explicit terms or criteria that you enter.With AI, you can implement a semantic search, which allows you to describe what you're looking for in more organic terms that don't necessarily explicitly match the text in a document.
There are a number of different ways you can approach this, but I'd recommend trying EmbeddingGemma first.It only has 308M parameters, and can run with less than 200MB of RAM when quantized.You can even truncate its output (embeddings) from 768 dimensions down to 128 with minimal quality loss.
It pairs well with AnythingLLM or Open WebUI directed at an Ollama server running EmbeddingGemma.Just remember that scanning a large document archive on a CPU will take hours.If you have low-power hardware, keep your AI job-specific If you want the fastest possible setup, use llama.cpp.
However, for everyone else, I'd recommend Ollama.The convenience it offers is literally a game changer.If you don't have any specific model in mind for a project, I'd strongly recommend starting with Gemma 4's edge variants.
They can do a ton of different things that make them useful and will run on a potato.
Read More
