Totally depends on your hardware, and what you tend to ask it. What are you running? What do you use it for? Do you prefer speed over accuracy?

I have a MacBook 2 pro (Apple silicon) and would kind of like to replace Google's Gemini as my go-to LLM. I think I'd like to run something like Mistral, probably. Currently I do have Ollama and some version of Mistral running, but I almost never used it as it's on my laptop, not my phone.

I'm not big on LLMs and if I can find an LLM that I run locally and helps me get off of using Google Search and Gimini, that could be awesome. Currently I use a combo of Firefox, Qwant, Google Search, and Gemini for my daily needs. I'm not big into the direction Firefox is headed, I've heard there are arguments against Qwant, and using Gemini feels like the wrong answer for my beliefs and opinions.

I'm looking for something better without too much time being sunk into something I may only sort of like. Tall order, I know, but I figured I'd give you as much info as I can.

I have a MacBook 2 pro (Apple silicon) and would kind of like to replace Google's Gemini as my go-to LLM. I think I'd like to run something like Mistral, probably. Currently I do have Ollama and some version of Mistral running, but I almost never used it as it's on my laptop, not my phone.

I'm not big on LLMs and if I can find an LLM that I run locally and helps me get off of using Google Search and Gimini, that could be awesome. Currently I use a combo of Firefox, Qwant, Google Search, and Gemini for my daily needs. I'm not big into the direction Firefox is headed, I've heard there are arguments against Qwant, and using Gemini feels like the wrong answer for my beliefs and opinions.

I'm looking for something better without too much time being sunk into something I may only sort of like. Tall order, I know, but I figured I'd give you as much info as I can.

Actually, to go ahead and answer, the "fastest" path would be LM Studio (which supports MLX quants natively and is not time intensive to install), and a DWQ quantization (which is a newer, higher quality variant of MLX models).

Hopefully one of these models, depending on how much RAM you have:

https://huggingface.co/mlx-community/Qwen3-14B-4bit-DWQ-053125

https://huggingface.co/mlx-community/Magistral-Small-2506-4bit-DWQ

https://huggingface.co/mlx-community/Qwen3-30B-A3B-4bit-DWQ-0508

https://huggingface.co/mlx-community/GLM-4-32B-0414-4bit-DWQ

With a bit more time invested, you could try to set up Open Web UI as an alterantive interface (which has its own built in web search like Gemini): https://openwebui.com/

And then use LM Studio (or some other MLX backend, or even free online API models) as the 'engine'

Alternatively, especially if you have a small RAM pool, Gemma 12B QAT Q4_0 is quite good, and you can run it with LM Studio or anything else that supports a GGUF. Not sure about 12B-ish thinking models off the top of my head, I'd have to look around.

This is all new to me, so I'll have to do a bit of homework on this. Thanks for the detailed and linked reply!

I was a bit mistaken, these are the models you should consider:

https://huggingface.co/mlx-community/Qwen3-4B-4bit-DWQ

https://huggingface.co/AnteriorAI/gemma-3-4b-it-qat-q4_0-gguf

https://huggingface.co/unsloth/Jan-nano-GGUF (specifically the UD-Q4 or UD-Q5 file)

they are state-of-the-art at this size, as far as I know.

Totally depends on your hardware, and what you tend to ask it. What are you running? What do you use it for? Do you prefer speed over accuracy?

My HomeAssistant is running on Unraid but I have an old NVIDIA Quadro P5000. I really want to run a vision model so that it can describe who is at my doorbell.

TBH you should fold this into localllama? Or open source AI?

I have very mixed (mostly bad) feelings on ollama. In a nutshell, they're kinda Twitter attention grabbers that give zero credit/contribution to the underlying framework (llama.cpp). And that's just the tip of the iceberg, they've made lots of controversial moves, and it seems like they're headed for commercial enshittification.

They're... slimy.

They like to pretend they're the only way to run local LLMs and blot out any other discussion, which is why I feel kinda bad about a dedicated ollama community.

It's also a highly suboptimal way for most people to run LLMs, especially if you're willing to tweak.

I would always recommend Kobold.cpp, tabbyAPI, ik_llama.cpp, Aphrodite, LM Studio, the llama.cpp server, sglang, the AMD lemonade server, any number of backends over them. Literally anything but ollama.

...TL;DR I don't the the idea of focusing on ollama at the expense of other backends. Running LLMs locally should be the community, not ollama specifically.

Thanks for Lemonade hint. For Ryzen AI: https://github.com/lemonade-sdk/lemonade (linux=cpu for now)