The Ollama library contains a lot more than the models that get written about most. While everyone knows Llama 3.1, Mistral, and Qwen 2.5, there’s a tier of excellent models that rarely comes up in conversations — models that either launched quietly, serve a niche particularly well, or have been overshadowed by newer releases despite still being highly capable. This guide covers the most worthwhile underrated models in the Ollama library, what makes each one worth trying, and who should use them.
Gemma 2 9B: The Still-Excellent Predecessor
Gemma 2 9B is Google’s previous generation model, now overshadowed by Gemma 3. But here’s the thing: Gemma 2 9B is still genuinely excellent, and it fits in a specific hardware gap that Gemma 3 doesn’t cover as cleanly. The Gemma 3 family jumps from 4B to 12B — Gemma 2 9B sits right in the 8–10GB sweet spot that the newer family skips.
ollama pull gemma2:9bAt around 5.5GB in Q4, Gemma 2 9B runs comfortably on 8–10GB RAM with a good margin for context. The instruction-following quality is excellent — it was one of the best instruction-tuned models at the 9B scale when released and hasn’t degraded. For users on 8GB machines who want better quality than Llama 3.2 3B but can’t fit a 12B model, Gemma 2 9B is a strong option that’s often overlooked.
Where it particularly stands out is long-form writing and explanation quality. The output reads naturally and coherently over longer responses — a characteristic of Google’s training approach that persists in Gemma 2. For documentation writing, technical explanation, and educational content, it’s often preferable to same-size alternatives.
The main reason to choose Gemma 2 9B over Gemma 3 4B: if you specifically need the 9B parameter count for task quality and your hardware sits in the 8–12GB range, Gemma 2 9B gives you more capacity than the 4B while fitting in less memory than the 12B. It’s the right choice for users in that specific hardware window who’ve already tried the 4B and found it insufficient.
Solar 10.7B: Depth-Upscaled Excellence
Solar 10.7B from Upstage used a novel depth upscaling technique — combining layers from two Llama 2 13B models in a specific way to create a model that punches above its apparent parameter count. When it was released it beat models significantly larger than itself on multiple benchmarks, and it remains impressive despite its age.
ollama pull solarAt around 6.1GB in Q4, Solar 10.7B runs on 8–12GB RAM and produces output quality that often rivals 13B models. Its particular strengths are reasoning tasks and instruction following — the depth upscaling seems to give it more coherent multi-step reasoning than you’d expect from a 10.7B parameter count. For logical analysis, structured Q&A, and tasks requiring careful reasoning over multiple steps, Solar frequently surprises users who expect 7B-class output.
Solar is especially worth trying if you’ve been using Mistral 7B and want an upgrade that doesn’t require much more memory. The jump in reasoning quality from Mistral 7B to Solar 10.7B is more pronounced than the size difference suggests, and both fit comfortably on the same hardware.
Mistral Nemo 12B: The Practical Multilingual Model
Mistral Nemo 12B was a collaboration between Mistral AI and NVIDIA, and it’s quietly one of the best models at the 12B scale for multilingual use. While Qwen 2.5 gets the attention for multilingual capability, Mistral Nemo has particularly strong European language support — French, German, Spanish, Italian, and Portuguese all feel native rather than translated.
ollama pull mistral-nemoAt around 7.1GB in Q4, it needs about 10–12GB RAM comfortably. If your work involves European languages and you want a model that handles them with the same fluency as English, Mistral Nemo is worth keeping alongside your main model. It also benefits from a very long context window — 128K tokens — and handles long documents in European languages better than alternatives that were trained primarily on English text.
Orca Mini: Reasoning in a Tiny Package
Orca Mini might be the most underrated model for users with very constrained hardware. It’s a 3B model fine-tuned using Microsoft’s Orca method — training on chain-of-thought reasoning data generated from GPT-4. Despite its size, it produces notably more structured and logical responses than typical 3B instruction models.
ollama pull orca-miniAt under 2GB, Orca Mini runs on machines with 4GB RAM and generates at 15–25 tokens per second on CPU. For users who can’t run anything larger — old laptops, low-spec machines, Raspberry Pi — it produces more thoughtful responses than its size would suggest. It won’t match a 7B model but it’s a meaningful step above the typical tiny instruction-tuned model for tasks requiring any reasoning.
Nous Hermes: The Fine-Tune That Outperformed Its Base
Nous Hermes is a series of instruction fine-tunes by Nous Research that consistently produced models better than their base model equivalents. The Hermes 3 variants based on Llama 3.1 are particularly worth knowing about:
ollama pull hermes3 # Hermes 3 on Llama 3.1 8B
ollama pull hermes3:70b # Hermes 3 on Llama 3.1 70BNous Hermes models are fine-tuned with a particular focus on following complex system prompts, maintaining character consistency in roleplay and creative contexts, and producing long-form structured output reliably. If you’re building applications where system prompt adherence is critical — chatbots with specific personas, structured data extraction pipelines, or any use case requiring consistent formatting — Hermes 3 is often better than the vanilla Llama 3.1 fine-tune at the same size.
Figure 1 — Underrated Models: Hardware and Use Case Overview
Yi 34B: The Chinese-English Powerhouse
Yi 34B from 01.AI is one of the strongest models for Chinese-English bilingual work outside of Qwen. Where Qwen 2.5 dominates on Chinese tasks with modern training, Yi 34B has a different character — its Chinese output is particularly natural and its ability to switch fluidly between Chinese and English in the same conversation is excellent.
ollama pull yi:34bAt around 20GB in Q4, Yi 34B needs 24GB+ RAM or VRAM. It’s a niche pick — primarily useful for users who specifically need high-quality Chinese language generation and find Qwen’s output style doesn’t match their preferences. The model’s general English capability is also strong, making it a viable primary model if you’re doing Chinese-English work and have the hardware for the 34B size.
Open Hermes Mixtral: MoE with Better Instructions
Open Hermes Mixtral applies the Nous Hermes instruction-tuning to the Mixtral 8x7B base, combining Mixtral’s MoE efficiency with Hermes’s superior instruction following. If you’re already running Mixtral 8x7B and want better system prompt adherence and structured output, this is a direct upgrade:
ollama pull openhermes # based on Mistral/Mixtral variantsThe Hermes fine-tuning is particularly noticeable on tasks that require the model to maintain a specific format or persona across a long conversation. Where base Mixtral occasionally “breaks character” or reverts to a generic assistant style despite system prompt instructions, Open Hermes Mixtral maintains the specified behaviour more reliably.
Phi-3 Medium: The Forgotten Microsoft Model
While Phi-4 gets all the attention, Phi-3 Medium (14B) is still in the Ollama library and still excellent for its size. More importantly, it’s one of the most polished models for safe, structured deployment — Microsoft trained it with a strong focus on avoiding harmful outputs and maintaining consistent behaviour, which matters for production applications.
ollama pull phi3:medium # 14B parameter versionAt around 7.9GB in Q4, it fits in the same hardware tier as models significantly smaller. The quality is competitive with Llama 3.1 8B on most tasks and exceeds it on structured reasoning. If you’re deploying a local AI assistant in an environment where consistent, predictable outputs matter more than peak capability, Phi-3 Medium’s careful training makes it more reliable than alternatives in its size class.
How to Find More Hidden Gems
The Ollama library at ollama.com/library contains far more models than get covered in popular guides. A few strategies for finding worthwhile ones that haven’t made the rounds yet. Sort by “Recently Updated” to see what’s been added or updated — active models getting fresh uploads usually means active community use. Check the pull count for models you haven’t heard of — a model with 50K+ pulls is being used for a reason even if it’s not getting written about. Look at the tags for models you already use — if you like Mistral, check what else is tagged “mistral” or “fine-tune” — there are often excellent community fine-tunes of your favourite base models with specific improvements for particular use cases. And periodically check r/LocalLLaMA on Reddit — it surfaces high-quality new releases and community fine-tunes faster than any other source, often with practical first-hand benchmarks from users who’ve tested on real hardware rather than just citing academic numbers.
Figure 2 — When to Try an Underrated Model
The Case for Keeping an Old Favourite
A recurring pattern in the local LLM space is abandoning perfectly good models the moment something new releases. Gemma 2 9B, Solar 10.7B, and Mistral Nemo are all still excellent models — the fact that Gemma 3 and Qwen 2.5 are better on aggregate benchmarks doesn’t mean the older models have become useless. Benchmarks measure average performance across many tasks; your specific use case might be one where an “older” model is actually the better fit. Solar’s depth upscaling produces reasoning patterns that some users prefer over more recent models even when benchmarks say otherwise. Gemma 2’s writing quality has a particular character that some users find more natural than Gemma 3’s more precise outputs. These are legitimate reasons to keep older models in your toolkit alongside the newest releases, and the small disk cost of having a few extra models downloaded is worth the flexibility of having options when the default recommendation doesn’t quite fit your needs.
Testing an Underrated Model: A Practical Approach
The best way to evaluate whether any of these models fits your workflow is a structured test rather than just running a few casual prompts. Pick three representative tasks from your actual work — maybe a code review, a document summary, and a complex explanation request. Run the same prompts on your current model and the candidate model. Compare not just accuracy but output style: is the length appropriate? Does it follow the instruction structure you gave? Does it handle the specific domain knowledge your tasks require? This kind of side-by-side comparison reveals things that benchmark scores don’t — a model that scores slightly lower on MMLU might produce better-structured output for your specific use case, or might handle your domain’s terminology more reliably. Keep the comparison honest by using prompts you’d actually send in real work rather than synthetic test cases, and give each model two or three attempts on the hardest task before deciding — single-run comparisons are noisy, especially for complex prompts where there’s variance in how models interpret the instruction. The goal isn’t to find the universally best model; it’s to find the best model for your specific combination of hardware, use case, and quality preferences.
Maintaining a Curated Local Model Library
Once you’ve explored the underrated options, the practical challenge is deciding what to keep permanently versus what to pull on-demand. A useful framework: keep your primary workhorse (likely Llama 3.1 8B, Qwen 2.5 7B, or similar) always downloaded and loaded. Keep one or two specialists that you use regularly enough to justify the disk space — a coding model, a vision model, or a multilingual model depending on your work. For the more niche options — Yi 34B, Open Hermes Mixtral, Solar — keep them on a list of models to pull when you have a task that specifically fits their strengths, rather than keeping them permanently installed. The Ollama pull process is fast enough on a decent connection that pulling a 5–6GB model when you need it doesn’t feel like a burden, and your main SSD will thank you for not keeping 200GB of models permanently downloaded. The discovery process itself has value too — regularly spending an afternoon testing a model you’ve never used before is one of the best ways to stay current with what’s available locally, and the underrated models often contain surprises that inform how you think about your primary model choices.
Why These Models Don’t Get More Attention
It’s worth understanding why models like Solar 10.7B and Gemma 2 9B fly under the radar despite being genuinely good. The primary reason is the news cycle — model releases happen fast, and each new model pushes discussion of the previous one aside even when the previous model is still competitive. Solar 10.7B was genuinely impressive when it released but was quickly overshadowed by the Llama 3 and Mistral releases that followed. The second reason is that most comparison guides focus on a fixed set of flagship models — Llama, Mistral, Qwen, Gemma — because they’re the ones with the most community documentation and integration support. A model needs to be in common guides to get integration support, and it needs integration support to appear in guides — a circular problem that keeps lesser-known models perpetually overlooked. The third reason is hardware diversity: models like Mistral Nemo 12B require a specific hardware tier (10–12GB RAM) that sits between the most common consumer setups (8GB) and the setups people write about most (24GB+). The middle tier of hardware and models is underserved in most coverage, which is exactly where models like Solar, Gemma 2 9B, and Mistral Nemo live. Understanding this gap is useful because it means there are likely good models you haven’t tried yet that would fit your hardware and workflow well — the exploration is worth doing.
Combining Models for Different Tasks
One underappreciated approach is using different models for different task types rather than forcing a single model to do everything. A practical setup might be: Qwen 2.5 Coder 7B for code tasks (fast, specialised), Gemma 2 9B for writing and explanation (excellent long-form output quality), and Solar 10.7B for reasoning-heavy tasks (depth upscaling gives better logical analysis). This three-model setup covers most development and writing work with models that each fit the 8–10GB RAM range — you’re switching between them in Ollama with a single model name change rather than maintaining one large model that’s mediocre at everything. The switching overhead is the load time (15–30 seconds for a cold load) which is only paid once per session if you keep keep-alive set appropriately. The quality improvement from using a specialist rather than a generalist for each task type is worth that one-time cost. Thinking about your local model library as a collection of specialists rather than a single all-purpose model is a more mature approach to local AI, and it’s one that the underrated models in this guide make practical without requiring you to upgrade to 24GB+ hardware.