A single 24GB graphics card has quietly become the practical floor for serious local AI inference in 2026. The old hobbyist playbook of cramming the largest possible 70B quantized model onto the card is now outdated. The sharper approach is to run modern 20B to 35B models that leave breathing room for context, KV cache, and runtime overhead while still delivering fast enough response times for coding, chat, and agentic workflows.
The shift is driven by a wave of permissively licensed, tightly optimized models released in the first half of 2026. All of the models discussed here ship under either Apache 2.0 or MIT licenses, and every one fits comfortably on a single 24GB card at the Q4_K_M quantization level. The question is no longer whether you can run local AI. It is which model matches your actual workload.
How 24GB Gets Spent
Three things eat VRAM during inference: model weights, the KV cache, and runtime overhead. Get the split wrong and the model crashes mid-generation.
Model weights are the biggest chunk. At Q4_K_M, the default for most home setups, each parameter costs roughly 0.58 bytes. A 32B dense model therefore needs about 18 to 20GB just to sit in memory. The KV cache grows with prompt and session length. Longer conversations mean more VRAM reserved for attention state. Runtime overhead, including CUDA buffers and temporary tensors, typically adds another 1 to 2GB at short context.
The common trap is Mixture-of-Experts, or MoE, models. In architectures like Mixtral, every expert stays resident in VRAM even though only a small subset routes per token. Memory planning must use total parameters, not active parameters. A 35B MoE with 3B active per step still needs space for all 35B. This is why some seemingly smaller MoE models feel tighter than their dense counterparts.
Quantization is the main lever for making things fit. Q4_K_M is the home standard for a reason: it preserves enough quality for coding and reasoning while keeping footprints manageable. Q5_K_M and Q6_K trade memory for fidelity. Q8_0 and BF16 are usually non-starters for 30B-class models on a single 24GB card.
The Models That Actually Fit
Qwen3-Coder-27B from Alibaba is the strongest single default for this tier. Released in April 2026 under Apache 2.0, it is a dense 27B model built for agentic coding, repository-level reasoning, and frontend development workflows. At Q4_K_M it consumes roughly 16GB, leaving generous headroom for long files and multi-turn debugging sessions.
Qwen3-32B-A3B is the MoE sibling. It carries 35B total parameters with about 3B active per token, which gives it far faster decode speeds than a dense 35B model. The memory footprint still tracks the full 35B, so budget around 20GB at Q4_K_M. It is a tight but valid fit, best suited for general chat and tool use where latency matters.
Gemma 4 from Google DeepMind shipped on April 2, 2026, and was notable as the first Gemma generation with a fully open Apache 2.0 license. The family includes edge models, a 12B unified multimodal variant, a 26B MoE with 3.8B active parameters, and a larger dense flagship. The 26B MoE is the natural pick for 24GB setups that need vision input and coverage across 140-plus languages.
Mistral Small 3.2 targets everyday assistant workloads with an emphasis on low latency. It is a 24B dense model under Apache 2.0, and at roughly 14GB at Q4_K_M it is the lightest footprint in this group. That spare room lets you push context length further than the heavier 32B options allow. Mistral shipped larger successors in 2026, but those sit above the single-card tier.
OpenAI o4-mini is an open-weight reasoning model under Apache 2.0. It uses an MoE design with 21B total parameters and 3.6B active per token, shipping in a native MXFP4 4-bit format that loads in about 14GB. It is strong at structured reasoning and tool use, though weaker on broad world knowledge compared to the larger dense models.
DeepSeek-R1-Distill-Qwen-32B distills reasoning traces from DeepSeek's R1 into a smaller dense model built on a Qwen2.5 base. Released under MIT, it is the tightest fit here at roughly 18 to 20GB at Q4_K_M. It exposes its chain of thought through visible reasoning tokens, which is useful for slow, deliberate problems where you want to audit the model's logic.
What Does Not Fit (And Why That Matters)
The frontier open models of 2026 are massive sparse MoE systems that simply will not run on one consumer card. GLM-5.2 from Z.ai sits at roughly 753B total parameters. Moonshot's Kimi K2.7 is around 1 trillion. DeepSeek's V4 preview, released in April 2026, includes a V4-Pro checkpoint near 1.6T parameters. Alibaba's Qwen3.5-397B and Mistral Large 3 occupy the same server-class range.
Because MoE memory tracks total parameters, all of these need multi-GPU rigs or high-memory unified systems. They are worth knowing as API options. They just do not change what is possible on a single 24GB card.
This matters for the growing community of developers, researchers, and privacy-conscious users who want to run models locally without cloud dependency. The gap between what fits on a desktop GPU and what runs in a datacenter is widening, but the models in the middle are now good enough that the gap hurts less than it used to.
Runtimes and Where to Start
Three inference engines cover nearly every 24GB setup. LM Studio is the simplest path, with automatic quantization selection and an OpenAI-compatible local API. llama.cpp gives fine-grained control over GGUF quantization and CPU-GPU offload splits. vLLM is the throughput-focused server for heavier concurrent workloads.
The practical advice is to start with one model that matches your main job, not the biggest file you can load. Keep context length under control, let quantization do the heavy lifting, and treat the 24GB card as a precision tool rather than a storage bin. The models are now good enough that the bottleneck is rarely raw parameter count. It is picking the right architecture for the task at hand.
Watch for Alibaba's next Qwen-Coder release and any Mistral Small revision that adds multimodal depth. Both are likely to land in this same memory envelope and could shift the default recommendation before the year ends.