The Archive

llama.cpp release b9180 ships MTP support, enabling improved inference optimization for local LLM deployment.

Community discussion of Qwen 27B quantization and inference optimization on single RTX 3090 GPU.

Qwen3.6 27B-MTP shows 11.5% faster wall-clock time vs base on single-turn; 35B-MTP regresses 11.2%, with generation speedups offset by prompt processing slowdowns.

MTP support merged into llama.cpp master branch (PR #22673); no context on feature impact.

MTP PR merged into llama.cpp; technical details absent.

Looks like it finally happens... MTP getting approved for llama.cpp. Time to prepare for the update.

Engineer deployed Gemma 4 E4B on Jetson Orin NX with 30+ sensors, 200ms TTFT, offline multimodal robotics stack using llama.cpp and Piper TTS.

RTX 5060 Ti local LLM configuration guide covering vLLM and llama.cpp serving of Qwen 27B/35B models with quantization and long-context presets.

Llama-Studio: open-source WebUI for managing multiple local llama-server instances and configurations.

Hugging Face releases ml-intern, an agent framework for local LLM research automation supporting llama.cpp/ollama with Qwen and Claude models.

Multi-token prediction + TurboQuant quantization achieves 40% throughput gain (21→34 tokens/s) on Qwen 27B/35B via LLaMA.cpp on M-series Mac.

Qwen 3.6 35B and Gemma 4 26B MoE models achieve 20–24.5 tok/s on GTX 1080 with 128k context via llama.cpp quantization.

Community Docker images for llama.cpp with MTP support to simplify local model inference setup across CUDA versions.

Reddit discussion comparing vLLM vs llama.cpp for single-user local inference on AMD GPUs.

r/ClaudeAI • also crosspost to r/LocalLLaMA and r/artificial I lost $187 to this and want to save others the same headache. **What happened** I run Claude Code headlessly via Windows Task Scheduler. My project repo has a `.env` file with `ANTHROPIC_API_KEY` set — legitimately, for a separate Express server doing AI-based transaction classification. Nothing to do with Claude Code itself. Claude Code reads environment variables from the `.env` in its working directory on launch. When it finds `ANTHROPIC_API_KEY` there, it silently uses that key for billing instead of your OAuth ...

Luce ships DFlash+PFlash optimizations for AMD Ryzen AI MAX+ 395, achieving 2.23x decode speedup on Qwen 3.6-27B vs llama.cpp HIP.

llama.cpp adds llama-eval benchmarking tool supporting AIME, GSM8K, GPQA for local quantized model evaluation.

RTX 4090 power optimization for llama.cpp: reduce consumption 40% via power limits without performance loss.

Tuning llama.cpp ubatch and n-cpu-moe parameters improves gpt-oss-120b prompt processing from 240 to higher tok/s on RTX 3090.

Empirical study across 288 model calls identifying JSON output failures in Llama 3, Mistral, Command R, DeepSeek, Qwen; failure modes consistent across open and closed models but vary by rate.

JSON parsing bug in llama-server's chat-template-kwargs for Qwen3.6 preserve_thinking parameter requires whitespace-free formatting.

ExLlamaV3 adds Gemma 4 support, improved caching, and DFlash optimization for faster LLM inference on consumer hardware.

User successfully quantized and ran DeepSeek V4 Pro locally on AMD EPYC + RTX PRO hardware using modified llama.cpp with Q4_K_M compression.

User shares llama-server configuration for running Minimax 2.7 at 100k context on Strix Halo hardware with detailed tuning notes.

Reddit user seeks advice on LLaMA inference harnesses; discusses fragmentation and compatibility issues with local LLM tooling.

BeeLlama.cpp fork adds DFlash, TurboQuant, and vision support; runs Qwen 3.6 27B Q5 on RTX 3090 with 200k context at 135 tps.

User reports 1.5–2x speedup running Qwen 27B with MTP optimization on dual AMD MI50 GPUs via llama.cpp.

User achieves 80 tok/sec with 128K context on RTX 4070 Super using Qwen3.6 35B quantization and llama.cpp MTP implementation.

Reddit user asks about llama.cpp timeline for Vulkan/HIP MTP support on Strix Halo Windows 11.

Linear probing and activation patching reveal latent planning representations in Qwen3, Gemma-3, Llama-3; future constraints encoded at layer boundaries.