Llama

Microsecond-cost anomaly detection for LLM agent failures using one-class echo-state networks trained only on healthy runs; tested across Qwen, Llama, and Gemini agents.

ARB benchmark evaluates AI-text detectors against LLM-rewritten human content using Llama-3.2 and Qwen2.5 generators.

Structured extraction of event type, impact scope, and temporal horizon from financial news using LLaMA-3.1-70B outperforms sentiment-only prediction.

Benchmark of six VLMs (Gemini, GPT-4V, Qwen, Gemma, Llama, Ministral) on zero-shot anomaly detection for game geometry clipping in agent-driven QA.

DRL trading system for Bitcoin/Tesla using policy gradient and Q-learning with LLaMA 3.2 sentiment analysis and technical indicators.

Study compares production vs. perception asymmetry in Llama-3.1-8B via token probability analysis, finding LLMs lack functional production-perception distinction.

Benchmark-backed Ollama has amassed 176,000 stars, and nearly 17,000 forks on Github by helping developers easily run AI on their PCs.

PALS adjusts per-layer sparsity in LLM pruning via activation percentiles, improving LLaMA-2-7B perplexity by 15% at 50% sparsity over uniform Wanda.

Meta is launching the first AI image generation model made by its Superintelligence Labs division. The Muse Image model now powers the image-making tools across the Meta AI app, Instagram, and WhatsApp, and it's coming soon to Facebook and Messenger, according to an announcement on Tuesday. It's part of the growing Muse family of AI models that replace Meta's Llama lineup. Alexandr Wang, who Meta hired to head up its Superintelligence Labs last year, says on Threads that Muse Image is "agentic," meaning it works with its Muse Spark large language model "to reason through your prompt, search t...

Systematic study of reward function design for RL-based BPMN process model generation using Llama 3.1 and Qwen 2.5 across 48 configurations.

Production-scale clinical NLP study of inference-time gating with Llama-3.3 70B generator and MMed-Llama-3.1 70B verifier over 167K narratives shows pattern-memory filtering limitations.

Three-method study across Qwen2.5-Coder-32B, Llama-3.1-8B, and Gemma-3-27B shows internal probes read situation not pre-action intent, limiting misalignment monitoring efficacy.

Extractive-abstractive hybrid summarization for legal case judgements using tree-of-thoughts with DeepSeek and Llama.

Fine-tuning language models on insecure code induces emergent misalignment with poorly understood internal structure. We investigate whether this misalignment corresponds to a causally actionable activation-space direction shared across architectures. Across four instruction-tuned model families (Qwen2.5-1.5B, Gemma-2-2B, Llama-3.2-1B, Ministral-3-3B) finetuned identically, a difference-in-means direction achieves 99.6% separation of aligned and misaligned activations at each model's final layer. Causal steering by subtracting this direction reduces code spillover by 21-51 points, while a sec...

Recent advances in large language models (LLMs) have produced many specialized multimodal LLMs (MLLMs) that share common foundational LLMs, forming distinct model lineages. It remains unclear whether a fundamental behavioral link exists between the foundational LLMs and downstream variants. We investigate this question by quantifying head-level context-truthfulness scores. Across diverse LLM and MLLM lineages, including Vicuna-, Qwen2.5-, LLaMA2-, and Mistral-based models, we find that Truth Scores are strongly preserved within model families, even after instruction tuning or multimodal adapt...

Real-time, full-duplex speech interaction is a key feature of next-generation spoken chatbots, allowing the model to listen and speak at the same time and to handle natural phenomena such as overlap, hesitation, and barge-in. Existing speech language models (SpeechLMs) such as LLaMA-Omni and GLM-4-Voice are still turn-based and rely on an external Voice Activity Detection (VAD) module to mark the end of the user's turn, which fundamentally limits their interactive ability. In this paper, we introduce BayLing-Duplex, a native full-duplex SpeechLM where a single autoregressive LLM decides when ...

This study examines training dynamics in a small Llama-style language model trained under a fixed, compute-constrained token budget. Rather than evaluating efficiency solely through endpoint performance, the study uses a quantitative experimental repeated measures design to analyze how validation loss, validation perplexity, rolling volatility, backslide behavior, spike behavior, and between-seed variability change across token-based training intervals. Six independent training runs were conducted on a 4.26-million-parameter model using the TinyStories corpus, CPU-based full-precision trainin...

This study investigates cross-lingual distributional skew (the Shibboleth Effect) in frontier large language models (LLMs) subjected to sustained adversarial conditions. We develop a multi-agent geopolitical wargame, the Cerulean Sea Crisis, a synthetic maritime territorial dispute designed to mirror the structural dynamics of Eastern Mediterranean conflicts. Six frontier models (GPT-4o, Llama-4, Mistral-Large, Gemini-3.1-Pro, Qwen3.6-Plus, and DeepSeek-R1) participate in a between-groups experiment (N = 10 games per arm, K = 5 rounds per game) in which the sole manipulation is the language o...

AutoMegaKernel (AMK) compiles a HuggingFace Llama-family model into a single persistent cooperative CUDA kernel that runs the whole forward pass in one launch, with no per-model hand-written CUDA. The contribution is the system, not raw speed. A frozen schedule-IR validator statically certifies deadlock-freedom and race-freedom via static graph checks (not a mechanized proof), so an unsafe agent-proposed schedule is rejected before launch: across 7,160 adversarial schedules (6,091 unsafe) it had zero false-accepts and accepted all 360 real lowerings. The same source retargets sm_80/sm_90/sm_1...

We propose a preconditioning (PC) layer, a weight parameterization via polynomial preconditioner that ensures stable weight conditioning throughout LLM training. The PC module reshapes the singular-value spectrum of weight matrices via low-degree polynomial preconditioning. After training, the preconditioned weights can be merged back into the original architecture, incurring no inference overhead. We demonstrate the advantage of the proposed PC layer over standard transformers in Llama-1B pre-training, for both the AdamW and Muon optimizers. Theoretically, we justify this spectrum-control pr...

AI-assisted coding agents are bottlenecked by input-token cost. Two pathologies of raw human input drive much of this overhead: tokenization inefficiency for non-English text and structural entropy in conversational prompts. Existing approaches act reactively by compressing already-bloated contexts or intervening after failures occur. We introduce a pre-flight, edge-side prompt-rewriting middleware that operates between the developer and the cloud agent. A local Llama 3.2 (3B) model performs cross-lingual translation into English, structural rewriting into a compact task-oriented format, and ...

Temperature-zero BF16 LLM inference is often treated as reproducible, yet the same request can emit different tokens when decoded alone or inside a larger batch. Existing fixes use batch-invariant operators or LLM-42's per-token verification, incurring cost even when most steps are stable. We ask whether verification can be applied exclusively to flipped tokens. Across five models, batch-induced token flips are sparse on the flip-rate benchmarks: on MATH500, Llama-3.1-8B flips on $0.48\%$ of synchronous decode steps, and all tested models stay within the 0.3-1.3% range on MATH500, GSM8K, and ...

So, last week I tried to update my unused local LLM setup. I had to stop using it because quality was too low and deepseek was too cheap. First thing I stopped using Ollama and now I only use llama.cpp built in server that works really great. The quality improvement from Q4 to Q6 is outstanding and finally a local LLM server can work very similarly to paid APIs. That's great! And MTP makes a big performance gain, on a dual 3090 (downvolted and limited to 65°C) it generates from 20 to 50 tokens per second with minimal heat generation. So yes, that time has finally arrived! Local coding age...

Here's my article with **38 quant pairs** thoroughly benchmarked in KLD with **3 different Qwen 3.6 27B configs**: Q5\_K\_S + 64k context, IQ4\_XS + 64k context, IQ4\_XS + 128k context. This allows us to track not only how cache quantizations affects the precision in a vacuum, but also how it interacts with noise from the model itself. All benchmarks were done using my [BeeLlama.cpp](https://github.com/Anbeeld/beellama.cpp) fork, allowing to include a number of quant types that are not present in mainline llama.cpp: vanilla TurboQuant, TCQ 3-bit/2-bit, and q6\_0. [https://anbeeld.com/articl...

[Cuda 13.3 Downloads](https://developer.nvidia.com/cuda-downloads) [Release Notes](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html) Anybody already tried llama.cpp with 13.3?

Here's the PR by pedapudi. https://github.com/ggml-org/llama.cpp/pull/21344 It's merge request has been denied so it will not be in mainline llama.cpp. The changes are so small that I just put them into whatever the current release of llama.cpp is. Read the PR for more info. It will only work with MOEs. Also, it gives the most boost at low context. As the context rises, the gain diminishes. Pedapudi explains why that happens in the PR. Here are some numbers. It really works well. The tiny amount of time it takes me to apply the code to the current release of llama.cpp is time well spent. ...

I've fine-tuned Qwen 3.5 0.8B on the dataset provided by Pangram with their EditLens paper. It's available via a [Chrome extension](https://chromewebstore.google.com/detail/slop-hammer/gfjdmhfokmhedlgfggmmgchpppmhkdgg); you can just click selected text and it's going to give you the probability distribution of how likely it is AI-generated. It takes under 1s on my M1 MacBook Pro. Pangram did release Llama 3.2 3B trained on their dataset, but I found this model slightly too legacy (too big for the capabilities). Qwen 0.8B (base) ended up being as good after roughly 20h of fine-tuning on a sin...

llama.cpp adds fast Walsh-Hadamard transform (FWHT) for CUDA, yielding 1–2% prompt-processing and 7–9% token-generation speedups with quantized KV-cache.

Financial Times reports Heretic tool removes guardrails from Meta's Llama 3.3 in <10 minutes; 3,500+ decensored variants downloaded 13M times.

llama.cpp PR addresses checkpoint creation inefficiency when context optimization tools modify conversation history in agentic workflows.

Community discussion on local LLM frontends and limitations of llama-server as default interface.

Benchmark of vision LLMs vs. OCR pipelines on 30 long, image-heavy PDFs from MMLongBench-Doc shows LlamaCloud + Azure premium achieving 59.6%–58.5% accuracy; agentic RAG and native PDF vision approaches compared on cost and accuracy.

llama.cpp server adds native tool support (shell execution, file ops) via experimental --tools flag.

Wanted to share a workflow I tested on a real flight, in case anyone else is trying to set up offline Claude Code. The core idea: using ollama to pull the needed model of what you need, and then use it to run claude code The setup, in order: 1. Pull a model on home wifi the night before. \`ollama pull &lt;model&gt;\` — \~9 GB for a 14B, \~17 GB for a 26B. Don't try this at the gate. 2. In Claude Code, point at Ollama. The cleanest path I found is wrapping it in two aliases: alias claude-local='ollama launch claude --model gemma4:26b' alias claude-cloud='claude' 3. Verify on the ground with...

User reports Apex quantization for Gemma 4 26B achieves 38 tokens/sec at 90K context on RX 9060 XT with llama.cpp.

User configures dual AMD RDNA GPUs (48GB VRAM) with llama-cpp via Vulkan for local inference.

BeeLlama v0.2.0 achieves 4-5x token throughput gains on RTX 3090 via DFlash optimizations for Qwen 27B and Gemma 31B models.

Community fork of llama.cpp optimizes MoE inference on 12GB VRAM by loading only active experts rather than full layers.

cHunter789 releases Qwen-27B IQ4_KS quantization (14.1GB) optimized for 16GB NVIDIA GPUs via ik_llama.cpp.

SupraLabs released Supra-50M, a 50M-parameter Llama-style language model trained on 20B educational tokens with competitive benchmark performance.

llama.cpp b9274 fixes VRAM leak in speculative decoding by properly freeing draft context and decoder resources on server sleep.

Qwen 3.7 open-weights model released; community discussion on LocalLLaMA highlights adoption momentum.

llama.cpp PR #22929 fixes prompt processing performance issue affecting OpenCode and Pi model inference.

User reports 110 tok/s inference speed on Qwen 35B with 12GB VRAM using ik_llama.cpp, outperforming standard llama.cpp after MTP merge.

User shares llama.cpp configuration for running Qwen 3.6 27B locally with ROCm acceleration and optimizations.

llama.cpp PR #23287 optimizes MTP (multi-token prediction) draft sampling by moving logic to backend, improving inference performance.

Early-stage Gemma 4 MTP compilation work-in-progress shared on LocalLLaMA.

LM Studio 0.4.14 adds MTP Speculative Decoding support via llama.cpp 2.15.0 for faster inference.

llama.cpp PR #23269 introduces MTP (Multi-Token Prediction) improvements for faster local LLM inference.

llama.cpp MTP speculative decoding merged; Qwen 27B/35B inference speedups 1.24–2.44× on consumer GPUs/APUs with quantization.

Just wanted to share that I'm pretty happy about Qwen 35b a3b agentic coding performance. I'm running the model in q80 quant, kv cache both q8\_0 as well, with 262144 in 4090 + 5060 ti, via llama.cpp backend with claude code pointing to localhost. For demo/data analytics purposes, it works pretty well. I haven't used it for large codebases, but it definitely is better than gemma4 26b in my use case. One thing that surprises me is that it seems to get better outcome in agentic coding, than chat. When using it with just chat UI, i found the code qwen35b provide a bit too clunky. I wonder o...

Llama.cpp performance improved 1.5–1.8x in recent updates; MTP and prompt processing issues partially resolved.

MTP KV cache quantization in llama.cpp Qwen models reduces VRAM overhead without apparent inference degradation.

OpenBMB releases BitCPM4-CANN family (1B–8B params) with BitNet quantization; awaiting llama.cpp support.

Qwen 3.6 27B performance benchmarks across llama.cpp backends on RTX 3090: ik_llama.cpp achieved 1261 tok/s prefill, 72.9 tok/s decode with 156k context.

llama.cpp PR #23198 optimizes prompt decode by avoiding logits copying in MTP, improving inference speed.

llama.cpp fork adds quantized KV cache support for tensor parallelism across dual GPUs, addressing long-standing inference bottleneck.

Llama.cpp multi-token prediction on Qwen 3.6 27B shows 42% prefill slowdown but 85% token generation speedup on RTX 3090.

Benchmarking llama.cpp MTP (multi-token prediction) on Qwen 3.6 with RTX 5090, comparing inference speed with draft-mtp flag toggled.

User seeks faster inference alternatives to Ollama/LM Studio for local model serving (Gemma, Qwen, OpenBioLLM) on 64GB RAM.

llama.cpp release b9180 ships MTP support, enabling improved inference optimization for local LLM deployment.

Community discussion of Qwen 27B quantization and inference optimization on single RTX 3090 GPU.

Qwen3.6 27B-MTP shows 11.5% faster wall-clock time vs base on single-turn; 35B-MTP regresses 11.2%, with generation speedups offset by prompt processing slowdowns.

MTP support merged into llama.cpp master branch (PR #22673); no context on feature impact.

MTP PR merged into llama.cpp; technical details absent.

Looks like it finally happens... MTP getting approved for llama.cpp. Time to prepare for the update.

Engineer deployed Gemma 4 E4B on Jetson Orin NX with 30+ sensors, 200ms TTFT, offline multimodal robotics stack using llama.cpp and Piper TTS.

RTX 5060 Ti local LLM configuration guide covering vLLM and llama.cpp serving of Qwen 27B/35B models with quantization and long-context presets.

Llama-Studio: open-source WebUI for managing multiple local llama-server instances and configurations.

Hugging Face releases ml-intern, an agent framework for local LLM research automation supporting llama.cpp/ollama with Qwen and Claude models.

Multi-token prediction + TurboQuant quantization achieves 40% throughput gain (21→34 tokens/s) on Qwen 27B/35B via LLaMA.cpp on M-series Mac.

Qwen 3.6 35B and Gemma 4 26B MoE models achieve 20–24.5 tok/s on GTX 1080 with 128k context via llama.cpp quantization.

Community Docker images for llama.cpp with MTP support to simplify local model inference setup across CUDA versions.

Reddit discussion comparing vLLM vs llama.cpp for single-user local inference on AMD GPUs.

r/ClaudeAI • also crosspost to r/LocalLLaMA and r/artificial I lost $187 to this and want to save others the same headache. **What happened** I run Claude Code headlessly via Windows Task Scheduler. My project repo has a `.env` file with `ANTHROPIC_API_KEY` set — legitimately, for a separate Express server doing AI-based transaction classification. Nothing to do with Claude Code itself. Claude Code reads environment variables from the `.env` in its working directory on launch. When it finds `ANTHROPIC_API_KEY` there, it silently uses that key for billing instead of your OAuth ...

Luce ships DFlash+PFlash optimizations for AMD Ryzen AI MAX+ 395, achieving 2.23x decode speedup on Qwen 3.6-27B vs llama.cpp HIP.

llama.cpp adds llama-eval benchmarking tool supporting AIME, GSM8K, GPQA for local quantized model evaluation.

RTX 4090 power optimization for llama.cpp: reduce consumption 40% via power limits without performance loss.

Tuning llama.cpp ubatch and n-cpu-moe parameters improves gpt-oss-120b prompt processing from 240 to higher tok/s on RTX 3090.

Empirical study across 288 model calls identifying JSON output failures in Llama 3, Mistral, Command R, DeepSeek, Qwen; failure modes consistent across open and closed models but vary by rate.

JSON parsing bug in llama-server's chat-template-kwargs for Qwen3.6 preserve_thinking parameter requires whitespace-free formatting.

ExLlamaV3 adds Gemma 4 support, improved caching, and DFlash optimization for faster LLM inference on consumer hardware.

User successfully quantized and ran DeepSeek V4 Pro locally on AMD EPYC + RTX PRO hardware using modified llama.cpp with Q4_K_M compression.

User shares llama-server configuration for running Minimax 2.7 at 100k context on Strix Halo hardware with detailed tuning notes.

Reddit user seeks advice on LLaMA inference harnesses; discusses fragmentation and compatibility issues with local LLM tooling.

BeeLlama.cpp fork adds DFlash, TurboQuant, and vision support; runs Qwen 3.6 27B Q5 on RTX 3090 with 200k context at 135 tps.

User reports 1.5–2x speedup running Qwen 27B with MTP optimization on dual AMD MI50 GPUs via llama.cpp.

User achieves 80 tok/sec with 128K context on RTX 4070 Super using Qwen3.6 35B quantization and llama.cpp MTP implementation.

Reddit user asks about llama.cpp timeline for Vulkan/HIP MTP support on Strix Halo Windows 11.

Linear probing and activation patching reveal latent planning representations in Qwen3, Gemma-3, Llama-3; future constraints encoded at layer boundaries.

Multi-Token Prediction optimization for LLaMA.cpp achieves 40% speedup on Gemma 4 quantized models via parallel token drafting.

Chrome allegedly downloads 4GB LLM checkpoint without user consent, raising privacy and transparency concerns for browser-embedded AI.

Xiaomi releases Mimo v2.5, a 310B sparse MoE multimodal model with 1M token context supporting text, image, video, and audio.

Qwen3.5/3.6 inference optimization guide: NextN MTP speculative decoding achieves 2.9× speedup on RTX 3090 Ti via llama.cpp with zero quality loss.

User reports successful local inference with Qwen3.6-35B on AMD R9700 GPU, generating functional code and tests via llama-cpp.

Qwen3.6-27B with Multi-Token Prediction achieves 2.5x throughput via Unsloth quantization and llama.cpp integration.

Qwen 3.6 27B achieves 2.5x inference speedup via MTP speculative decoding in llama.cpp; 262k context on 48GB with fixed chat templates.

Qwen 27B achieves 54 t/s on V100 GPU with MTP optimization in llama.cpp, nearly 2x baseline speed for code review and tool use tasks.

Cyera reports critical unauthenticated memory leak vulnerability in Ollama enabling unauthorized data access.

User reports successful MTP speculative decoding on AMD Strix Halo (AI Max 395) with llama.cpp achieving 60-80 tok/s on Qwen 3.6B GGUF.