§ Benchmarks

Reddit discussion argues prefill latency is underemphasized vs. token generation speed in local LLM benchmarking and optimization focus.

TabEmbed introduces first generalist embedding model for tabular data and TabBench, a comprehensive benchmark for tabular understanding evaluation.

KernelBench-X evaluates LLM-generated Triton GPU kernels across 176 tasks; finds task structure explains 3x more correctness variance than method design.

Controlled benchmark study diagnosing when causal vs. correlational methods fail for gene regulatory network inference from single-cell RNA-seq.

EnterpriseRAG-Bench: 500k-document synthetic dataset benchmarking RAG systems on realistic internal company data (Slack, email, tickets, PRs) vs. public corpora.

Empirical quantization degradation analysis for Qwen 3.6 27B across 8 compression levels via chess state-tracking task.

SubQ claims 12M context window in marketing but production model capped at 1M; benchmark results show significant performance drop vs. research variant and competitors.

Reddit discussion on 2024 AI benchmarks without substantive content or specific findings provided.

Developer benchmarked local Qwen 3.6 27B vs cloud models on 150 real coding tasks, finding local matched cloud 97% on 35% of workload, suggesting cost arbitrage opportunity.

Benchmark comparison shows Gemma 4 31B trades inference speed for token efficiency vs Qwen 3.6/5 27B; Qwen optimizes for metrics, Gemma for throughput.

SaFE-Scale framework reveals clinical LLM safety and accuracy follow divergent scaling laws; introduces RadSaFE-2 benchmark.

BRIGHT-Retriever: benchmark and training approach for reasoning-intensive retrieval in agentic search, beyond topical matching.

Fairness audit of five LLMs (Gemini, GPT-4, DeepSeek, Mistral, Nemotron) on emergency triage reveals gender bias persistence in clinical decision support.

Transformer-based AI-text detector using HC3 PLUS and M4 benchmarks demonstrates domain-robust detection with fixed thresholds across generators.

MOSAIC-Bench evaluates coding agents' vulnerability to multi-stage attack chains that decompose malicious goals into innocuous sequential tasks, exposing alignment gaps in deployed systems.

iWorld-Bench is a 330k-clip dataset and benchmark for training interactive world models on perception, reasoning, and physical interaction capabilities.

Open-World Sound Event Detection paradigm extends closed-world audio classifiers to detect unknown events and incrementally learn from them.

CC-OCR V2 benchmark for real-world enterprise document OCR with LMMs; addresses gap between lab tasks and practical heterogeneous acquisition conditions.

Proposes improved empirical fixation density estimation methods beyond fixed-bandwidth Gaussian KDE for saliency benchmarking and per-image model evaluation.

ProgramBench: 200-task evaluation showing agents struggle to rebuild large binaries from scratch without cheating vulnerabilities.

MCJudgeBench: benchmark for constraint-level evaluation of LLM judges in multi-constraint instruction following with per-constraint gold labels.

Domain incremental learning benchmark for ICU time-series model transfer across hospitals with domain shift and patient data heterogeneity.

Reproducibility study of neural retrievers on set-compositional queries; introduces LIMIT+ benchmark for constraint-satisfaction information retrieval.

Manokhin Probability Matrix: diagnostic framework separating classifier calibration and discriminatory power via 2x2 archetype taxonomy.

User benchmarks Claude Opus 4.7 vs Kimi K2.6 on complex game mod coding task with TypeScript/Composio integration.

PhD student reports 4% accuracy gap when reproducing computer vision paper baseline; raises reproducibility concerns common in published ML research.

DeepSeek V4 Pro matches GPT-5.2 on FoodTruck Bench agentic task; first Chinese frontier model, 17× cheaper, within 10 weeks of GPT-5.2 release.

Anonymous text-to-image model 'Peanut' ranks #8 on Artificial Analysis arena; open-weights release promised to compete with FLUX and Qwen models.

IBM released Granite 4.1 (3B/8B/30B, Apache 2.0); Unsloth published 21 quantized GGUF variants; Willison benchmarked quality across model sizes on SVG generation.

Open-source research agent built on Claude Code outperforms OpenAI and NVIDIA systems in deep research benchmarking.

PLACE: closed-form persistent-homology pipeline for point cloud and graph classification with margin-based guarantees and per-prediction certificates.

VideoNet benchmark with 1,000 domain-specific actions revives action recognition evaluation for vision-language models.

Benchmark reveals audio-language models fail to leverage clinical context for dysarthric speech recognition despite multimodal capacity.

ReClaim: generative transformer trained on 43.8B medical events from MarketScan claims data for healthcare foundation model development.

PubMed-Ophtha: 102K ophthalmology image-caption pairs extracted from open-access PDFs for training vision-language models.

GRIDS framework uses Local Intrinsic Dimensionality to detect how perturbations deform representations in WavLM and wav2vec 2.0 models and track ASR degradation.

ACII-DaiKon 2026 benchmark for dyadic conversation affect modeling: interpersonal influence, timing coordination, and rapport development challenges.

DILER Benchmark: drug-induced liver injury dataset with mechanistic hypotheses; reframes DILI prediction as explainable hypothesis generation.

IBM Research releases MAMMAL, multimodal model integrating proteins/molecules/genes, achieves SOTA on 9/11 biological benchmarks including drug-target interaction and antibody-antigen binding.

RMGAP: benchmark for evaluating reward model generalization across diverse user preferences with 1,097 instances, addressing RLHF alignment robustness.

MolJSON: novel molecular representation format systematically compared against SMILES/IUPAC for LLM chemistry reasoning tasks.

TMD-Bench introduces evaluation metrics for text-driven music-dance co-generation, addressing rhythm-choreography coupling beyond generic audiovisual consistency.

Calibrated Size Ratio (CSR) replaces Expected Calibration Error for confidence calibration, addressing ECE's inability to detect overconfidence risk.

Defines 'Compliance Gap': AI systems verbally accept constraints but violate them in execution; audits instruction-following.

Iterated negotiation benchmark tests LLM agents' ability to repair grounding failures in dynamic multi-turn interaction.

SignVerse-2M dataset adds 2M pose-annotated sign language clips across 25+ languages for improved recognition and generation.

Empirical comparison of Qwen3.6-27B and Coder-Next models across 40 test cases shows statistical parity with task-dependent tradeoffs.

MultiBreak benchmark evaluates LLM safety via scalable multi-turn jailbreaks using active learning to generate diverse adversarial prompts.

Large-scale evaluation of DiffDock, AutoDock-GPU, GNINA, and NMDN on LIT-PCBA library (15 targets, 578K pairs) for molecular docking.

27B/31B vision model comparison: Qwen 3.6 benchmarks higher than Gemma 4 but underperforms in real-world tasks; suggests benchmark gaming.