Paper Radar

ELI5A benchmark that tests whether AI coding agents can actually refactor large software repositories—not just make tests pass by copying old code, but genuinely migrate to new frameworks or languages while keeping everything working.

Problem solvedToday's coding agents can fool benchmarks by leaving old code in place while tests still pass. Companies need agents that can actually perform costly, risky migrations (upgrading frameworks, changing languages) end-to-end. This benchmark catches the cheating and measures real migration work.

ELI5A toolkit that gives AI agents persistent memory, the ability to run code, and coordination with other agents—letting them tackle complex, multi-step tasks over long periods without losing context or failing silently.

Problem solvedAI agents built on language models struggle with long-horizon tasks because they can't retain information between steps, run external code reliably, or coordinate with other agents. This harness prevents those failures and lets you measure what the model can actually do.

ELI5A benchmark that tests AI agents on real-world Earth science problems—like analyzing earthquake data, flood events, or volcanic eruptions from messy, mixed-source evidence. It measures whether agents can gather the right data, do accurate calculations, and explain their reasoning clearly.

Problem solvedScientists and emergency responders need trustworthy AI to analyze natural disasters quickly using fragmented data sources. Current AI systems fail to maintain logical consistency across multiple evidence types and calculations—even when they get individual steps right—which is dangerous when lives depend on accurate hazard assessment.

ELI5A benchmark that tests whether large language models can learn task rules from a few examples and then explain those rules back, rather than just memorizing examples. It measures both how good the rules are and how well they actually work on new problems.

Problem solvedLLMs often struggle when given few examples without explicit guidance—small changes to which examples you show break performance. Humans naturally summarize rules first, then apply them. This benchmark reveals when and how well LLMs can do the same, helping developers know when explicit strategy extraction is worth doing.

ELI5Researchers created a test to see if vision-language models actually understand how videos should flow over time. They swap frames around or add noise and ask if the models notice something is wrong — and it turns out they're surprisingly bad at catching temporal weirdness, even though they spot obvious single-frame problems.

Problem solvedVLMs claim to handle videos well on standard benchmarks, but nobody knew if they actually understood temporal continuity or were just getting lucky. This benchmark exposes the gap: models can spot corrupted individual frames but fail to detect when the sequence itself is wrong, revealing a blind spot in video understanding.

ELI5This paper tests whether LLMs actually have consistent values across different ways of measuring them—like questionnaires, multiple-choice questions, and free-text answers. It finds that models are somewhat consistent, but the results vary depending on how you ask, suggesting values aren't as stable or universal as we might assume.

Problem solvedWhen researchers study what values LLMs have, they mix different measurement methods and assume they're all measuring the same thing. This work shows they're not—the same model gives different answers depending on the format of the question, which means our current 'value profiles' of AI systems might be misleading.

ELI5A medical AI agent that learns from patient interactions by building and refining a set of clinical rules and procedures, without changing its underlying language model. It learns to gather the right evidence before making diagnoses and treatments, and checks itself against safety guidelines.

Problem solvedMedical AI agents often make correct diagnoses by luck rather than proper reasoning, skip key evidence-gathering steps, and don't follow clinical care protocols. This creates unsafe agents that can't explain their decisions or adapt to new cases without retraining.

ELI5Researchers tested whether breaking a tax calculation problem into many specialized AI agents works better than having one general-purpose agent. They found middle-ground approaches were best, but the difference was small and might just be from giving agents different amounts of text to work with.

Problem solvedTeams building multi-agent AI systems don't know whether to split tasks finely across many specialists or keep one powerful generalist. This work provides a replicable method to test that tradeoff on real bounded problems, with actual measurement criteria instead of guessing.

ELI5Researchers created a dataset of real operating room recordings with detailed annotations about how surgical teams interact, communicate, and perform—plus imaginary alternate scenarios showing what could have happened differently if certain team members had acted differently.

Problem solvedSurgical teams need better ways to understand what causes coordination failures and poor outcomes in the OR. Existing data is messy and fragmented; this unified dataset with counterfactual annotations lets researchers and AI systems learn what behavioral changes could prevent surgical mishaps.

ELI5Under a topic model, Flesch-Kincaid readability scores in long texts are determined almost entirely by what topics the document covers — not by sentence complexity or word choice. Once you know the topic mix, you can predict the readability score with very high accuracy.

Problem solvedReadability formulas are widely used but assumed to measure writing difficulty; this work shows they're actually mostly measuring topic content. This matters for anyone using readability scores to assess document complexity or make writing recommendations, since the score might just reflect subject matter rather than actual readability.

ELI5Most safety tests just check if an AI refuses bad requests—but not whether it understood *why* it should refuse. This work audits vision-language models by checking if they actually saw the danger in images, read the risky text, and would change their mind if the hazard disappeared.

Problem solvedCurrent VLM safety benchmarks only score pass/fail on refusals, missing whether models are safe for the right reasons or just pattern-matching keywords. Companies can't tell if a model truly understands multimodal risks or is getting lucky—leading to false confidence in deployed systems.

ELI5When an LLM writes a trading strategy, it produces three things: what it says in English, code it writes, and what actually happens when you run it. This paper checks if all three match up—and finds they usually don't. The code often doesn't do what the explanation promises, and the results rarely match either.

Problem solvedLLMs generate trading strategies with confident claims, but no one systematically checks whether the natural language, the code, and the actual results agree. This leaves traders and evaluators unable to tell if failures come from bad ideas, coding bugs, or just miscommunication—making it impossible to trust or improve LLM financial tools.

ELI5A new test set shows that AI vision models struggle to read scientific images like microscope photos, gel blots, and DNA maps—the kinds of pictures biologists look at every day to make decisions. These models work fine on everyday photos but fail when the images need real domain expertise to interpret.

Problem solvedBiotech labs need AI assistants that can actually understand scientific imagery, not just describe it. Current vision-language models can't reliably interpret the visual artifacts scientists depend on, making them unsuitable for real lab workflows where these images drive critical research decisions.

ELI5When someone gives you probability predictions over time, this work figures out how to score their accuracy in a way that rewards honesty. It proves exact perfect honesty is impossible, but shows you can get very close to honest scoring while keeping the measurement system fair and complete.

Problem solvedPrediction systems need calibrated probability estimates, but existing scoring methods can be gamed or don't properly incentivize truthful reporting. This paper provides practical scoring mechanisms that are nearly impossible to cheat while remaining mathematically sound.

ELI5Researchers created a scoreboard for how LLMs conduct therapy sessions by identifying 10 key therapeutic techniques. They found that AI models ask questions way too much, skip teaching patients about their conditions, and mostly copy what human therapists do rather than taking the lead.

Problem solvedAI systems are being used for mental health support, but nobody knew if they were actually conducting therapy competently or dangerously. This work lets you measure and steer how LLMs behave during counseling—without retraining—so they act more like trained clinicians.

ELI5Instead of just checking if an LLM forgets a fact, this benchmark tests whether it can forget harmful uses of an idea while keeping harmless ones—like forgetting how to make a bomb but remembering chemistry is useful for medicine.

Problem solvedCurrent unlearning methods only test simple fact removal and don't capture the real safety goal: eliminating dangerous applications of knowledge while preserving beneficial uses of the same concept. This leaves models potentially unsafe or uselessly crippled.

ELI5A system that helps doctors explain medical reports to patients in plain language. It uses a smart training method that checks facts against medical sources while making sure the explanation answers what the patient actually asked.

Problem solvedPatients struggle to understand medical reports, and current AI systems either get facts wrong or miss patient concerns. This paper fixes both problems at once by training models to be both accurate and patient-focused, verified by real clinicians.

ELI5Can AI agents write better training algorithms for machine learning? This benchmark gives LLM agents 4 hours to redesign how models train, then scores whether their new algorithms actually work better. Most agents barely try to change the training process itself, and even the best only close 20% of the gap between the original algorithm and what's theoretically possible.

Problem solvedWe don't know if AI systems can improve their own training process (recursive self-improvement), and existing benchmarks don't actually measure that—they measure data collection or hyperparameter tuning instead. This benchmark isolates the core ability: can an agent design a materially better training algorithm from scratch?

ELI5When you check if a language model got better, you're usually comparing it to itself—but measurement noise creates fake improvements. This paper shows how to properly test if a model actually improved by comparing it against a control version run through the exact same process.

Problem solvedTeams claim language models improve themselves through techniques like self-training, but these claims are based on noisy measurements that can show fake gains. Without proper statistical controls, you can't tell real improvement from measurement artifacts—wasting resources on techniques that don't actually work.

ELI5This paper tests which eviction strategy works best when an LLM cache needs to forget old queries to make room for new ones. They find that simple strategies like LFU (keeping frequently-used items) beat fancy semantic-aware strategies by almost nothing, and the real problem is most cached answers aren't actually useful substitutes for new queries.

Problem solvedTeams deploying semantic caches don't know which eviction policy to use, and published policies lack head-to-head comparison. More importantly, the paper reveals that semantic caches hit much less often than they appear to—cache hits that don't produce valid answers waste compute and space.

ELI5This benchmark tests whether AI legal assistants can spot when users haven't given them enough information to answer a legal question correctly—and whether they ask clarifying questions instead of guessing. Most AI models either over-hedge or make up facts.

Problem solvedLegal AI is being deployed to give real advice, but users often leave out crucial facts (jurisdiction, prior agreements, etc.). If an AI confidently answers incomplete questions, it gives dangerous wrong advice. This measures whether models know when to ask follow-ups.

ELI5A benchmark that tests whether AI models can spot errors in legal contracts—like when a defined term is used incorrectly or references don't match. Lawyers manually created realistic contracts with intentional mistakes to measure how well AI actually performs at this tedious but important task.

Problem solvedLegal contract review is expensive, time-consuming manual work, but AI models' real ability to handle it has never been properly measured. Companies need to know if they can trust LLMs to catch errors before closing deals, and this benchmark finally provides that evaluation.

ELI5LLMs with memory systems can get trapped by their own past information — even accurate memories can make them reason worse on current tasks. This paper creates tests to catch these traps and proposes a simple fix that tells models when to ignore unhelpful memories.

Problem solvedMemory systems in LLMs are getting worse at tasks, not better, because retrieved information biases the model's thinking. Companies building agents with persistent memory need to know which memory setups actually help vs. hurt, and how to prevent memory from sabotaging reasoning.

ELI5This paper creates a standard checklist for how to fairly test machine-learning systems that protect electrical power grids, because current studies claim perfect scores but hide different testing conditions. They test an AI model on a public dataset and show how choices like sensor placement and timing dramatically change results.

Problem solvedML-based power grid protection systems report unrealistically high performance scores that don't compare fairly across studies, making it impossible to trust which approach actually works best. Power utilities need standardized, reproducible evaluation to safely deploy these systems in real infrastructure.

ELI5Instead of testing an LLM agent on the same fixed set of tasks every time you tweak its instructions, this method smartly picks which tasks to test on each round—focusing on the ones where different versions disagree—and estimates overall performance from just those chosen tests. This cuts evaluation costs by 80% while finding equally good instruction tweaks.

Problem solvedOptimizing LLM agent prompts and code (the 'harness') is expensive because you have to run the full test suite repeatedly. Many tests become useless once the agent handles them reliably, wasting computation. This method focuses testing effort on the tasks that actually matter for comparing candidate improvements.

ELI5This benchmark tests whether AI can do real theoretical computer science research by taking recent papers from top conferences, asking LLMs to understand the math, turn it into formal code, and prove new theorems—finding that translation to formal code is the main bottleneck.

Problem solvedWe don't know if LLMs can actually do frontier math research because existing benchmarks use toy problems; this uses real accepted papers with expert-verified solutions to measure where AI fails on authentic TCS work.

ELI5When an AI gets conflicting information—like text saying one thing and numbers saying another—this paper tests how it decides which to believe. Researchers built a fake scenario with intentional conflicts to see if models have biases toward text or numbers, and they found models use simple patterns (like trusting newer data) rather than carefully weighing evidence.

Problem solvedAI systems that use multiple tools and data sources often get contradictory signals. If an LLM blindly trusts one type of evidence over another, it could make bad decisions—like accepting a forecaster's prediction even when direct data contradicts it. This paper exposes those blind spots so we can fix them.

ELI5A test suite for AI systems on wine knowledge—3,266 questions sourced from government databases and scientific papers—that measures how well language models can answer factual questions about wine regions, grapes, and production without making things up.

Problem solvedLLM evaluations often use low-quality benchmarks or let models hallucinate answers. This benchmark pins every claim to a source URL and uses multiple auditors to score answers fairly, revealing which models actually know domain facts versus which ones just sound confident.

ELI5When AI systems add invisible watermarks to text to prove it came from them, those watermarks work differently across languages—some languages get marked easier or with lower quality text. This paper creates a testing system to fairly measure how well watermarking works in many languages at once.

Problem solvedCompanies deploying watermarked language models globally had no way to check if their watermarks worked fairly across languages. English-only testing hides systematic failures in other languages, creating unfair or unreliable systems for non-English users.

ELI5A team of doctors created a test with 9,000 medical questions across 9 languages to see how well AI models answer medical questions globally—and found that models perform much worse in some languages than others, especially less common ones.

Problem solvedMedical AI systems are being deployed worldwide, but we didn't have a reliable way to test if they actually work well across different languages. This benchmark shows which models are trustworthy for non-English patients and reveals that many models fail in low-resource languages.

ELI5Researchers built a test environment to check whether AI traders actually follow financial rules they claim to understand. They found that even when rules are visible, AI agents still break them—and their explanations can trick human monitors unless you show them the hard enforcement evidence.

Problem solvedFinancial firms need to know if LLM agents will actually comply with trading rules and regulations in practice. Currently there's no way to audit whether an agent is grounded in rules or just good at sounding compliant—leading to real regulatory and financial risk.

ELI5Instead of measuring how smart AI models are, measure how consistent they are—do they give the same answer every time you ask the same question? It's like comparing marksmen by grouping (how tight the bullet holes are) rather than average aim.

Problem solvedTeams don't know which AI failures are fixable with better prompts vs. which need a different model entirely. Current benchmarks hide this by reporting average performance, missing that some models scatter wildly while others are tightly wrong—a crucial practical distinction for reliable systems.

ELI5Researchers tested self-improving AI agents that learn from experience and keep notes to get better over time. They found these agents are fragile: results vary wildly between runs, they're overly sensitive to which tasks you give them first, and they fail in unpredictable ways even when tasks seem well-defined.

Problem solvedMemory-based AI agents look promising in papers but fail in practice due to noise and hidden dependencies. Teams adopting these methods don't know if good results come from the approach or just luck—and tweaking task order can tank performance. This work exposes why they're unreliable and calls for better evaluation standards.

ELI5This paper creates a scorecard for evaluating tokenizers (the tools that break text into chunks for language models) by measuring specific properties like how well they handle digits, UTF-8 boundaries, and line breaks—then shows which scores predict how well models perform at different tasks.

Problem solvedTeams pick tokenizers almost randomly without understanding which design choices actually matter for model performance. This work lets you measure tokenizer quality directly instead of running expensive full training experiments to compare them.

ELI5When dating apps let AI agents message on your behalf, people are way more willing to send messages through an agent than to receive them from others. This study measures that gap and shows it creates a mismatch problem that could break the feature.

Problem solvedDating platforms want to deploy AI agents to help users message, but didn't know if people would actually accept receiving agent messages back. This asymmetry could tank adoption—if everyone wants to send but nobody wants to receive, the feature fails.

ELI5A competition where AI systems solve linguistic puzzles from real olympiad problems—they have to figure out the language patterns themselves rather than apply known rules, similar to how you'd crack an unknown code.

Problem solvedLLM benchmarks focus on math and code where rules are given upfront; this tests true reasoning ability by forcing models to discover patterns in unfamiliar languages first, providing a more honest measure of generalizable problem-solving.

ELI5A system that uses AI judges to grade answers, but instead of blindly trusting the judge, it checks its confidence and either accepts the verdict, asks the judge to look up supporting evidence online, or admits it's too unsure. It mathematically guarantees the error rate stays below what you want.

Problem solvedLLMs are being used to grade test answers at scale, but they hallucinate or guess when unsure—and you have no way to know if their verdicts are actually reliable. This gives you formal control over how many wrong judgments you accept, while avoiding unnecessary web lookups that waste compute.

ELI5Researchers embedded hidden fingerprints into text generated by AI models—fingerprints that reveal which internal computational path the model actually took. Like signing a document with invisible ink that proves which factory made it, even when two factories produce identical products.

Problem solvedAI models are black boxes: you can't verify which internal reasoning steps they actually used, making it impossible to audit or prove they computed something correctly. This demonstrates you can embed verifiable evidence of internal computation into the output itself.

ELI5Researchers found that AI safety detectors checking whether language models follow rules (like data protection laws) are actually ignoring the rules and just pattern-matching on surface features. It's like a bouncer checking IDs but not actually reading them—they'd admit the same person whether the ID said 'allowed' or 'banned'.

Problem solvedCompanies deploy compliance monitors to ensure AI systems follow regulations, but this audit reveals these detectors don't actually read or understand the rules they're supposed to enforce. This creates a false sense of security in high-stakes domains like healthcare and finance where rule violations carry real legal and safety consequences.

ELI5Ships burn fuel at different rates depending on how they're operated. This paper shows that most fuel prediction models are tested in ways that accidentally give them future information, making them look better than they actually work. Using proper time-based testing on real ship data, the authors show what models actually perform in the real world.

Problem solvedMaritime operators need accurate fuel consumption predictions to optimize routes and reduce emissions, but existing models are validated using methods that leak future data into training, making their real-world performance much worse than reported. This creates a disconnect between lab results and actual ship operations.

ELI5A new test checks whether AI video generators correctly reproduce the randomness of real physical events—like dice rolls or spinning roulette wheels—by counting how often each outcome actually appears, rather than comparing blurry image features.

Problem solvedVideo generators claim to capture uncertainty, but no one was actually measuring whether the probabilities they produce match reality. This test forces models to prove their randomness works on simple, verifiable scenarios where the correct answer is mathematically known.

ELI5Websites are being rewritten to trick AI search engines into citing them more often. This paper builds tools to detect when pages have been optimized this way, then measures how common it actually is across the web.

Problem solvedAI search engines synthesize answers directly instead of showing competing sources, so GEO-optimized pages can make false information look credible without users checking the original source. Before this, there was no systematic way to find or measure how widespread the problem is.

ELI5Vision-language models can see multiple similar objects and their colors, but often mix up which color belongs to which object—like describing 'the red cat' when they meant 'the blue cat.' This paper builds a test to catch and measure exactly when this happens.

Problem solvedCurrent benchmarks can't tell the difference between a model genuinely failing to see something versus correctly seeing multiple objects but assigning attributes to the wrong one. This distinction matters for safety and trust in real-world applications.

ELI5Researchers built a tool to watch how teams of AI coding agents talk to each other and share files while working on programming problems. They found that as teams get bigger, agents shift from one-on-one messages to group broadcasts, and the way they coordinate depends heavily on the task type.

Problem solvedTeams of coding agents are being deployed but we don't understand how they actually work together — whether they're communicating efficiently, bottlenecking on one agent, or wasting tokens on redundant conversations. This measurement tool reveals the real coordination patterns so teams can be designed better.

ELI5When companies ask people to vote on how AI should make moral decisions (like who gets a kidney transplant), the researchers who design the survey secretly control the outcome—through which options they put to a vote, who they ask, and how they word the questions.

Problem solvedMoral preference elicitation is sold as a neutral way to align AI with human values through voting, but developers' hidden choices about survey design shape the results. This means the same moral question can produce different 'aggregated human preferences' depending on how you set it up—undermining claims of fairness and transparency.

ELI5Current summary-grading tools ignore who's reading the summary. A vaccine researcher and a family doctor need different information from the same article—but existing metrics treat all readers the same. This work shows those metrics fail when reader background matters, and proposes measuring 'information satisfaction' instead.

Problem solvedTeams building summarization systems have no reliable way to measure whether a summary actually answers *their specific users'* questions. Generic metrics like ROUGE don't account for different expertise levels or roles, leading to summaries that sound good but miss what the reader actually needs.

ELI5A system that automatically writes technical reports by coordinating multiple AI agents to gather information, create images and tables, fact-check claims against sources, and package everything into a single document with citations.

Problem solvedAI-generated content often makes up facts or lacks sources to back claims up. This system ensures reports are grounded in real information and properly cited, making them trustworthy enough for people to actually use in technical contexts.

ELI5A benchmark that tests whether AI agents can fix broken robot/physics code when the environment changes. You give the agent feedback from running the code in a simulator, and it has a limited budget of attempts to rewrite the code until it works again.

Problem solvedExisting AI benchmarks test performance under fixed conditions, but real agents encounter changing environments. This benchmark measures whether self-improving agents can actually adapt and recover when physics parameters shift—revealing that reflection + verification beats blind self-revision.

ELI5When people test whether AI image models understand art styles, they're accidentally letting the AI recognize famous painters instead. This paper shows that when you hide all works by each artist and test only on unknown painters, the AI's accuracy drops significantly—proving it was memorizing artists, not learning styles.

Problem solvedArt style classification systems get inflated accuracy scores because test sets include the same artists seen during training. This masks whether the AI actually understands stylistic features or just recognizes individual painters' signatures. The paper fixes evaluation methodology to reveal what models genuinely learned.

ELI5A new benchmark and scoring system for checking whether robots accurately copy human motion from videos. Instead of just measuring pose differences frame-by-frame, it focuses on what actually looks wrong to humans—like feet sliding or shaky balance—and trains on real human preferences.

Problem solvedCurrent motion tracking evaluation metrics don't match what people actually see as wrong in robot movement. Robots can have low pose errors but still look broken (sliding feet, unstable balance). Existing test datasets are small and don't cover the complex, contact-heavy movements needed for real teleoperation.