Corin Wagen
Agentic AI systems are now good at scientific reasoning and analysis tasks, which in turn is leading to their use in increasingly complex scientific capacities: literature review, molecular design, reaction optimization, and data analysis. Recent work from Anthropic explored how Claude could be used to assist in the interpretation of NMR spectra, one of the most ubiquitous analytical methods in the organic chemistry lab. In their words, “a general-purpose model can be competitive with NMR software and even make 1D inverse elucidation tractable.”
The Anthropic work distinguishes between “forward prediction,” which predicts the NMR spectrum from a given molecule, and the harder “inverse prediction” problem, which asks the model to figure out which molecule generated the given spectrum. While the forward-prediction problem is solved routinely in academia and industry by a variety of in silico tools, including quantum chemical methods and faster surrogate models like ChemDraw’s ChemNMR, MNova’s NMRPredict, and CASCADE, the inverse-prediction problem is typically performed by human scientists, who use their chemical intuition and accumulated expertise to enumerate potential structures and test each one. Anthropic found that Opus 4.7 could directly solve inverse-prediction problems for simple molecules, while more complex molecules required additional structural priors.
We recently released an NMR workflow which solves the forward-prediction problem using an equivariant graph transformer (developed with collaborators at Merck, MIT, Harvard, &c—see the preprint on ChemRxiv). We’re generally interested in understanding how and when to integrate simulation tools with AI agents, and we hypothesized that giving an AI agent access to our forward-prediction NMR tool might improve agent performance on inverse-prediction problems. There are intuitive reasons why this might be true: use of an NMR simulation tool acts as an external check on agentic reasoning and allows for iterative experimentation, ideally allowing the model reason coherently for longer. On the other hand, it’s possible that imperfect simulation tools are simply distracting to AI agents, consuming context & reasoning tokens without contributing any insight.
In this post, we report the results of our initial benchmarking in this area. Our results support our hypothesis: external simulation tools do noticeably improve agent performance, allowing models to think for longer and solve problems that they can’t solve otherwise.
We wanted to focus our benchmark on difficult structural elucidations that would require extended reasoning to solve correctly. To identify molecules with potentially challenging spectral assignments, we looked for cases where a molecule’s structure had been misreported and later corrected with the aid of 13C NMR computations. We compiled 8 such cases for this benchmark from Novitsky and Kutateladze’s 2022 work. While this work was released before the model’s training cutoff, the agents’ performance makes us relatively unconcerned about data leakage, with some exceptions (Appendix 3).
NMR shieldings were predicted using our recently reported MagNET model, which was trained on millions of DFT-computed isotropic chemical shifts. Like many in silico models of experimental data, MagNET is predictive but imperfect: on average, MagNET-based simulation of the true structures gave predicted shifts with an MAE of 1.20 ppm and an RMSD of 1.60 ppm relative to reported experimental values.
Each agent was given the chemical formula, the experimental 13C chemical shifts, and the ability to predict 13C NMR shifts up to 10 times via a simple MCP integration with Rowan. Web search and other tools were disabled for this evaluation. We kept the prompt simple to give the agents room for creativity—here’s the prompt for the first question:
Prompt:
Solve this closed-book 13C NMR structure-elucidation problem.
Molecular formula: C10H12OS
Solvent: CDCl3
Experimental 13C chemical shifts, δ ppm, unordered and with duplicates
preserved: [158.5,136.1,128.7,128.2,128.2,125.5,125.5,85.0,72.0,15.1]
You must obtain at least one completed simulate_13c(smiles) result for a valid,
formula-matching candidate. You may invoke simulate_13c at most 10 times total.
Every invocation consumes one call, including invalid or formula-mismatched
candidates.
Use the simulation results to compare and refine plausible structures before
answering. Lower MAE indicates better agreement with the experimental 13C NMR
data and makes a candidate more likely to be correct; high MAE is evidence
against a candidate.
Return exactly:
{"final_smiles": "..."}
It’s possible to follow the agents’ problem-solving process by watching which simulations they chose to run. Here’s a sample successful set of simulations for the above prompt: the agent correctly identifies that there’s an alkene, a benzene ring, an ether, and a thioether right away, but has to try a few different ways of combining the functional groups before figuring out the correct answer.
We ran 3 replicates per molecule across 8 molecules for a total of 24 runs per model. We also ran all of the models without any external tool calls, removing that section of the prompt, to understand the effect of giving the model the tool. After a few early investigations with different models (Appendix 2), we opted to use GPT-5.6 Sol for the bulk of this study.
A run was counted as successful only if the agent supplied the exact SMILES of the product (or its enantiomer). To track partial successes, we also (1) recorded the percentage of answers with correct connectivity but incorrect stereochemistry and (2) tracked the mean Tanimoto similarity (Morgan fingerprint, radius 2) of submissions to the true answer.
We found that use of the external NMR tool led to a dramatic improvement in agent performance. While GPT-5.6 Sol was only able to solve 25% of the problems on its own, the success rate improved to 63% with the tool integration. Use of increased reasoning seemed to help the tool-use runs, with the best results found at “xhigh” effort (see full data in Appendix 1):
(The absolute number of trials is pretty small here, so we’ve added the actual number in parentheses beneath the percentage score. This post is statistically underpowered and as such results should be taken as preliminary.)
Since scoring exact structural success penalizes near-miss guesses (e.g. incorrect stereochemistry) the same as totally incorrect answers, we also compared the average Tanimoto similarity of all guesses. Once again, tool use generally led to improved performance:
