Teaching LLMs to Give Better Answers

Ask an LLM a question and you’re likely to receive a response that sounds like it came from a Ph.D; whether the information is accurate—or even based in reality—is another issue. Somewhere between training, inference, and the real world of spitting out words and code lies a persistent problem: these models are unable at times to recognize basic errors.

As AI seeps into everyday life—and agentic systems emerge to handle bookings, transactions, code deployment, medical decisions, and business workflows—accuracy and dependability aren’t negotiable. Even a minor hallucination can trigger a botched financial transaction, a misrouted shipment, or a life-threatening medical error.

As a result, researchers are exploring ways to embed better logic into AI. The goal isn’t so much to make LLMs smarter; it’s to improve their decision-making skills. “There’s a need for better mechanisms in places where trust is required,” said Byron Cook, Distinguished Scientist and vice president at Amazon.

It’s no small challenge. Formal methods deliver mathematical guarantees, but they often fall short in the fuzzy world of LLMs. Neuro-symbolic approaches sidestep this problem by connecting LLMs to external verifiers but ensuring that the right one exists at the right moment is difficult. In addition, emerging world models, neural networks that understand the dynamics of the real world, deliver grounding—but these systems remain in research labs.

“Today’s AI is quite good with uncertainty and messy data,” said Clark Barrett, professor of computer science at Stanford University. “But it is not good at providing guarantees.” The goal is to reach a level of automated reasoning that delivers “some kind of guarantee” that the decision is correct.

Experts have a name to describe today’s AI shortcomings: jagged intelligence. LLMs can solve incredibly complex problems yet fail at a simple task. This occurs for a couple of reasons. First, LLMs rely on word patterns rather than any real understanding of the world. Second, not all logic is black and white. Answers often depend on context and events.

The problem is rooted in the way LLMs work. They excel at math and code generation but struggle with basic reasoning skills. Intermediate tokens that lead to a response tend to “mumble a lot before they deliver a final answer,” said Subbarao Kambhampati, professor at the School of Computing and AI at Arizona State University. Essentially, these intermediate tokens serve as placeholders that allowed the model to get to an answer. “Models don’t reach a conclusion in any logical way. The tokens that they emit don’t necessarily have any end user semantics,” he added.

Agentic AI dials up the stakes. Even a small error or hallucination can escalate into disastrous consequences across a 20-step workflow. In fact, Kambhampati described LLMs as “pattern completers rather than planners.” In many cases, he noted, they produce plausible-looking plans and scenarios that don’t work in the real world. “There is a distinction between generating tokens that look like reasoning and actual reasoning,” he explained.

Adding to the challenge: the knowledge gap that exists between the physical world and the digital space. Because LLMs learn from text, their intrinsic understanding of things extends no further than word patterns. While a tool like Retrieval Augmented Generation can serve as a fact-checker for an LLM by connecting it to documents, facts, and specifications, it still lacks basic grounding in real-world physics.

As researchers explore ways for LLMs and AI agents to reason better, they’re venturing into new digital territory. For instance, in the formal space, Amazon Web Services has built automated reasoning checks into Amazon Bedrock Guardrails. The feature translates model outputs into formal logic statements and verifies them against a defined policy, such as rules from an HR handbook or a set of financial regulations. Amazon claims the service achieves up to 99% accuracy. The company runs more than one billion automated-reasoning checks daily across its services.

The result, Cook said, is ongoing communication between the language model and the formal logic. If the verifier finds that the output and rules don’t match, it sends the text or code back to the LLM along with a reason. The model then goes back to work. This cycle continues until the verifier confirms a match or it gives up and declines to give an answer. “Verification tools that required a lot of human intervention in the past are becoming far more powerful,” he explained.

At Stanford, Barrett is pursuing a different path. Rather than asking the LLM to check its output against a set of approved answers, the system examines the LLM-generated code and verifies that it meets a formal specification. It doesn’t evaluate output on a case-by-case basis; instead, it ensures that the code is correct for every possible input. Barrett helped create the CSLib project, backed by Google and Amazon, which relies on an established proof assistant, the Lean Theorem Prover, to keep the code on track. “The verifier will not let it pass unless it gets it right,” he said.

Other approaches are surfacing. In the neuro-symbolic space, researchers are studying ways to let the LLM do what it specializes in—producing plausible output—while a separate system examines results for correctness. For example, Amazon Nova connects automated reasoning with machine learning to improve chain-of-thought reasoning. DeepMind and DeepSeek are pursuing similar neuro-symbolic approaches. Meanwhile, Kambhampati has developed an LLM-Modulo framework that allows the LLM to draft output while external verifiers—including a symbolic planner and syntax checkers—determine whether the steps work in sequence. Output that doesn’t make the cut goes back for revision.

Researchers are pushing neuro-symbolic research even further by folding probability models into the architecture. This helps the system manage uncertainty while boosting commonsense reasoning. At Massachusetts Institute of Technology (MIT), for example, Joshua Tenenbaum, a professor of cognitive science, is developing a framework that meshes neural networks, symbolic logic, and probability into a single architecture. It has displayed encouraging results in domains like intuitive physics and causal reasoning—places where LLMs frequently stumble.

Emerging world models could further upend the way LLMs process logic. Plugging in video, sensor data, or physics simulations allows an LLM and other AI to not only learn from text, but also to gain an understanding of how things work. Several groups are developing these frameworks, including World Labs’ Marble, DeepMind’s Genie, and Yann LeCun’s V-JEPA. Adding this grounding would allow AI to predict, plan, or reason about what comes next. “If you limit yourself to the information that humans produce, it’s actually a very limited and biased set of data,” Cook explained.

While both formal and neurosymbolic approaches offer promise, researchers believe that no single approach will dominate. Instead, models will incorporate a combination of these tools to get to the right outcome. A chatbot fielding questions about history or sports needs different reasoning and guardrails than an agent executing financial trades or making medical decisions. “As a community, we’re still trying to figure out where the dials need to be set,” Cook said.

Because LLMs are better at generating plausible output than they are at verifying it, external controls and verification methods—formal checkers, symbolic reasoning systems, and world models—will almost certainly prevail, Kambhampati said. “There will always be a role for some type of verifier. LLMs can’t do it alone,” he said.

To be sure, the path to better reasoning is emerging. Concluded Barrett: “Without greater rigor, we risk being buried by mountains of buggy code and inaccurate information. Using these methods, we have a tremendous opportunity to bring rigorous reasoning to software and systems through AI.”

Samuel Greengard is an author and journalist based in West Linn, OR, USA.