← All AI Engineer talks

AI Engineer Summit 2025

Your LLM Ran Out of Knowledge — Now What?

12:51

Read the talk

Your LLM Ran Out of Knowledge—Now What?

When professional judgment never makes it into training data, explicit domain rules can guide a model’s reasoning, from classifying a problem to generating scenarios for expert review.

Before you start: Familiarity with LLM prompts and basic application workflows is sufficient; no geopolitical modeling background is required.

Reasoning still needs domain knowledge

How can a model solve a problem in a domain whose knowledge was never available for training? Moving from chatbots to agents to reasoning models makes this question more pressing: greater reasoning capability does not automatically supply missing professional experience.

Minecraft library scene with empty shelves, a Risk Management sign, cobwebs, and the heading Research (LLM PoV). — Empty shelves labeled “Risk Management” in the “Research (LLM PoV)” sequence.

Coding, mathematics, law and physics have substantial bodies of structured material that can become training data. Other fields leave much less of their decision-making process on paper. Consider a corporate acquisition: a newspaper reports that company A wants to acquire company B, and a later headline reports the outcome. Those reports expose the starting point and the result, but not what happened in the negotiating room. MBA guidelines may describe how negotiations should work; they do not recover the actual discussions.

That missing middle contains practical judgment: knowledge held in people’s minds and never written down. Its absence can leave some professions with increasingly capable models while others struggle to apply them to everyday problems. Speaking from risk and security management, the presenter locates the bottleneck in practitioners’ documentation habits. If experience never becomes a record, model trainers have little material from which to teach it.

Suggest correction

Correction for “Reasoning still needs domain knowledge”

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:01 · section reference included

Give the model a practitioner’s rules of thumb

An intelligent graduate entering a profession faces a similar problem. They can reason, but they have not accumulated the experience needed to recognize every situation. Giving them guidelines and rules of thumb helps them apply their intelligence in a particular domain while they build that experience.

The proposed technique applies that pattern to an LLM: supply domain heuristics for the model to use during reasoning. Rather than waiting for a comprehensive training corpus, ask practitioners to express useful judgment as rules the model can apply to a new problem. The intended benefit is to compensate for gaps in practical wisdom; supplying rules does not itself establish equivalence to an experienced expert.

Suggest correction

Correction for “Give the model a practitioner’s rules of thumb”

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:16 · section reference included

Match the problem and standardize the scenario

The demonstration begins with a long intelligence-estimate prompt entered into a frontend. Before attempting the analysis, the application determines what kind of problem it has received and which rules apply. It also reformats the question into a standard representation. These are preparation steps: the simulator has not yet started reasoning about the scenario.

The backend uses a dedicated LLM prompt to match the user’s input against a list of available heuristics. The presenter describes this as similar to tool selection, but uses a more explicit matching prompt because tool use had not worked as well as hoped in this implementation. That is a local design choice, not evidence that prompt-based routing universally outperforms tool use.

The preparation flow has two distinct responsibilities:

Match the domain. Identify which available rule family fits the user’s problem.
Parse the scenario. Return a consistent structure containing the material the simulator needs.

The frontend lets the user edit the parsed scenario before proceeding. This gives the later reasoning stage a predictable input shape without making the parser’s interpretation irreversible.

Suggest correction

Correction for “Match the problem and standardize the scenario”

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:28 · section reference included

Assemble explicit, checkable domain rules

Once a match exists, the application retrieves the appropriate rules and behavioral guidance. The demonstration contains two rule families: corporate negotiations and geopolitical strategy. Their directives use a concrete must-do or must-not-do form rather than broad advice. One negotiation rule prioritizes agreements with the highest combined value. One geopolitical rule calls for at least three independent paths for critical resources. These are supplied heuristics, not validated universal requirements.

Code editor showing negotiation rule lists and the beginning of geopolitical strategy rules, with a small presenter inset. — Negotiation rules organized into “must_do” and “must_not_do” lists.

The purpose of that specificity is to make application of a rule easier to inspect. A collection of relatively simple directives can express a more sophisticated approach to negotiations or strategy. A higher-level behavioral prompt then incorporates the selected domain rules. The rule sets remain separate so that the prompt is easier to manage and the domain guidance easier to edit.

A compact Python representation of this separation is:

python

import json

rule_sets = {
    "negotiations": {
        "must_do": [
            "Prioritize agreements that create the highest combined value."
        ]
    },
    "geopolitics": {
        "must_do": [
            "Maintain at least three independent paths for critical resources."
        ]
    },
}


def assemble_instructions(domain: str, behavior: str) -> str:
    selected_rules = rule_sets[domain]
    return f"{behavior}\n\nDomain rules:\n{json.dumps(selected_rules, indent=2)}"

The important boundary is between general behavior and selected domain guidance. Adjusting a rule for a particular situation changes the instructions supplied to the model without requiring the entire application prompt to be rewritten.

Suggest correction

Correction for “Assemble explicit, checkable domain rules”

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:36 · section reference included

Send the prepared scenario to the reasoning model

Back in the frontend, the original question has been classified as geopolitics and reformatted for the reasoning or planning model. The displayed semiconductor scenario separates its goal, constraints, conditions and additional parameters. The application has now identified the problem and packaged the user’s material into a consistent input for analysis.

Orac Reasoning Engine showing Type: geopolitics, a semiconductor scenario goal, constraints, conditions, and additional parameters. — Orac labels the scenario as geopolitics and displays its structured summary.

The demonstrated analysis goes to an Anthropic model using the presenter’s adaptation of WorldSim, a prompting-based simulation approach whose inspiration he credits to Nous Research and Karan. The exact Anthropic model is not specified. Here, the additional simulation instructions aim to give the geopolitical analysis a more realistic context.

That context includes physical geography and institutions such as the UN and WTO where relevant. These become additional parameters and constraints for the model to consider; they do not turn generated scenarios into a validated forecast. WorldSim is useful to this particular example, but the preparation pipeline does not depend on one model provider. The prepared scenario and rules can be sent to whichever reasoning model the practitioner chooses.

Suggest correction

Correction for “Send the prepared scenario to the reasoning model”

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:11 · section reference included

Inspect how the output applies the rules

The generated response first follows the requested presentation structure. More substantively, the output references two-times leverage and greater-than-50-percent control, both drawn from the supplied geopolitical rules. These numbers are rule criteria, not measurements of model performance. Their appearance shows that the response is incorporating the domain instructions rather than merely adopting the requested formatting.

The clearest example is scenario five: the output eliminates it because it would breach a rule. The rules therefore affect which proposed options survive, not just how those options are described. The scenario concerns US semiconductor development, but the presenter explicitly sets aside whether the resulting strategy is the best policy approach. Visible rule application is the demonstrated result; strategic optimality is not.

Suggest correction

Correction for “Inspect how the output applies the rules”

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:51 · section reference included

Keep experts involved while exploring more options

The approach remains under development, with initial results the presenter describes as encouraging. In his implementation, the larger matching prompt provides more granularity than the tool-use approach he tried. Subject matter experts then supply tight rules and heuristics for the reasoning stage to apply. This makes expert participation part of constructing the system, before anyone reviews a generated answer.

Reasoning tools and simulation prompts can then explore alternatives, aiming to identify a preferred solution or return multiple scenarios. The presenter still wants a human subject matter expert in the discussion. He anticipates that experts could generate dozens of scenarios for review in a fraction of their usual time, but supplies no timed benchmark. The proposed benefit is a larger set of options for expert judgment, not the removal of that judgment.

Broader exploration could help with harder problems, decisions made under time pressure, or approaches that a practitioner might not otherwise consider. The reusable combination is a parsing engine, explicit domain heuristics and LLM problem solving. Where practical knowledge is scarce in the training record, this offers a way to bring some of that knowledge into the reasoning process—and extend model usefulness into professional domains that remain underexplored.

Suggest correction

Correction for “Keep experts involved while exploring more options”

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:53 · section reference included

Resources

WorldSim: Karan Malhotra's demonstration and discussionArticle2024-04-27
An April 2024 demonstration transcript explains WorldSim's Claude prompting, simulated terminal interface and open-ended world generation.

Read the complete timestamped transcript

0:01
So it wasn't that long ago that we were all talking about chatbots, and it felt like only yesterday that we were excited about agents. But today, all we're talking about is reasoning and how these models can now reason through complex problems.
0:16
But how is a model going to solve a problem when we haven't been able to give it any training data or knowledge in that specific domain? [upbeat music]
0:43
So what I want to do here is share a technique that you might want to try in one of these low-knowledge domains. So what do I mean by that?
0:52
Well, we know that the models are great at coding and the models are great at math, and that's not surprising because we have large bodies of trainable material, well-structured information and data that we can give to the models to train them for math and coding and the law and physics because everything is structured in a consistent way
1:13
and it's in a format where we can easily convert that into training data and give it to a model. The problem that we have is that we also have other areas where we don't have well-structured information.
1:24
So as an example, think about, uh, corporate negotiations. We know what the start point was because there was a story in The Wall Street Journal that company A wants to acquire company B, and we know what the end point was because CNBC or the Financial Times had a headline explaining what the outcome was.
1:42
But what we don't know is what happened in the room. What did those negotiations actually look like? We might have some rules or guidelines that, uh, come from an MBA course, but we don't know what those discussions were.
1:53
So there are a lot of domains where that specific knowledge, that wisdom, is locked away. It's in people's minds. It's never written down. And where this is taking us or where this could take us is that we still have these areas where this knowledge gap is not being filled.
2:07
So actually, we have, uh, sectors or professions who are becoming, um, better served by these models, who have more powerful models at their disposal. They can do more and more.
2:19
But then we have other domains who are almost being left behind. [upbeat music] [crickets chirping]
2:32
They don't have models that understand what they do, uh, and aren't able to apply them to the problems that they come across. So what we wanna do is actually close that gap.
2:41
How can we close that gap where we do have this lack of knowledge? And as practitioners in the sector I'm from, risk and security management, it is the fault of the practitioners.
2:50
We don't write things down. So this is not the fault of the people training the models, and this is nobody's particular fault. But the sheer fact is we don't write things down and therefore there won't be, there isn't and there won't be the material to train a model.
3:04
So what do we do to overcome that? How can we overcome this knowledge gap, uh, in order to make sure that we have models that can work in all of the domains where we want to apply their expertise, skill, and ability?
3:16
Well, one of the things we can do is we can use a technique that we already use. We already use this in the workplace. Someone who has just left college, they're a recent graduate, highly intelligent, but probably lower in wisdom.
3:27
They haven't had the experience. Now, there are lots of things we can do, but one of the things that works really well is giving them guidelines, giving them rules of thumb, heuristics, whatever you want to call them.
3:37
We can give them some guidelines to apply in specific domains. What that then means is they can use their intelligence, apply these rules for that domain in order to solve problems, in order to understand what's happening, and start to work through things in that specific domain while they build up their personal knowledge.
3:56
And so the technique that I'm talking about today takes that same approach, except we do that with the LLMs, with the models. We're gonna give them a set of rules to follow for specific domains and ask them to apply those on top of the very powerful reasoning capabilities that they now have.
4:15
And that's gonna give us the benefit of the powerful LLM, the benefit of the reasoning, but overcome these wisdom gaps and give them the subject matter expertise they need to start working in these other domains.
4:28
All right, so let's see what this looks like in practice. So I'm gonna toggle between this front end, which is an easier way to show the, the, the inputs and outputs, and, uh, we'll go and look at the back end as well and see what's happening behind the scenes and how the logic's working.
4:42
But this is an easier way just to show the process. So I'm gonna drop in a prompt, and right away you'll see this is quite a long prompt. It concerns an intelligence estimate.
4:52
And so what's happening in the background now is it's trying to figure out what kind of problem do I have and what rules should I apply, and we'll see how that works in a second.
5:01
The other thing it's gonna do, it's gonna reformat the, uh, question that I give it, and it's gonna put it into a standard format because, again, what I found, uh, is that standardization really helps these models work most effectively.
5:16
Okay, so here's what's going on behind the scenes. Um, the heuristic match, this is basically the same as tool use. Um, I found that tool use wasn't working out, uh, as well as I would hope, um, and so I've taken a little bit more of a, a long-winded way of doing it.
5:31
So there's a specific LLM, there's a specific prompt, um, that is going to match up, um, the user input and look for a match with one of the heuristics, and you'll see that these are listed, um, down here.
5:44
It's analyzing for those. And so that's the first thing that it's doing. Whatever you've presented to this model It's looking for a match for the specific rules that you want it to apply.
5:55
We want it to be consistent in the way it's presenting the information to the actual simulation. So we haven't got to the simulation yet. When we get there, though, we want to make things as consistent as possible and make sure that they're formatted so that the kind of material the, um, the s- the simulator needs is, is
6:13
contained in the prompt. And so we've asked it to parse the scenario and return it in the structured format, and you'll see at the front end, you actually have the opportunity to go in and edit it if necessary.
6:23
But we're just trying to make things consistent so that each time it gets a scenario, it's in the standard format. So the first two things it's doing is looking for a match, standardizing the format.
6:36
So once we have a match, we're now gonna come over to the rule set, uh, and find the appropriate rules and behavior for that particular type of problem. So we have two, uh, different types of, uh, scenario rules in the, uh, in the demo.
6:51
Negotiations, geopolitics. So these are the rules. Um, these are written in a, a fairly binary way. You must do, must not do. They're provable. It's provable if you have prioritized agreements that create highest combined value.
7:04
It's provable if you've got these, um, three-plus independent paths for crit-critical resources. And that's deliberate because we want to make it nice and straightforward rather than having sort of abstract, vague rules.
7:14
We want to give it quite clear rules. But by the time you combine these together, you've quite a sophisticated model for, uh, these are corporate negotiations or geopolitical strategy.
7:23
So these are the rules. These are the same heuristics we might give to our, our consultant that we were imagining before. And then we have this heuristics list, which is a higher level prompt, a sort of system level prompt, how you want it to behave, and you can see at the end it, it then pulls in, uh,
7:40
the rules either for negotiations or geopolitical strategy. And the only reason these are split out is just because it, it helps avoid the prompt becoming too unwieldy. It's a little easier to go in and edit these.
7:51
And so you would have these rules for the rules you want to apply. You might want to adjust these in different scenarios and different situations. So you've quite a lot of flexibility around here.
8:01
But what this gives you is this high level, uh, very comprehensive prompt set of rules and guides that really focus the model in on how you want it to behave.
8:11
Okay, so we've now got the result. So here's my original, um, my original question, and you can see it's done two things. First, it's identified the type of problem as, uh, geopolitics, and now it's re, uh, reformatted the question into a specific format, and this is just so that we've got consistent input to the actual, uh, reasoning
8:32
or planning model. So a lot of work's happened up front to take everything that the user has presented to it, identify what that is, package it up correctly, and then it's gonna send it off to the model for analysis.
8:45
So as far as the model's concerned, you can send it to whichever model you're comfortable with, whichever model you think will perform best. In this case, it's actually going to Anthropic because I'm running this WorldSim and, um, this is something, you know, I owe a great deal to, uh, Noosh Research and Karan because I first saw this,
9:03
um, uh, from one of his demos. And so it's running this WorldSim version to give it this sort of realistic view of what the world looks like, because we're dealing with geopolitics, so it's particularly relevant to this particular example.
9:16
And that's just gonna add on more realism and give it some additional, uh, constraints to think about. So the, the physical geography of the world, uh, things like the UN and the WTO need to be considered where appropriate.
9:28
So we're just sort of very quickly giving it a, a whole set of other parameters to consider or to lean upon. And so particularly, as I said, for this example of, uh, geopolitics, this is useful.
9:39
But at this stage, you could point it towards whichever model, um, you see fit, uh, and whichever one you're comfortable with. So when we come back and look at the results, obviously we've got these in a, in a presentable format.
9:51
They're easy to read. We can see right away it's following the structure that we've given it, so we know that it's obeying the general rules. And as we go down and look at the output, you'll see that it's starting to reference two times leverage, greater than fifty percent control.
10:06
And those are all rules that it was given in that large, uh, rule set for, uh, geopolitics. So we can see that it's applying those in our thinking. We know it's applied the rules that we asked it to in the considerations.
10:18
And then when you come down to examples like scenario five, it would breach one of the rules, and therefore it's being eliminated. And so not to get into the detail of whether this is the best approach or not for US, uh, semiconductor development, um, the point is we can see now that it is not only using its
10:35
reasoning ability, the intelligence that we know the model has, but it's also overcoming perhaps domain expertise or a lack of insight into how these decisions might be made. But it's overcoming that by applying the rules that we gave it, and we can see that the r- the rules are being applied.
10:53
So by this point, I'm hoping that it's given you a sense of one other approach. Um, it is a, an approach I'm still working through. However, so far, the first results are very positive.
11:04
It is able to identify the type of problem, and using, uh, a larger prompt instead of just tool use gives us a lot more granularity there. We're able to then work with a subject matter expert to give us really tight rules and heuristics to follow, and then it can apply those on top of these powerful reasoning tools
11:22
and things like the WorldSim to start to iterate through lots and lots of different options and come up with the optimum solution or give us multiple scenarios. And again, I would always want a human in the loop.
11:34
I would want the subject matter expert to be part of this discussion. But obviously, these machines are able to work so much more quickly that a human expert could generate dozens of scenarios to review in a fraction of the time it would take them normally.
11:48
And that allows us to either challenge, um, problems or take on more challenging problems, to work through, uh, many more options in a, in a finite amount of time where time is at an essence, or even look for new, uh, s- approaches that we might not have thought about.
12:03
So if you come across one of these areas, I'm hoping this technique, this approach will help you, where you're using the parsing engine, you're using the rules of thumb and, uh, applying those in that problem-solving in addition to using all of the LLMs.
12:19
Uh, and I hope that that helps, uh, us move these models into other areas, uh, that are currently a little, uh, underexplored.