AI is a compiler, not a runtime

I have been building a system that turns messy logs into structured data, and I want to tell you about the moment I almost built it wrong. A lot of people are making the same mistake right now, and it gets expensive fast.

Everyone is racing to put a model in the loop. Every log line, every event, every payload goes straight to the LLM so it can figure things out at runtime. It feels like the future. Most of the time it isn't.

Logs are a good example to think this through with, so I will use them throughout. The constraints are simple and unforgiving. Every service writes its logs a little differently, the parsed fields have to be exactly right, and you process millions of lines. If a field is wrong, an alert misfires or a number on a dashboard is quietly false. There is no "roughly correct" here.

So the real question is where the AI actually belongs.

The obvious answer is a trap

My first instinct was the same as everyone else's. Send every line to a model. Pass the raw text, write a good prompt, parse the JSON it hands back. It demos really well.

Then you live with it for a week.

The first problem is that it isn't deterministic. I watched the same line come back two different ways on two different runs. That isn't a bug you patch. It's how the thing works.

The second problem is cost. You pay per line, forever, for work that is basically identical every time. At log volume that bill is its own disaster.

The third problem is that you can't test it. There is no unit test for a prompt. You can't diff a feeling. When it breaks three months from now, you won't know why, and you won't know when you've fixed it.

For something creative, that's all fine. For a high volume pipeline where being correct is the entire point, those three properties rule it out.

A pattern: let the AI write the program, then retire it

The shift in how I think about this was realizing the AI doesn't need to read every line. It needs to read a handful.

When a new log format shows up, the model reads a sample, works out the shape, and writes a small declarative rule document. A little program that describes how to parse that format.

{
  "source": "acme-service",
  "pattern": "<timestamp> <level> [<service>] <message> took=<latency_ms>",
  "types": { "timestamp": "iso-8601", "latency_ms": "integer" }
}

After that, a plain parser replays those rules on every future line.

def parse(line, rules):
    # no model call here, just code
    fields = apply_pattern(line, rules["pattern"])
    return coerce_types(fields, rules["types"])

That's the whole idea. The AI is the compiler. It runs once, when a new format appears. The parser is the runtime, and it never calls the model at all. The expensive work, understanding the format, happens one time. The repeatable work is ordinary code.

This fixes the three problems. It's deterministic, so the same line gives the same output every time. It's cheap, because you pay once to write the rules and the millionth line costs nothing. And it's testable, because the rule document is a real artifact you can version, diff, and run against real lines.

The discipline is what makes it trustworthy

Treating the AI as a compiler is the easy part. Trusting what it writes is harder, because AI authored code is still AI output. It's confident, it's plausible, and sometimes it's wrong. Most of the real work is in not trusting it blindly.

Three ideas do most of that work. This is the flow they produce:

No retry loop, on purpose

The popular design is to let the AI fix itself. Generate, check, and if it's wrong, hand it the error and let it try again until it passes.

I almost did that. It's the obvious move, and plenty of agent frameworks push you toward it. I backed off when I understood what I would actually be shipping. A model that keeps retrying until the output happens to match isn't reliable, it's lucky. I don't want a parser that's correct on the fifth attempt. I want it correct the first time, or rejected. One attempt to write the rules, then a strict deterministic check. If it fails, it fails loudly and a person looks at it. An honest failure beats a model that argued its way to a pass.

It has to pass its own tests before it exists

I learned this one the hard way. The first pattern I generated looked perfect. It parsed every line in the sample I tested it on, and I was happy for about ten minutes. Then I ran it against a fresh batch from the same service and it fell apart. Some entries wrapped across several lines, a few carried an extra field, and the pattern had quietly memorized the sample instead of understanding the format.

So I set a rule. A generated pattern doesn't get saved because it looks right. It has to parse a whole set of real lines, all of them, exactly, before it is allowed to be saved. Pass, or it doesn't ship. The AI writes the candidate. A pile of real log lines decides whether the candidate is any good. Writing is cheap and creative. Checking is strict and boring. You want both, and you want them to be separate systems.

Some things the AI cannot decide

This is the part people skip, and the one I got most stuck on.

I hit a case where a request retried itself internally and showed up as several lines. Should that be one event or several? Both are valid. I spent far too long trying to get the model to pick the right answer with better prompts and more examples before it clicked. The answer isn't in the log. Whether those lines are one event or many depends on what your metrics treat as an event downstream. That's a modeling decision, not a parsing problem. No model can infer it, because the information simply isn't on the line.

So I stopped asking. That choice lives in a small config a human confirms, and the AI works inside it. Working out where that line sits, what the model gets to decide and what a person has to, turned out to be most of the design.

The payoff

In steady state you barely pay for AI at all.

You pay once, when a new format shows up, to write the rules. Everything after that runs through plain code. The AI bill doesn't scale with volume. It scales with the number of distinct formats you handle, which is a much smaller and slower growing number. That's the difference between a cost that grows against you and one that levels off.

What I actually took from it

AI is very good at understanding something once. It's the wrong tool for doing the same thing for the ten millionth time.

The people who get the most out of this won't be the ones who put a model in the most places. They will be the ones willing to take it back out of the hot path, using it to produce something durable, checking that thing carefully, and running ordinary code in production where ordinary code belongs.

Use the AI to write the program. Don't make it the program.