You can't dump it into the AI and call it done

A complete AI answer can still be a bad product.

I’ve been adding frontier language models, the same kind of tool as ChatGPT or Claude, to software products for the last year.

Not as a chat box on the side.

As the step that’s supposed to read messy work and tell a person what matters.

The usual pitch is simple. Dump the files in. Dump the inbox in. Dump the spreadsheet in.

The AI will figure it out.

It won’t. Not in a way a business can use.

It’ll miss the one change that costs money. Or it’ll treat every small shuffle like something someone has to decide today.

Both look like “we used AI.”

Neither helps the person who has to act.

The two failures

There’s a quiet failure and a loud one.

Quiet: the AI skips a material change. The person reading the result thinks nothing important moved.

That silence is expensive.

Loud: it catches every header repeat, every page break, every list that got reshuffled, and treats each one like a decision.

You’ve got a complete answer.

You don’t have a useful one.

Best effort is showing what you can support and marking the rest.

Pasting the raw dump in front of someone is something else.

One case

This showed up in a document comparison app.

Someone uploads an older file and a newer file. They don’t want 80 pages of homework. They want to know what changed, and which changes affect cost, time, or risk.

They still need the full list when they want to check the source.

I ran a production-shaped pair through the live system. An 82-page comparison produced 593 possible changes.

One awkward page span was enough to abort the whole read. An imperfect table could do it. So could a bad batch from the AI.

Then the product published a fallback with no ranking at all.

One defect, and the useful part of the answer disappears.

When the run finished, the AI labeled 322 of 593 items as potentially important (54.3%). A second check cooled that to 127 of 593 (21.4%) and reversed 199 of those first-pass “important” calls.

That was better.

It still wasn’t something a person could act on.

A tidy lab example had already scored 19/19, then failed on ordinary page reflow.

Clean samples are clean.

Real files aren’t.

The point isn’t document comparison.

Same thing happens anywhere you ask an AI to read a pile and pick what matters. Email. Forms. Call notes. A folder of PDFs.

The pile is complete.

The decision isn’t.

What I changed

I didn’t start by swapping the AI.

When one item failed, I isolated it, kept the finished work, and published what I could support.

One broken item shouldn’t erase hundreds of good ones.

I wrote down what High priority, Review, and All changes mean. Labeled a known-good set. Locked a test. Then I changed the product.

If you tweak first and invent the score later, you’ll flatter yourself.

Then I fixed one named failure at a time.

A clause rewrite split into four pieces. A money line sat on a different page from its balancing total. Identical table rows only changed order. Requirement IDs glued themselves to the wrong neighbor.

Each fix had to pass the locked test without creating new unsafe merges.

The AI can stay uncertain. A changed dollar amount, percent, or day count can still get promoted. A drawings-index shuffle can stay off the first page.

I capped that first page at 5 to 10 findings and kept the full cited list.

After those rules, the locked test recalled every must-see item (25/25) and didn’t invent extras (overfire 0). The AI still predicted 31 High priority findings against a first-page cap of 10, so ranking stayed in the product.

Those labels are founder-labeled test evidence, not an independently validated accuracy study. A bake-off between models is written down and not run.

What I’d keep

Finding everything and making a decision are different jobs.

593 complete items can still be a poor decision aid.

A local defect should stay local. Retry it, isolate it, or mark it unresolved.

One bad AI response is a reliability bug.

It isn’t a reason to throw away the whole result.

Write the test before you tune. One flattering sample isn’t a go-live gate.

The AI can say an item has consequence.

You still decide what ships on the first page.

Missing IDs and invented fields are reliability bugs even when the prose sounds fine.

The AI is one step.

The product is the rest of it. What evidence it’s allowed to see. What you do when one unit fails. What a person sees first.

If you treat “did the AI find it” as the source of truth, you’ll ship a dump.

A short first page has the opposite failure. You hide a money term and call that restraint.

I’d keep the complete, cited list.

Then decide, on purpose, what a person sees first.

That’s the work I do when I put AI on a real workflow. Human review stays in. Exceptions stay with a person. The full record stays available.

If you’ve got a process like that, and someone is telling you to dump it into ChatGPT, I wouldn’t.

That’s usually the part they skip.