You walk into a bookstore and ask the clerk whether they have a particular book. The first clerk says, "Aisle five, second shelf," without looking up. You walk to aisle five, the book isn't there, and you've lost ten minutes. The second clerk says, "I think so, but let me check." They type something and squint at the screen. Then they come back with the book in hand, or with a plain "we're out, but I can order it for you."
Both clerks know about the same amount about the shelves. The difference is that the second one tells you how sure they are. That one extra piece of information lets you decide whether to trust the answer or go and look for yourself.
A new type of AI model does exactly this, and I spent a few days finding out how well it works.
A model that answers with a number
Most AI models write text. You ask a question and get a paragraph, and the paragraph sounds equally confident whether it's right or wrong. Over the last few weeks, three companies have released models that work differently. You give them some text or an image, plus a few questions that have fixed types of answers. The answer is yes or no, one choice from a list, or a score from low to high. The model answers each question and attaches a probability. It doesn't write anything.
TypeSafe AI released one called Jev. AWS released an open one called Strands Decider that you can run on your own machine. OpenAI added a Decisions endpoint to its API. They're fast, a fraction of a second per answer, and cheap, because there's no text to generate.
I ran the AWS model on my laptop first, with made-up scenarios. One customer message said the pages of a birthday gift were printed upside down and a refund was needed by tomorrow. The model called it a return request at 95 percent and urgent at 85 percent. That was a confident answer, and a correct one.
Then I gave it a purchase order with every sign of fraud. The supplier was new, the price was far below market, the payment was a wire to a new account, and the total was four times the spending limit. The model split almost evenly between approve and deny, 41 percent to 40, and reported a confidence of 12 percent. The label it picked was wrong. The number was right, because a confidence that low means the system is telling you to ask a person. That's the second clerk.
A test that could fail
Made-up scenarios flatter a model, because I wrote them knowing the answers. I wanted something I could grade.
My sticker shop puts a small maker's mark on every sticker, a short word and an emblem. A vision model checks each image for it before the sticker goes on sale. That check leaves a record: for each past sticker, the answer the current model gave. I pulled fifty of them and sent the images to OpenAI's Decisions API, the one of the three that accepts images. I asked a single question: is the mark there, and is it spelled correctly?
It agreed with the existing check on 49 of 50. The one disagreement was a sticker where the mark was there, tiny, below a line of text, and the new model said 22 percent. I looked at it myself, and the new model was wrong.
That result told me very little. Of 207 stored verdicts in the shop's records, exactly one was a failure. A test where nearly everything passes can't show whether a model catches failures, so I made some.
I took 46 stickers where I could find the mark, and made four broken copies of each. In the first copy I painted the mark out. In the second I cut the word in half. In the third I dropped the last letter, so the word read "shilpiwork". In the fourth I swapped two letters, so it read "shilpiwroks". That made 184 images, and every one of them should fail.
What it caught and what it missed
It caught every image with the mark painted out and every one with the word cut in half, 46 of 46 each, and it scored them near zero. It also caught 44 of the 46 with a dropped letter.
The swapped letters were a different story. It caught 21 of 46 and passed the other 25 as correct. Twenty of those 25 got a score of 90 percent or higher. The median score on a broken mark was 60 percent, which means the model leaned toward calling it fine.
Raising the bar doesn't fix this. If I flag everything under 95 percent, I catch 90 percent of the broken marks and get two false alarms among the 46 good stickers. If I flag under 99 percent, I catch 91 percent and get eight false alarms. Most of the misses that remain are swapped letters the model is confident about.
I can't say why. My guess is that a model reading a short word takes in its overall shape, and two swapped letters in the middle barely change the shape. People skip over the same mistake, which is why proofreading is hard. But I didn't test that guess, so I'm leaving it as a guess.
Where each check belongs
Think of buying a house. You can walk through it in ten minutes and see that it's standing, that it has a roof, and that water comes out of the taps. You can trust that quick look. Whether the joists under the kitchen floor have rotted is a different question, and answering it takes an inspector with a probe. A decision model is the walk-through. The comparison in code is the inspector.
The check my shop already runs doesn't ask a model whether the word is spelled right. It asks the model to read the mark out as text, then compares the letters in code and measures how many edits separate the result from the correct word. Two swapped letters come out as a small distance, and the code flags it for a look. A decision model can't do that, because it never returns the text.
So the split is straightforward. Use a decision model for the coarse question, whether the thing is there at all, where the number held up. Keep the exact question, whether these letters match, in code that compares them. A quick first look still can't replace the exact check, but it rejects the obvious failures cheaply, so the slower check runs on fewer images.
What this test doesn't show
A few limits apply. I drew the swapped and dropped letters in a plain font, while the real marks use a serif face. The model may have been reacting partly to a font that didn't match. I didn't run the existing check on the broken copies, so I can't say it does better. I ran only OpenAI's model on images, ran the AWS model only on text scenarios, and didn't run Jev at all. And only one failure out of 207 was a real one, so the false-alarm rate rests on thin evidence.
Back to the aisle
The second clerk wasn't smarter than the first. They were better at saying where their knowledge ran out, and that's the only reason their "I think so" was worth anything.
These models do the same, but only for some questions. For "is it there," the number meant something. For "is it spelled right," it often meant almost nothing. So the next time a system tells you it's 95 percent sure, ask which question it was answering. For "is it there," I'd take the number. For "is it spelled right," I'd walk to the aisle myself.