I love using stupid models.
Precisely because of how unpredictable they can be… kinda like our end users.
Last Thursday, a founder asked me during a consultation (we’ve been talking about automated SaaS product reviews)
“Wait, you’re not using Fable here? Is it because of token optimization?”
Well, yes and no.
Ofc, using a cheaper model matters when you want to run hundreds of synthetic user sessions. That’s token economy
But there’s another reason I deliberately use older, less capable models for some of them:
they get stuck.
They misunderstand things and click something that makes absolutely no sense.
They fail to figure stuff out
And thats exactly what I want.
When I’m testing an agent that writes software, I want the smartest model I can get.
When I’m simulating someone using that software, I’m not so sure.
Because your users aren’t sitting there thinking:
“Let me carefully infer the intended interaction pattern from the information architecture.”
They click the big red button
They don’t read the description.
They get confused and leave.
So when one of my deliberately “stupid” agents gets stuck in a flow, my first reaction isn’t:
“Oh no 😱, the model failed.”
It’s:
“Interesting. What made it fail?”
Maybe a smarter model could reason its way through the interface.
But perhaps that’s actually hiding a problem your users will run into.
We’ve been playing with this idea in Flow. Using synthetic users to go through an actual product and see where they succeed, fail, get confused or find a path we didn’t expect.
And one thing I’m increasingly convinced about:
The best model for testing your product isn’t necessarily the smartest one.
Sometimes you want the one that’s just stupid enough to behave like a real user.
You can apply for early access of impl / flow in here https://www.theimpl.com/flow