Two Claims Inside "We've Done This Before"
Every agency says it has done this before, and the sentence covers two very different claims. Three written questions, a relevance filter and a fair way to judge a thin track record separate the one that predicts your outcome from the one that doesn't.
- agency selection
- vendor evaluation
- track record
- due diligence
Every agency you talk to will say it, and each of them will mean something different. "We've done this before" is the most common sentence on an AI sales call and the least tested. It's two claims wearing one coat, and the gap between them is where most of your risk sits.
The first claim is: we've worked in this category. They've built AI tools, shipped deployments, sat in rooms with companies in your industry. For most agencies that's true, and it tells you almost nothing. An agency can have fifty engagements behind it and no idea whether any of them were still working a year later. Activity isn't outcome.
The second claim is: we've solved this specific problem, for a company about your size, and we can show you what it looked like twelve months after launch. That one is much rarer, and it's the only version that predicts how your engagement will go. A sales call is built to let the first claim pass as the second. Your job is to pull them apart.
Three questions that separate them
Ask these in order, in writing, and keep the answers.
First: what did you build, for what kind of company, and what did it produce six months after launch? Not at launch. Six months after.
An agency with real history answers with a described client, a specific workflow, and a before-and-after number that someone measured independently of the launch celebration. An agency without it tells you the launch story and stops, because the six-month story means going back to the client, and an agency that never went back doesn't have one. Test the number itself: it needs a denominator and a date. "Cut handling time by 40%" fails. "Median handling time on tier-one tickets fell from eleven minutes to seven, measured by the client's helpdesk over the quarter after launch" passes. (That's an invented example of the shape, not a benchmark.)
Second: can I speak with the person at that company who owned the project? Ask for the owner, not the signatory. The signatory approved the invoice. The owner lived with the result.
A reference call is the only reliable check on anything an agency says about its past. Agencies with real outcomes have clients who'll take the call, because the result held. Agencies without them find reasons: the client has moved on, there's an NDA, they'd have to check. Every one of those can be true and every one can be deflection. The only way to tell is to ask each agency the same way and watch the pattern.
Third: what would you do differently if you ran that engagement again? Specific opinions are what learning looks like: a scope drawn too wide, a handoff nobody rehearsed, a governance rule added in month four that belonged in week one. An agency that hasn't done the work gives you answers about communication and alignment, because those are universal, safe, and sound like lessons.
Relevance is its own test
Suppose the claim survives all three. It still has to be about work like yours, and three variables decide that: company size, workflow type, and industry.
Size sets how much governance and spare organizational capacity exists around the project. Workflow type decides where the expertise lives, since claims intake and sales forecasting share little beyond the word "AI". Industry sets data sensitivity and compliance limits.
Ten implementations at companies with 500 or more employees is relevant experience for another company that size. It doesn't transfer reliably to a 75-person professional services firm on a basic CRM plan, where there's no data team to hand the work to and no governance to inherit. So ask for three case studies filtered to your size bracket, your workflow, and your industry. A match on all three is rare, and you should expect that. A mismatch isn't disqualifying. An unmentioned mismatch is. If the agency doesn't tell you where its experience differs from your situation, ask yourself why you had to find it.
When a thin record still works
A short track record doesn't end the conversation if other evidence compensates. Three things count.
Documented method comes first: redacted outputs you can read, rather than a description of a method. Next is principal involvement, meaning the person you're evaluating is the person who'll run your project, not someone you meet at kickoff. Last is a fee structure that ties part of the payment to outcomes rather than milestones.
The third is the strongest signal because it costs the agency something to be wrong. An agency willing to be paid on results has said "we know how to do this" in the one currency a pitch can't fake. It has limits, though. Outcome-based fees only work when both sides agree in advance how the outcome gets measured, and many engagements never settle that. If the conversation is hard, you've learned something about the measurement you needed anyway.
The test for tomorrow
Take the last agency pitch in your inbox and find the sentence where they say they've done this before. Send the three questions in writing and give them a week. What comes back, and how long it takes, will tell you which claim you were being sold.
If you want that question answered for your specific situation, the Forge Playbook does it. Answer a few questions about your business and we'll put together a tailored outline of which workflows are worth automating and what a realistic budget looks like for each. Free, no obligation, takes about three minutes.
