PM’s Vendor Evaluation Checklist: What to Ask Before You Sign a Tool Contract
A vendor evaluation that starts with a Google Form and ends with a signed contract three weeks later is not evaluation, it’s a rubber stamp with extra steps. A nine-person product team picking an analytics tool because the AE was responsive and the demo had confetti animations, then discovering four months later that the tool couldn’t do cohort retention without a paid add-on nobody scoped, is a common enough story to be a pattern rather than an anecdote. The build vs. buy decision was right. The vendor selection underneath it was not, and that’s the part most PMs skip.
Once a team has decided to buy rather than build, using something like the build vs. buy decision framework, the real work starts: comparing finalists, catching the gaps between the sales deck and the actual product, and building a case procurement and finance will actually approve. This is a different skill than the build/buy call itself, and most PM training skips it entirely.
Why Vendor Evaluation Breaks Down Even When the Decision to Buy Was Right
Most vendor evaluations fail in one of three predictable ways. The first is scope creep in reverse: the team evaluates against today’s requirements and ignores where the product needs to be in eighteen months, then re-litigates the whole decision at renewal. The second is letting the sales cycle set the pace — a vendor with an aggressive AE and an end-of-quarter discount can compress a six-week evaluation into six days, and rushed evaluations consistently under-test the edge cases that matter most. The third, and the most damaging, is treating the demo as the product. Demos are built by sales engineers to hide friction, not reveal it.
A recurring version of this: a vendor evaluation for a feature flagging platform where the demo environment has every flag pre-configured and every dependency mocked out. It looks instant. Against a real production service architecture, flag propagation can run into double-digit seconds — long enough that a kill switch for a bad release is nearly useless for the exact emergency it exists to solve. Catching that requires insisting on a sandbox trial against a staging environment instead of accepting the demo at face value. Teams that skip that step find out later as an incident, not a procurement mistake.
Build the Evaluation Criteria Before You Take a Single Call
The evaluation criteria have to exist before the first vendor call, not get reverse-engineered from whichever finalist a team is leaning toward. Write down, in order of priority, what the tool must do, what it should do, and what would be nice. Separate these three tiers explicitly, because sales reps are very good at reframing a “should” as a “must have, and we cover it” when the honest answer is a roadmap item six months out.
For teams evaluating something like feature flagging tools, the “must” tier usually includes propagation latency, audit logging, and SDK support for the actual stack in use — not a generic list of features copied from a review site. For teams evaluating accessibility testing tools, the must tier should include whether the tool tests dynamic content post-render, not just static markup, because that’s where most automated scanners quietly under-report real WCAG violations.
Weight each criterion. A simple 1-3-5 scale (nice to have, important, dealbreaker) forces the conversation about tradeoffs before anyone gets emotionally attached to a specific vendor. Teams that skip weighting tend to pick whichever tool “felt” most polished, which correlates more with the vendor’s design budget than with fit.
The Questions That Separate a Real Evaluation From a Sales Pitch
Ask these in every vendor call, and write down the answers verbatim, not a paraphrase of them:
What happens to your data if you cancel — is there an export path, and in what format? Vendors answer this vaguely more often than any other question, because the honest answer for some of them is “you lose it or pay us for a migration.”
What’s the actual SLA, not the marketing page’s uptime claim, and what’s the penalty structure if they miss it? A 99.9% SLA with no financial remedy is a marketing statistic, not a commitment.
Who owns the roadmap item being called “coming soon”? If the answer is vague or the timeline has already slipped once during the evaluation, treat “coming soon” as “not happening” for decision purposes.
What does the actual support experience look like at 2 a.m. on a Saturday, not during a sales-assisted trial? Ask for the support tier breakdown and, if possible, talk to a current customer outside the reference list the vendor hand-picked.
How does pricing scale, specifically — per seat, per event, per API call — and what does the bill look like at 3x current usage? Usage-based analytics tools have a well-documented pattern of a bill quintupling within a single growth quarter when nobody modeled the pricing curve past current volume.
When the Team Disagrees on What Actually Matters
Weighting criteria sounds like a mechanical exercise until an engineering lead wants to weight “API quality” at a 5 and a support lead wants “response time SLA” at a 5, and the scorecard only has so much room before every criterion is a dealbreaker and the exercise becomes meaningless. In a feature-flagging evaluation shaped like the one above, an engineering lead might score SDK ergonomics as a must-have, arguing that a clunky SDK would slow every future flag rollout, while a support lead pushes back that a nine-second propagation delay was the thing that actually caused the outage everyone was trying to prevent — a slightly awkward SDK is an inconvenience, not a repeat of the incident.
The resolution is to go back to the actual incident report and ask, plainly, which of these two things would have prevented what happened. Propagation speed usually wins that argument, and SDK ergonomics moves down to “should have.” That’s the discipline weighting is supposed to enforce — forcing the team to argue from evidence instead of personal preference — but it only works if someone is willing to be the tiebreaker when two reasonable people land in different places. Skip that step and the scorecard just becomes a way to make an already-decided outcome look rigorous after the fact.
When This Breaks: The Failure Modes of a Rushed Evaluation
The most common failure is skipping the reference call, or worse, only calling references the vendor hand-selected. A vendor-selected reference almost never volunteers anything critical about the product. Going around the list and finding a customer independently — through a mutual connection or a public case study with a name attached — usually produces a different picture, sometimes significantly.
The second failure is not testing with real data volume. A tool that performs beautifully with the sample dataset in a sandbox can degrade badly at actual scale. A user research repository tool indexing and searching transcripts fine at 50 interviews and becoming nearly unusable past 2,000 is exactly the volume a maturing research practice hits within a year — and exactly the kind of failure a sandbox test with a small sample dataset won’t catch.
The third failure is signing before legal and security review finishes, because the sales rep created urgency around an end-of-quarter discount. If a vendor’s best price requires skipping the diligence process, that’s information about the vendor, not a reason to skip it. The discount reappears the following quarter far more often than sales reps want anyone to believe.
Recovery from a bad vendor pick is expensive in a way that’s easy to underestimate going in: migrating off a tool after six months costs roughly triple the evaluation time it would have taken to do it right the first time, once data migration, retraining, and the parallel-running period most teams need before fully cutting over are all counted.
A Worked Example: Choosing a Feature Flagging Vendor Under Time Pressure
A 30-person B2B SaaS company at roughly $4M ARR needed a feature flagging tool after a bad release took down checkout for two hours with no fast way to roll back the specific feature. The team had six weeks before the next major release and wanted the tool live before then.
They wrote requirements first: sub-second flag propagation (must), audit log with user attribution (must), percentage rollout by user segment (must), SOC 2 Type II (must, given they sold into enterprise accounts), and a visual dashboard non-engineers could read (should). Three vendors made the shortlist.
The team ran a real sandbox test against their staging environment, not the vendor’s demo instance, and timed propagation under load. One finalist that looked identical to the others in the sales deck came in at nine seconds under moderate load — a dealbreaker they never would have caught from the demo alone. They called two references found independently through a PM Slack community rather than the vendor’s list, and one flagged a billing surprise around per-environment pricing that wasn’t in the vendor’s initial quote. That question alone saved roughly $1,800 a month once they accounted for their actual environment count (dev, staging, and two regional production environments).
They signed with the remaining finalist, but only after pushing the vendor to write the data-export guarantee into the contract itself rather than accept it as a verbal assurance from the AE. Eight months later, an acquisition changed that vendor’s pricing model, and the export clause is the only reason the migration took a weekend instead of a month.
Vendor Evaluation Scorecard Template
| Criterion | Weight (1-3-5) | Vendor A | Vendor B | Vendor C | Notes |
|---|---|---|---|---|---|
| Meets must-have functional requirements | 5 | Verified in sandbox, not demo | |||
| Data export path on cancellation | 5 | Get this in writing | |||
| SLA with financial remedy | 3 | Not just an uptime percentage | |||
| Pricing model at 3x current usage | 5 | Model the curve, don’t take current price at face value | |||
| Independent reference check (not vendor-selected) | 3 | Find at least one reference yourself | |||
| Roadmap item timeline history | 1 | Has “coming soon” slipped before? | |||
| Support tier and real response time | 3 | Ask about weekend/off-hours coverage |
Place this table early in the evaluation process, not as an afterthought after a favorite has already been picked — scoring retroactively just rationalizes the choice that was already made emotionally.
Getting Procurement and Finance to Actually Approve It
Procurement and finance don’t care that the tool has a nicer UI than the competitor. They care about total cost of ownership, contract risk, and whether this duplicates something already licensed elsewhere. Before bringing a recommendation to them, know the answer to: does anything currently licensed already do 80% of this job, even badly? If a product ops manager role exists on the team, loop them in early — tooling decisions that touch multiple teams are exactly the kind of cross-functional coordination product ops is built to own, and skipping them creates rework later when they discover the purchase after the fact.
If budget is a real constraint, it’s worth an explicit pass through free product management tools before assuming a paid vendor is required — sometimes the honest answer is that a free tier or an adjacent tool already paid for covers the must-have tier well enough that the business case for a new contract doesn’t clear the bar.
Bring finance a one-page summary: total contract value including implementation costs, the pricing curve at 3x usage, what’s being replaced or not replaced, and the risk of doing nothing for another quarter. Skip the feature comparison — that’s not what they’re evaluating, and burying the cost story in a features table is how approvals stall for weeks in follow-up questions.
What to Do the Next Time You’re Handed a Vendor Shortlist
The next time someone hands over three vendor names and asks which one to pick, resist the urge to jump straight to a bake-off. Write the criteria and weights first, even if it takes an extra day. Insist on a sandbox test against the real environment, not the demo. Find at least one reference independently. And get the data-export and pricing-at-scale answers in writing before signing anything — not because the worst is expected, but because the vendors who balk at putting those in writing are revealing something true about what happens after the contract is signed.
References
- Nielsen Norman Group — usability and evaluation heuristics for enterprise tool selection
- Harvard Business Review — negotiation strategy coverage relevant to vendor sales-cycle pressure
- Silicon Valley Product Group — build vs. buy and product organization decision-making
