Why Your ATS Demo Lied to You, and the Eight Questions That Expose It
We give demos. Everything below applies to ours, which is why it is worth reading before you sit through another one.
A software demo is a rehearsed performance on curated data. The database is clean, the search returns the right person because somebody checked that it would, and the workflow runs a path the presenter has run two hundred times. None of that is dishonest and none of it predicts what the platform is like on a Tuesday with your data. Eight questions change that.
Last reviewed August 2026. Written from the vendor side, about vendor practice including our own.
Key takeaways
- Demo data is curated, which is the whole problem. Search works beautifully on a database assembled so that it would.
- The rehearsed path avoids the ugly cases. Duplicates, ambiguous replies and failures do not appear unless you ask.
- Most disappointment is discovered at renewal. Add-on pricing, missing integrations and adoption failure all surface later.
- Eight questions move the conversation from performance to evidence. All eight apply to our demos too.
What a demo cannot show you
Your data. The single largest gap. A demo database has no duplicates, consistent formatting and complete records. Yours has none of those, and search quality is mostly a function of data quality.
The failure cases. A candidate replies with something ambiguous. Two records exist for one person. A sync fails silently. None appear in a rehearsed path.
Adoption. The most common failure in this category is a capable platform nobody switched to, and no demo predicts it.
The real price. Which modules you would actually license, what implementation costs and what it renews at.
The eight questions
1. Can we run this on a sample of our own data? The most important one. A vendor who declines is telling you something.
2. Show me what happens when the same person exists twice. At the point of entry, not at cleanup.
3. Show me a case the workflow was not configured for. An ambiguous candidate reply. Does it escalate or act?
4. Open a score and explain it. If the answer is a number with no reasoning, your recruiters will ignore it.
5. What is the all-in three-year cost, including every module we would license and the renewal escalation?
6. What does the migration include, in writing, and what will not carry across?
7. What happens in week one, week four and week twelve? A vague answer is the answer.
8. What does leaving look like? Notice period, export contents, and whether it includes notes and activity history.
Why question one matters most
Everything else is secondary to running the platform against your own records.
An audit of one firm's 3 million candidates found roughly 10 percent duplicates, 10 percent unusable and 80 percent real people with out-of-date records. Search behaves completely differently on that than on a curated demo set, and the difference is exactly what you are trying to evaluate.
The specific test: take a role you have already filled, describe the person you hired, and see whether they appear in the top twenty. You already know the right answer, which makes it the only benchmark that cannot be gamed. More on this in natural language search.
The questions vendors dislike
Five and eight, consistently.
Question five exposes packaging. A platform publishing an entry price while quoting AI, automation and analytics separately has a headline that is not a price, which we go through in ATS pricing in 2026.
Question eight exposes lock-in. A vendor confident in the product answers it in a sentence. A vendor who becomes vague about export contents is telling you what the renewal conversation will be like.
Neither question is aggressive. Both are what a competent procurement function would ask, and most staffing firms do not have one.
What to do after the demo
Write down which number will move and by how much. Placements per recruiter, portal spend, time to shortlist. Then check at day ninety. Almost nobody does the second part.
Talk to a customer the vendor did not select. Reference customers are chosen. Find one yourself.
Ask your most resistant recruiter to sit through it. Their objection is usually rational and it is better heard before signing than after.
Get the migration plan in writing before you commit, not after.
Frequently asked questions
What should you ask in an ATS demo?
Whether you can run it on your own data, what happens when a duplicate record exists, how it handles a case nobody configured, how a score is explained, the all-in three-year cost, what migration includes, what happens in weeks one to twelve, and what leaving looks like.
Why do software demos mislead buyers?
Because the database is curated so search works, the workflow follows a rehearsed path that avoids failure cases, and neither adoption nor the real all-in price can be shown in a scripted session.
How do you test recruitment software properly?
Take a role you have already filled, describe the person you hired in plain language, and check whether they appear in the top twenty results. You already know the correct answer, which makes it the one benchmark that cannot be gamed.
What is the most important demo question?
Whether you can run the platform against a sample of your own data. Search quality is largely a function of data quality, and a curated demo database tells you nothing about how it will behave on records with duplicates and stale fields.
Why does the all-in price question matter?
Because several platforms publish an entry price while quoting AI, automation and analytics separately. The headline is a floor rather than a price, and the difference usually surfaces at renewal rather than at purchase.
Should you ask a vendor what leaving looks like?
Yes, before signing. Notice period, what the export contains, and whether it includes notes and activity history. A vendor confident in its product answers in a sentence; vagueness predicts the renewal conversation.
The one thing to insist on
Refuse to evaluate any platform, including ours, without running it against a sample of your own records. Every other question on this page is secondary to that one.
A vendor who cannot accommodate it in a reasonable timeframe has told you what you needed to know without answering anything.
Bring your own data to the demo
Send a sample of your records and one role you have already filled. We will run it live rather than showing you ours.
Book a demoFounder of Sortinghat, an AI-native ATS and CRM for staffing, search and RPO firms. Writes about recruiter capacity, sourcing economics and what actually changes when AI reaches a delivery desk. More about the author
Related reading
ATS Pricing in 2026: What Staffing Firms Actually Pay
Talent PoolNatural Language Candidate Search: Five Real Queries, and Why Vague Ones Fail
Product Comparison 101ATS Migration: How to Switch Without Losing Ten Years of Notes
IntegrationsATS Integrations: What Syncs One Way, What Syncs Both, and How Often