Start with the problem, not the product
Most AI pitches begin with what the product can do. Turn that around. Before any meeting, write down the two or three problems you would actually pay to solve: quotes take too long to prepare, the shared inbox is overflowing, month-end reporting eats a day. Then ask the vendor to show, specifically, how their tool deals with one of those, using an example close to your own work.
If a vendor cannot connect their product to a concrete problem you have, it is probably not for you yet, however impressive the demo. Plenty of AI features are clever answers to questions a small business never asks. Estimate the value roughly before you go further. If a tool might save each of four staff twenty minutes a day on a task, that is a figure you can weigh against its price and the effort to set it up. If nobody can say which task it speeds up or by how much, the purchase is a hope rather than a plan.
Be sceptical of demos
Demos use carefully prepared data and well-rehearsed prompts. The AI summarises a tidy document, answers a predictable question and drafts a neat email. Your data is messier, your questions less predictable and your staff busier. The gap between demo and daily use is where many AI purchases disappoint. Ask for a trial with your own content, or at least to watch the product handle an example you provide on the spot. Pay attention to how it copes with mistakes, unusual inputs and questions it cannot answer. A good tool says when it does not know; a weak one invents something plausible.
Questions about data
Ask exactly what information the AI feature uses, where it is processed and stored, and whether it is sent to another company's AI model. Many products use models from one of the large AI providers behind the scenes, so your data may pass through more than one organisation. Ask whether your data is used to train any model, whether you can opt out, and how long prompts and outputs are kept. Ask who in your business can see what the AI produces, and whether it respects the permissions already set in the product. An AI search across your CRM that ignores access controls could show sales commissions or HR notes to the wrong people.
Questions about accuracy and control
Ask how the vendor measures accuracy and what they recommend for checking output. Ask whether the AI shows its sources, such as the documents or records an answer came from, so staff can verify it. Find out whether AI actions are logged, whether a person must approve anything before it is sent or changed, and whether administrators can turn individual features on or off. For anything that makes decisions about people, such as recruitment screening or customer credit, ask how bias is tested and how a decision can be explained to the person affected. If the vendor cannot answer clearly, keep those decisions with people.
Questions about cost and lock-in
AI features are priced in many ways: a per-user add-on, a higher plan tier, usage credits that run out, or free for now with pricing to follow. Ask for the full cost after the trial or introductory period, how it scales with users and usage, and what happens if you exceed limits. Check whether turning the feature off later removes anything you have come to depend on. Ask what happens to generated content, summaries and any AI-built configurations if you cancel. Can you export them? Are they deleted? A tool that holds your knowledge in a format you cannot take with you becomes harder to leave each month.
Running a fair trial
Agree success measures before the trial starts: minutes saved per task, error rate, staff willingness to keep using it. Choose a small group who do the relevant work, give them the tool for a few weeks and ask them to log both wins and problems. Compare with how the same work was done before. Include the person who will manage the product, since admin effort is a real cost.
At the end, decide on evidence. If the tool saves time and is accurate enough with reasonable checking, buy what you need for the people who use it. If the results are mixed, it is fine to wait. AI products are improving quickly, and the right choice next year may be different from the right choice now. Office IT can sit in on vendor meetings as an independent second opinion, but a careful trial run by your own staff answers most of the questions.
In short
- Define the business problem first and ask the vendor to solve that one.
- Test with your own data; demos are prepared to look good.
- Ask where data goes, whether it trains models and whether permissions are respected.
- Know the full post-trial cost and how to export your content if you leave.