Before buying an AI tool, ask the supplier to work through a task your team would actually use it for. Take the same example to each shortlisted product.
A prepared demonstration can tell you what a product is designed to do. Your test should establish what happens with the material you provide, what the answer gets wrong and how much work remains. These are the questions I would take into that discussion.
1. What does “document review” mean here?
Finding a clause, summarising correspondence and suggesting a change to a contract are different jobs. Ask the supplier to describe exactly which one the product supports.
Follow the task from the original file to the output your colleague will use. Include preparation, review and transferring the result into the firm’s system. Establish what the software does and what the user still has to do.
2. Can we choose the test?
Prepare a small fictional file with an answer you can check. For a delivery dispute, it could contain an original delivery date, an agreed extension and an email disputing whether the goods arrived.
Ask each product for a chronology. Does it record the extension? Does it treat the disputed delivery as an allegation? Include a missing piece of information and see whether the tool leaves it unresolved or invents an answer. Keep a record of the results alongside the supplier’s explanations.
3. How do we get back to the source?
Choose a statement in the answer and ask the supplier to show you the passage behind it. Can a reviewer open the document and find the relevant text without searching through the whole file?
Check whether that passage supports the statement. A real citation can sit beside a wrong conclusion. Ask what happens when the material is unreadable, incomplete or contradictory, and involve someone who will be responsible for reviewing the work.
4. What happens to our information?
Get answers about processing, retention, access and any use of information for model training. Check the terms and settings for the particular product and subscription, including connected services and support access.
Ask whether the firm can remove uploaded material and control who sees it. Find out which settings the firm must configure. Resolve unclear answers before deciding what information the product can receive; an initial test does not need client files.
5. What does the integration actually do?
Confirm the supported software versions and functions. A connection may read files without writing changes back, or require someone to transfer documents manually. Ask who sets it up, what it costs and how it is supported.
Test permissions with your IT support. Someone connecting a document library should not gain access to matters they could not otherwise see. Also check whether your existing licences already include a feature that could do the job.
6. What will we pay beyond the headline price?
Ask about minimum user numbers, processing allowances and additional usage charges. Include implementation, training and support. Check what changes if more people join or the firm processes more documents.
Measure staff time as well. Preparing a file and checking the answer are costs of using the tool. Compare the complete task with the current process, keeping additional capacity separate from an actual reduction in spending.
7. Can we leave without losing our work?
Check the commitment period, notice deadline and renewal terms. Ask which formats you can export and whether the exported work remains usable without the product.
Establish what happens to uploaded material after cancellation, including retained copies and deletion exceptions. Keep prompts, templates and working instructions outside the product where appropriate, so the firm is not dependent on access to one service.
Put the answers and test results in a comparison sheet. Leave unresolved points visible and name the person who will confirm them before a purchase is approved. I can help your firm run that comparison and implement the option it chooses.