Before buying an AI tool, ask these 12 questions

Why AI procurement needs a different checklist

Traditional software procurement often starts with a feature list, security review, price and integration plan. Those steps still matter for AI, but they do not answer the most important question: how does the system behave when the input is ambiguous or the output is plausible but wrong? Generative systems can produce different answers to similar requests, depend on external model providers and change as vendors update their services.

The right approach is not to treat every AI tool as uniquely dangerous. It is to match the depth of review to the use case. A tool that helps format internal meeting notes has a different risk profile from one that recommends credit decisions, drafts legal advice or changes production systems. Write down the intended use before evaluating vendors; otherwise, a broad promise like “AI productivity” is too vague to test.

1. What exact job will the tool do?

Describe the task in plain language and identify the person who owns the outcome. Is the system drafting, searching, classifying, recommending or taking action? Define what a successful result looks like, what must never happen and which decisions remain with a human. A narrow use case makes vendor claims easier to verify and prevents a pilot from becoming a collection of unrelated experiments.

2. Does it work on your own representative cases?

Ask for a time-limited pilot using a representative, appropriately protected set of examples. Include routine cases, incomplete inputs, edge cases and examples where the correct response is to ask a question or decline to answer. Agree on a scoring rubric before the pilot begins. Depending on the workflow, evaluate factual correctness, completeness, consistency, latency, formatting and correct use of tools.

Do not evaluate only the final output. Measure the work required to get there: retries, prompt editing, manual corrections, review time and escalations. A system that produces a plausible draft quickly but takes a specialist twenty minutes to verify may not create the expected productivity gain.

3. What evidence supports the vendor’s claims?

Ask for documentation about the model or service, known limitations, evaluation methods, supported languages, version history and intended uses. If a vendor quotes a benchmark, ask whether it resembles your task and whether independent testing is available. A benchmark score is evidence about a particular test, not proof that a system will perform well in your environment.

Request a clear description of what the vendor has tested and what remains unknown. If the system is built on another provider’s model, clarify which party is responsible for model updates, incident communication and the quality of the integrated product.

4. What happens to your data?

Map the information the tool receives: prompts, uploaded files, conversation history, retrieved documents, tool outputs and feedback. Ask whether this information is retained, for how long, where it is processed, who can access it and whether it may be used to train or improve services. The answer may differ between product tiers and account settings, so get the relevant terms in writing.

Classify data before the pilot. Do not send confidential customer information, employee records, credentials or regulated data to a service until the organization has approved that use. Check deletion options, audit logs, access controls, encryption and the process for handling a data incident. Privacy claims should be checked against the actual contract and configuration, not inferred from a marketing label.

5. Can security teams control access and actions?

Review single sign-on, role-based access, administrative controls, audit events, retention settings and integration permissions. If the AI can call tools or change records, separate read from write permissions and define which actions require approval. A system that can summarize a database should not automatically inherit permission to edit every record in it.

Ask how the vendor handles prompt injection, malicious files, data leakage and unauthorized tool calls. No checklist guarantees safety, but clear boundaries, testing and a way to disable integrations reduce the impact of a failure.

6. How reliable is it under real conditions?

Test busy periods, long inputs, unavailable integrations, rate limits and partial service outages. Determine whether the tool returns a clear error, retries safely or silently produces an incomplete result. For workflows where wrong outputs have meaningful consequences, define an independent validation step rather than assuming confidence language means correctness.

7. What does the full cost include?

Compare more than the subscription price. Include usage charges, minimum commitments, setup, integration, data preparation, training, monitoring, human review and support. Model ordinary use and high-demand scenarios. Ask how costs change when the model, context size, number of users or volume of requests increases, and what happens when a usage limit is reached.

Use a cost-per-completed-task measure where possible. A cheap response is not necessarily a cheap outcome if staff must repeatedly correct it. Conversely, a smaller or less expensive model may be entirely sufficient for a well-bounded task.

8. Will it fit the workflow?

Identify where the tool sits in the existing process, which system remains the source of truth and how the user verifies or corrects its work. If the output must be copied manually between disconnected systems, the expected efficiency may disappear. Integration should include error handling, permissions, ownership and a way to trace the result back to its source.

9. How will the vendor communicate changes?

AI services can change when the underlying model, retrieval system, safety rules or interface is updated. Ask for release notes, advance notice for material changes, deprecation timelines and a way to test a new version before it affects a critical workflow. A contract should identify how the customer can respond if a change materially reduces quality or breaks a required control.

10. What support and incident process exists?

Confirm support hours, escalation channels, severity definitions, incident notification and access to relevant logs. If the tool fails or produces a harmful result, the organization should know who to contact and what information will be available for investigation. Clarify responsibilities between the vendor, integrator and customer instead of assuming another party will handle the issue.

11. Can the organization leave?

Ask how to export prompts, configuration, knowledge sources, logs and user-created content. Understand which data formats are portable, what deletion confirmation is provided and how much work would be needed to switch vendors. Avoid building a critical workflow around undocumented behavior that cannot be reproduced elsewhere. A realistic exit plan is part of resilience, not a sign that the buyer lacks confidence.

12. Who owns the result after launch?

Name a business owner, technical owner and risk contact. Define who approves new use cases, reviews performance, responds to incidents and decides whether the tool should be paused. Set a review date and success measures before deployment. If nobody owns the result after the pilot team moves on, even a promising tool can become unmanaged software.

A simple decision rule

Procurement teams can score each requirement as demonstrated, partially demonstrated, not demonstrated or not applicable. Critical gaps in privacy, security, reliability or legal permissions should not be hidden by a high average score. A tool with excellent writing quality but unclear data handling may be unsuitable for confidential work; a secure product that fails the core task is not a good purchase either.

The final recommendation should state the intended use, evidence from the pilot, unresolved risks, operating cost, required controls and exit conditions. That creates a decision record that can be revisited when the product or vendor changes.

Questions buyers should keep asking

Should every AI purchase go through the same review?

No. A low-risk assistant for public information can usually follow a lighter process than a system used for sensitive records or consequential decisions. Use a risk-based review, but keep minimum requirements for data handling, ownership and support.

Is a successful demonstration enough?

No. A demonstration is useful for understanding a product, but it is not representative evidence. A pilot should test real tasks, failure cases, integration and the amount of human work needed to get a reliable outcome.

What is the most overlooked cost?

Often it is the work around the model: review, correction, integration, training and exception handling. Include those costs when comparing tools.

AI procurement is strongest when it treats the system as part of an operating process, not a magic feature. The objective is not to find a vendor with the boldest promise. It is to select a tool whose value, limits and responsibilities can be explained before people depend on it.