An AI project can stall because of model quality, insufficient data, an incomplete integration or a process that does not justify the cost. A demo alone cannot establish the cause. If you already have a prototype, the useful question is what still needs to be demonstrated before the team can operate it.
This guide proposes a review before committing to more development. Its output should be a documented decision: deploy within limits, narrow the scope or stop. These are proposed checks, not client results or a certification.
Separate the business problem from the chosen system
Describe one task with an input, an output and an owner. Preparing a draft for someone to review is a different scope from sending replies automatically. Compare the prototype with the current process and a simpler alternative: rules, configuration of existing software or a focused integration.
If you are still choosing an approach, start with AI agents versus workflows. This review comes later: is the chosen system ready to operate? If decisions and exceptions disappear between teams, examine Company Brain operational memory before asking a model to reconstruct missing context.
What evidence to request when reviewing an AI pilot
I would use this record in a joint business and engineering review. Every row needs retrievable evidence, an owner and an outstanding decision. A checked box without a supporting test does not close the work.
| Area | Evidence I would request | When to hold expansion |
|---|---|---|
| Usefulness | The same task compared with the current process, including human correction. | The output is unused or review removes the improvement. |
| Quality | Representative cases, errors by type and the versions evaluated. | Only selected demo examples exist, or serious failures remain unresolved. |
| Data and permissions | Authorised sources and denied-access tests across users or tenants. | The system can retrieve information outside its permitted scope. |
| Integration | A verifiable final state after a timeout, retry or invalid response. | An uncertain execution is reported as complete or repeats an action. |
| Operations | A tested pause, manual handoff and recovery path with an owner. | Nobody can stop the workflow or handle the exception queue. |
| Cost | Total cost per accepted task and available review capacity. | The budget covers model calls alone. |
Evaluate errors by consequence, without a universal pass rate
An overall accuracy score can hide a serious failure in a rare category. I would separate routine cases, incomplete inputs, exceptions and attempts to access information without permission. Record abstentions, corrections and tasks that never complete too. Correct results on a small sample do not establish reliable behaviour across every scenario.
Agree on criteria before testing and hold back examples that are not used to tune the system. Compare versions against those same cases and expand the set as new failures emerge. If one model grades another, compare its judgments with human review: Google Cloud's evaluation guide describes that comparison. A model's stated confidence does not replace this evidence.
Hypothetical example: a classifier routes requests correctly but confuses urgent complaints with general enquiries. Before automating routing, I would keep that category under review and measure the delay this creates. The decision may be to automate only part of the workflow; there is no need to invent an accuracy threshold for every business.
Calculate the cost of an accepted task
Separate the initial investment—integration, data preparation and evaluation—from operation. For a defined period and volume, operating cost per accepted task includes models, tools, infrastructure, retries, review, correction and maintenance, divided by tasks meeting the agreed criteria. If no tasks are accepted, there is no useful unit cost to compare.
Value review time using the team's own figures. Check what happens during a workload spike too: a workflow that escalates many exceptions can exceed available capacity. The software quote comparison guide helps separate included work, dependencies and maintenance without claiming savings that have not been measured.
Check that the team can operate and stop the system
Before rollout, I would test provider failure, invalid output, revoked permission and repeated delivery of the same job. An operator must distinguish pending, failed, under review and completed states. Record execution ID, input references, versions, checks and final state, with defined access and retention; do not copy complete sensitive inputs into logs by default.
The Adslyfy case shows monitoring runs and screenshots. This illustrates operational visibility; it does not establish AI accuracy or results from this checklist. For connecting a capability to an existing product, the AI integration guide for SaaS develops the technical boundaries.
Decide whether to deploy, narrow scope or stop
- Deploy within limits: evidence meets the agreed criteria for a specific scope, an owner is assigned and returning to the previous process has been tested.
- Narrow scope: only one category or team meets the criteria. Keep the rest outside the rollout and define the evidence needed for expansion.
- Stop or rethink: there is no useful improvement over the alternative, necessary data is missing or unresolved failures prevent operation within the agreed boundaries.
Record the authorised scope, evidence, limitations, owner and trigger for revisiting the decision. Repeat evaluation when the model, prompt, sources or tools change; an initial review cannot guarantee future behaviour.
What to commission to unblock the project
If the missing piece is deciding what to fix first, I can review the system and prioritise the work through fractional CTO and technical direction. If the scope is already clear, my automation and applied AI service covers building the workflow and its controls.
Bring the current process, a demo, permitted examples of successful and failed outputs, connected systems and the person who will operate the result. Tell me where your pilot has stalled so we can define what the next stage needs to demonstrate.