Applied AI is most useful when it improves observable work. Starting with a model demonstration can produce an impressive prototype without a route into daily operations. Start instead with the decisions, queues and hand-offs that shape a real service.
Find the operational constraint
Look for work that is frequent, time-consuming, inconsistent or difficult to prioritise. Examples might include classifying incoming requests, extracting structured details from documents, routing cases, summarising a history for review or drafting a response for a person to approve. These are candidate patterns, not automatic recommendations.
Describe the workflow before selecting a technology:
- What triggers the work?
- Who performs it and who relies on the result?
- Which inputs are trusted, incomplete or sensitive?
- Where is judgement genuinely required?
- What is the consequence of a late, incorrect or unexplained result?
- Which existing system remains the source of truth?
This view may show that workflow automation, a clearer form or a better integration addresses much of the problem. Use AI for work that benefits from language, pattern interpretation or probabilistic assistance.
Establish a baseline before the pilot
Without a baseline, a team can measure model activity while learning little about business value. Record current demand, handling steps, waiting time, rework, escalation and error patterns. Use measures the operational owner already understands where possible.
A balanced baseline can include:
- Volume entering and leaving the workflow
- Time spent actively handling work and time spent waiting
- Proportion requiring clarification, rework or escalation
- Service quality checks used by the team today
- Cost drivers such as specialist review or duplicate entry
- User concerns, particularly where transparency or accessibility matters
Do not invent precision that the current process cannot support. Careful sampling may be more credible than a poorly defined dataset.
Separate deterministic automation from AI
Use deterministic rules where the answer must always follow a known policy. Validation, calculations, permissions and mandatory routing conditions should normally remain explicit and testable. Applied AI is better suited to work such as interpreting varied language, locating relevant passages or proposing a classification when uncertainty is acceptable and managed.
A combined workflow might validate required fields with rules, use an AI component to propose a category, apply a confidence policy and then ask a person to review higher-risk cases. The system should record the final decision and the reason for any override without storing more personal information than the service needs.
Check whether the information is ready
An AI feature cannot repair unclear ownership of knowledge. Identify the approved sources, who maintains them, how changes are reviewed and what the system should do when sources disagree. Remove obsolete or duplicated material before treating it as authoritative.
Review data protection and security early. Consider whether prompts or retrieved documents contain personal, commercially sensitive or regulated information. Define retention, access and regional processing requirements. Test for instruction injection and unsafe data disclosure where the system processes untrusted content.
Important readiness questions include:
- Can the team identify the source used for an answer or recommendation?
- Is access filtered using the requesting user’s permissions?
- Can sensitive fields be minimised or redacted?
- Is there a safe response when evidence is insufficient?
- Can the organisation correct or remove source material?
Design a pilot around a decision
A useful pilot tests an end-to-end slice with representative inputs. It needs an owner, a bounded user group, acceptance criteria and a decision: stop, adjust, expand or reconsider.
Build an evaluation set that includes routine cases, ambiguous cases, incomplete inputs and known failure patterns. Keep some examples outside day-to-day tuning so the team can check whether changes genuinely generalise. Review outputs with the people accountable for the service, not only the delivery team.
The pilot should compare assisted work with the baseline. Possible measures include handling effort, correct routing, reviewer agreement, escalation behaviour and the time taken to recover from a poor suggestion. Model latency and consumption cost matter, but they do not by themselves demonstrate an improved service.
Keep people in control where risk requires it
Human review is not a checkbox. Decide what the reviewer sees, how uncertainty is presented, whether sources are available and how an incorrect suggestion can be rejected.
Define actions the system must never take without explicit authority. For consequential decisions, ensure the accountable person can understand the relevant evidence and apply policy outside the model. Provide a route for users to challenge or correct results.
Prepare the production controls
Moving from pilot to production introduces concerns that a demonstration can hide. Version prompts, evaluation sets, model settings and source indexes. Monitor input changes, response failures, policy violations, latency and cost. Use controlled releases and maintain a fallback for temporary model or dependency failure.
Assign ownership for:
- Evaluation and release approval
- Knowledge and data quality
- Security and privacy review
- Operational monitoring and incident response
- Supplier or model changes
- User feedback and correction
Re-evaluate when the workflow, source material or model changes. An approval based on one configuration should not silently apply to a materially different system.
Opportunity assessment checklist
Before funding a larger implementation, confirm that:
- A named operational problem and owner exist.
- The current workflow and baseline are understood.
- Deterministic automation has been considered first.
- Data sources, permissions and retention are defined.
- Representative evaluation cases include failure conditions.
- Human review is designed around risk rather than appearance.
- Success and stop criteria are agreed before testing.
- Production monitoring, fallback and change ownership are feasible.
Applied AI can create worthwhile operational leverage, but results depend on context, data and adoption. A restrained pilot may show that a simpler intervention is preferable. That is still a useful outcome: the organisation has reduced uncertainty before making a broader commitment.