An Albany business can quickly collect a long list of AI products without finding one that fits its work. The problem is often that the shortlist mixes different tasks: drafting a reply, searching business information and sending a message are treated as the same capability. They involve different inputs, checks and authority.
Use one recurring task to organise the selection. The examples below concern preparing a draft response to a routine customer enquiry. They are suggested evaluation exercises, not reports of a real business trial or instructions to upload customer records.
Problem: the task begins with a product demonstration
A demonstration can make a tool’s strongest feature look like your biggest need. The response is to write down where your own work becomes difficult before choosing products. For an enquiry reply, that might be locating the approved returns wording rather than composing friendly sentences.
The business.gov.au AI guide recommends identifying the problems to solve and considering tools already available. Build the shortlist around that problem, with a non-AI process improvement retained as a possible answer.
Problem: each candidate receives an easier example
If one product gets a simple delivery question and another gets several conflicting requests, the results are not comparable. Prepare the same fictional enquiry and approved reference information for each candidate. Make the invented customer and order details explicit so nobody mistakes them for real records.
Include a question the reference material cannot answer. Watch whether the product admits the gap, asks for clarification or produces an unsupported promise.
Also prepare a second version in which one approved business detail changes. For example, replace an invented collection arrangement with another invented arrangement and label the current version clearly. Check whether the candidate uses the new reference or repeats its earlier answer. This explores the actual maintenance problem: staff need a way to update operational facts, not simply a convincing response to the first demonstration.
Problem: polished wording hides an incorrect commitment
A reply can sound helpful while inventing a refund, delivery date or stock position. The solution is a review sheet that checks factual statements individually. Separate tone from correctness and distinguish an offered possibility from a commitment the business has authorised.
For each proposed reply, identify which reference supports its claims. A product that repeatedly needs substantive correction may add work even when its first draft arrives quickly.
Problem: a useful trial seems to require private information
Stop and redesign the trial rather than pasting a real inbox into a public tool. The OAIC’s commercial AI product guidance discusses due diligence and recommends, as best practice, avoiding personal and sensitive information in publicly available generative AI tools.
Use invented examples that preserve the task’s difficulty without representing an actual customer. Separately assess any proposed operational use involving personal information, including applicable privacy obligations and the provider’s data practices. A successful fictional test does not resolve that assessment.
Problem: drafting and acting are bundled together
A tool that prepares text is different from an agent permitted to send it, amend a record or authorise an adjustment. ASD’s small-business AI security guidance highlights the risks of excessive agent access and retaining oversight of important actions.
Keep the proposed workflow explicit: a staff member retrieves the appropriate reference, the tool prepares a draft, and an authorised person checks it. Evaluate any additional connection or action separately rather than enabling everything to make the demonstration smoother.
Problem: the product handles the easy case but not the handover
An unanswered enquiry still needs an owner. Ask how the proposed tool flags uncertainty, preserves the original question and hands work to a person. A generated answer that cannot be checked against the enquiry is difficult to review.
Test what happens when a draft is rejected. Staff should be able to complete the task through the ordinary process, without reconstructing the customer’s question from a summary that may have omitted details.
Problem: time saved excludes time spent checking
Measure the whole draft-to-approved-response process. Include finding the reference, preparing the input, correcting the output and recording the final decision. Compare it with completing the same fictional task through your current method.
Keep a short evaluation record:
- The precise task and permitted information.
- The reference used to verify the proposed answer.
- Unsupported claims or missing questions.
- Staff effort required to reach an acceptable draft.
- The unresolved requirements for any real deployment.
Problem: the shortlist grows without a decision
Remove candidates that cannot meet a necessary condition, such as suitable data handling or a workable human review. For the remainder, name the next unanswered question and who can resolve it. Do not keep adding products when the real obstacle is an unclear business process.
The result may be a limited drafting aid, a different tool or no purchase. A useful shortlist makes that decision explainable in terms of the actual work, rather than the number of AI features advertised.