How to Make Your App Voice-Controlled

There are two ways to make an app voice-controlled, and only one of them survives contact with a real product.

The old way: intents and dialog trees

You enumerate what users might say, group the phrasings into intents, map each intent to a handler, then maintain a decision tree for the follow-up questions. It works in a demo. Then:

• Someone phrases it a way you did not anticipate, and nothing matches.
• You ship a feature, and the tree does not know about it.
• Two intents overlap, and you spend an afternoon tuning confidence thresholds.
• A conversation needs context from three turns ago, and the tree has no memory of it.

The maintenance cost grows with your product, forever.

The capability-first way

Instead of enumerating what users might say, you describe what your app can do:

ai.register({ filterResults: { description: "Filter the results list by one or more criteria.", params: { minPrice: { type: "number" }, maxPrice: { type: "number" }, inStockOnly: { type: "boolean" }, }, handler: async (args) => { setFilters(args); return { matched: applyFilters(args).length }; }, }, });

The model reads the description and the parameter types, then decides — per utterance — whether this is the right function and what to put in each argument. "Show me the ones under fifty quid that are actually in stock" resolves to filterResults({ maxPrice: 50, inStockOnly: true }) without you having written that phrasing anywhere.

Add a function and it is immediately usable. Delete one and it is immediately gone. There is no parallel structure to keep in sync, because the capability list is the structure.

Give it eyes

A voice-controlled app that cannot see its own UI is guessing. Bind the state:

ai.bindState(() => ({ route: location.pathname, selectedRows: table.getSelectedIds(), filters: currentFilters, results: visibleResults.slice(0, 20), }));

Now "delete the three I selected" and "sort these by date" are answerable, because the agent knows what "these" refers to.

Keep the snapshot small and relevant. It is sent on every turn, so dumping your entire Redux store makes conversations slower and less accurate, not more.

Close the loop with return values

Handlers should return something useful:

handler: async ({ id }) => { const item = await api.archive(id); return { status: "archived", remaining: await api.countActive() }; }

Whatever you return goes back to the model, so it can say "done — you have four left" instead of a generic acknowledgement. This is the difference between an assistant that reports outcomes and one that just fires and hopes.

Put guardrails on the sharp edges

ai.register({ deleteWorkspace: { description: "Permanently delete a workspace and all its data.", params: { id: { type: "string", required: true } }, dangerous: true, handler: async ({ id }) => api.deleteWorkspace(id), }, });

dangerous: true forces a confirmation before the handler runs. For anything beyond that, middleware can inspect, log, rate-limit or block a call outright:

ai.use(async (ctx, next, cancel) => { if (ctx.action === "deleteWorkspace" && !user.isOwner) { return cancel("Only the workspace owner can do that."); } await next(); });

Treat the agent as an untrusted client, because it is one. Your server-side authorisation still applies — middleware is a fast, friendly first line, not a replacement for it.

What to register first

Start with the actions where clicking is genuinely slower than speaking:

• Multi-field forms and filters
• Navigation across a deep route tree
• Bulk operations on a selection
• Anything a user does while their hands are busy

Skip things that are already one click away. Voice is a shortcut, not a replacement for your UI.

Deeper detail on each piece: actions and capabilities, state awareness, and the quickstart.