LLM Tool Calling for Web Apps

Most tool-calling examples run on a server. The model asks for get_weather("Berlin"), your backend calls an API, and the result goes back. Straightforward.

Tool calling in a web app is a different shape, and the difference is what makes it interesting.

The tools are already there

In a browser app, the useful "tools" are the functions your UI already calls: addToCart, applyFilter, openRow, submitForm, navigate. They have access to component state, the user's authenticated session, and the DOM. A server-side tool has none of that.

So the interesting question is not "how do I write tools for the model" — it is "how do I expose the functions I already have, safely."

The shape of a client-side tool

ai.register({ applyFilter: { description: "Filter the current table by status and date range.", params: { status: { type: "string" }, from: { type: "string" }, to: { type: "string" }, }, handler: async ({ status, from, to }) => { setFilters({ status, from, to }); return { matched: rows.length }; }, }, });

Three parts, and each one earns its keep:

• description is the only thing the model uses to decide whether this tool fits. Write it for a competent stranger. "Filter the current table by status and date range" beats "filters" by a wide margin.
• params are typed, so the model gets a schema instead of guessing at argument shapes.
• handler is ordinary application code. It closes over your component state and runs with the user's session.

Why return values matter more than you think

A tool that returns nothing forces the model to narrate blind. A tool that returns a fact lets it be specific:

handler: async ({ sku }) => { const stock = await checkStock(sku); return { inStock: stock > 0, remaining: stock }; }

The model now says "yes, three left" rather than "I have checked that for you." Return values are how the conversation stays grounded in what actually happened.

Context: the part server-side tools do not need

A server tool is stateless. A browser tool almost never is — "archive that one" depends entirely on what is on screen. So the current UI has to travel with each turn:

ai.bindState(() => ({ route: location.pathname, visibleRows: rows.slice(0, 20).map((r) => ({ id: r.id, name: r.name })), selection: selectedIds, }));

The getter runs per turn, so the model reasons over live state rather than whatever was true when the session opened.

Keep it tight. Every extra field costs tokens and dilutes the signal — a focused twenty-row summary outperforms a full state dump.

Scoping: not every tool everywhere

Exposing every function on every page makes the model's job harder and your app riskier. Scope them:

ai.register({ refundOrder: { description: "Refund an order in full.", params: { orderId: { type: "string", required: true } }, scope: "/admin/orders", dangerous: true, handler: async ({ orderId }) => api.refund(orderId), }, });

scope limits where a tool is callable at all. dangerous forces a confirmation. Middleware via ai.use() handles the rest — auth checks, rate limits, audit logging.

The security model

A browser cannot hold a provider API key. The publishable key here (vox_pub_…) is not one: it is domain-locked, checked against an origin whitelist at the handshake, and proxied server-side so the raw model credentials never enter your bundle. Details in security and CORS.

That still leaves the obvious rule: the model is an untrusted caller. Handlers hit the same authenticated endpoints your buttons do, and your server enforces permissions exactly as before. Tool calling changes who initiates the request, not who is allowed to make it.

Where this goes

Once tools are registered and state is bound, "make my app voice-controlled" is mostly a UI question — see how to make your app voice-controlled — and the same tool definitions serve the text chat, the voice agent, and anything else that speaks the same protocol.

Reference documentation: actions and capabilities.