Route model calls through a relay server

Why a relay server

In the Editor AI SDK Playground you never choose a model or enter an API key, yet the document is edited by an LLM. Something between the browser and the model holds that key and talks to the provider on behalf of the host application, the application that integrates Editor AI SDK. That something is a relay server, and every Editor AI SDK integration that leaves a developer's workstation needs one.

The quickstart implements createMessage() in direct mode: the host application holds a test API key in the browser and calls the provider API itself. That is convenient on a personal workstation, because it needs no backend, but the key is visible to the page and to browser developer tools. Relay mode changes exactly one point. The key and the provider call move to a relay server in your backend, and createMessage() no longer calls the provider API directly; it calls the relay instead. The tool loop, tool validation, user approval, and tool.execute() stay as they are.

	Direct mode	Relay mode
Who calls the provider API	Host application (browser)	Relay server
Where the API key lives	The page (input field and JavaScript)	Relay server environment
Target of createMessage()	Provider API	Relay endpoint
Policy (model allowlist, authentication, quotas)	Nowhere to apply	Relay server
Use	Local testing	Shared and production deployments

This guide explains what a relay server is, which part of the host application's work it takes over, how one request travels through it, and how to run a small Node.js relay that you can grow into your own backend. The field-by-field contract is in Relay Server Protocol.

What a relay server is

A relay server is a small HTTP service that runs in your application backend. The host application sends it the request it would otherwise have sent to the model provider. The relay attaches the credential it holds, forwards the request to the provider, and returns the provider's response without changing it.

That job gives a relay three characteristics that keep it simple:

• The backend relay manages the API key, so the browser never has one. The provider key lives in the relay's environment. The browser never receives it, so nothing in JavaScript, developer tools, or a saved page can leak it.
• It decides the policy. Every model call passes through the relay, so it is the one place to say which providers and models are allowed, who may call, and how much.
• It keeps no conversation state. Each request carries everything the provider needs. The relay stores no history and executes no tools, so it scales like any stateless HTTP service, and one relay can serve many applications.

Responsibilities of the host application and the relay

Editor AI SDK tools act on the document that is open in the Thinkfree Office editor, so tool.execute() cannot move to a server: only the browser can reach that document. The relay therefore takes over exactly one thing, the provider call, and leaves the rest of the tool loop in the host application.

Concern	Host application (browser)	Relay server (your backend)	LLM provider
Conversation history and the current prompt	Owns and sends it	Forwards it	Reads it
Tool catalog from getTools()	Selects and sends the schemas	Forwards them	Chooses tool calls
tool.execute() and user approval	Runs here	Never involved	Never involved
Provider key	Never present	Holds it	Verifies it
Provider and model policy, caller authentication, quotas	Not possible here	Enforces them	Not involved
Response	Consumes content and stop_reason	Passes it through unchanged	Produces it

The Playground's default connection, labeled Thinkfree Relay Server, is this pattern running against Thinkfree's own relay: the page holds no key, and the relay decides which model is used. The direct-mode options in the same menu exist for local testing only.

Request flow

For every model call the host application builds two things and sends them together in one JSON body:

• tfAgentRelay describes what this turn means: request and turn identifiers, the selected provider and model, the system prompt, the transcript, the current input, and the tool catalog.
• tfAgentTransport describes which HTTP request to execute: the provider adapter, the model, whether to stream, and the provider-ready request body - the same payload (JSON) the host application sends when it calls the provider API in direct mode.

The envelope has two halves because the relay has two different jobs. To enforce policy it has to know which provider and model a request is aimed at; to forward the request it needs a body the provider accepts as is. tfAgentRelay serves the first job and tfAgentTransport the second. As a result the relay can enforce policy without interpreting each provider's request format.

The relay processes a request in this order. First it compares the provider and model named in tfAgentTransport with the values in tfAgentRelay and rejects the request as tampered when they differ; this check stops a request that appears to ask for one model while actually calling another. Next it applies the allowlist to settle the provider and model, attaches the credential, and posts tfAgentTransport.body to the provider. Finally it returns the provider's response to the host application unchanged: as one JSON body when streaming was not requested, or chunk by chunk as it arrives when it was. Because the response format does not change, createMessage() receives the same { content, stop_reason } shape it handled in direct mode.

1. The user asks for a change in the chat UI.
2. The host application posts the envelope to POST /ai-agent/relay.
3. The relay validates the envelope, resolves the provider and model against its allowlist, and attaches the provider key.
4. The relay posts the provider-ready body to the provider.
5. The provider answers, here with a tool call.
6. The relay returns the provider response unchanged.
7. The host application validates the call and asks the user to approve it.
8. The host application runs tool.execute() through Editor SDK, and the document changes on the same page.
9. The tool result goes back into the conversation.
10. The host application posts the next request. The loop ends when the model returns no tool call, which the Claude examples on this page read from stop_reason.

Steps 7 to 9 never touch the relay. That is the whole design: the secret moved to the server; the document work did not.

Node.js relay skeleton

The server below is complete enough to run the quickstart loop against your own key and small enough to read in one sitting. It has no dependencies. Install Node.js 22 or later and save the file as relay-server.mjs.

// relay-server.mjs - a minimal relay for Editor AI SDK. Node.js 22 or later, no dependencies.
// The browser keeps the conversation and executes tools. This server only holds the provider
// key, enforces which vendor and model may be used, and forwards one request at a time.
import http from "node:http";

const PORT = Number(process.env.RELAY_SERVER_PORT ?? 8787);
const RELAY_PATH = "/ai-agent/relay";
const ORIGIN = process.env.RELAY_SERVER_CORS_ORIGIN ?? "http://localhost:3000";
const BODY_LIMIT = 2 * 1024 * 1024;

// Policy: one entry per vendor you operate. `models` is the allowlist; the first one is the default.
const list = (value) => (value ?? "").split(",").map((s) => s.trim()).filter(Boolean);
const VENDORS = {
  claude: {
    aiProvider: "claude", apiKey: process.env.CLAUDE_API_KEY, models: list(process.env.CLAUDE_ALLOWED_MODELS),
    request: (model, stream, body, key) => ({
      url: process.env.CLAUDE_ENDPOINT ?? "https://api.anthropic.com/v1/messages",
      headers: { "x-api-key": key, "anthropic-version": "2023-06-01" },
      body: { ...body, model },
    }),
  },
  google: {
    aiProvider: "google", apiKey: process.env.GEMINI_API_KEY, models: list(process.env.GEMINI_ALLOWED_MODELS),
    request: (model, stream, body, key) => ({
      url: `${process.env.GEMINI_ENDPOINT ?? "https://generativelanguage.googleapis.com/v1beta/models/"}`
        + `${encodeURIComponent(model)}:${stream ? "streamGenerateContent?alt=sse&" : "generateContent?"}key=${encodeURIComponent(key)}`,
      headers: {},
      body,
    }),
  },
  openai: {
    aiProvider: "openai", apiKey: process.env.OPENAI_API_KEY, models: list(process.env.OPENAI_ALLOWED_MODELS),
    request: (model, stream, body, key) => ({
      url: process.env.OPENAI_ENDPOINT ?? "https://api.openai.com/v1/chat/completions",
      headers: { authorization: `Bearer ${key}` },
      body: { ...body, model },
    }),
  },
};

class RelayError extends Error {
  constructor(status, code, message, details = null) { super(message); Object.assign(this, { status, code, details }); }
}
const fail = (status, code, message, details) => { throw new RelayError(status, code, message, details); };
const isObject = (v) => v !== null && typeof v === "object" && !Array.isArray(v);

// 1. Validate the envelope: both halves must be present and must describe the same request.
function validate(body) {
  if (!isObject(body)) fail(400, "relay-request-invalid", "Relay request body must be a JSON object.");
  const relay = body.tfAgentRelay, transport = body.tfAgentTransport;
  if (!isObject(relay)) fail(400, "relay-request-invalid", "Relay request body.tfAgentRelay must be a JSON object.");
  if (!isObject(transport)) fail(400, "relay-request-invalid", "Relay request body.tfAgentTransport must be a JSON object.");
  if (!isObject(transport.body)) fail(400, "relay-request-invalid", "Relay request body.tfAgentTransport.body must be a JSON object.");
  if (typeof transport.adapterId !== "string" || !transport.adapterId.trim()) fail(400, "relay-transport-adapter-missing", "Relay transport adapterId is required.");
  if (typeof transport.aiProvider !== "string" || !transport.aiProvider.trim()) fail(400, "relay-transport-provider-missing", "Relay transport aiProvider is required.");
  if (typeof transport.stream !== "boolean") fail(400, "relay-transport-stream-invalid", "Relay transport stream must be a boolean.");
  const semantic = isObject(relay.aiProviderConfig) ? relay.aiProviderConfig : {};
  if (semantic.aiProvider && semantic.aiProvider !== transport.aiProvider) fail(400, "relay-provider-mismatch", "Relay semantic aiProvider and transport aiProvider must match.");
  if (semantic.model && transport.model && semantic.model !== transport.model) fail(400, "relay-model-mismatch", "Relay semantic model and transport model must match.");
  return { relay, transport };
}

// 2. Apply the policy: pick the vendor route and a model that the route allows.
function resolve(relay, transport) {
  const requested = (transport.vendor ?? relay.relayVendor ?? "").toLowerCase();
  const route = requested
    ? VENDORS[requested]
    : Object.values(VENDORS).find((v) => v.aiProvider === transport.aiProvider && v.models.length);
  if (!route || route.aiProvider !== transport.aiProvider) fail(403, "relay-vendor-not-allowed", `Relay vendor "${requested || transport.aiProvider}" is not allowed.`);
  if (!route.apiKey) fail(500, "provider-config-missing", `API key for "${transport.aiProvider}" is not configured on the relay server.`);
  const model = transport.model?.trim() || route.models[0];
  if (!route.models.includes(model)) fail(403, "relay-model-not-allowed", `Relay model "${model}" is not allowed.`, { model });
  return route.request(model, transport.stream, transport.body, route.apiKey);
}

async function readJson(req) {
  const chunks = []; let size = 0;
  for await (const chunk of req) {
    if ((size += chunk.length) > BODY_LIMIT) fail(413, "relay-request-too-large", "Relay request body exceeded the configured size limit.");
    chunks.push(chunk);
  }
  const text = Buffer.concat(chunks).toString("utf8").trim();
  if (!text) fail(400, "relay-request-empty", "Relay request body is required.");
  try { return JSON.parse(text); } catch { fail(400, "relay-request-json-invalid", "Relay request body must be valid JSON."); }
}

const json = (res, status, payload) => { res.writeHead(status, { "content-type": "application/json; charset=utf-8" }); res.end(JSON.stringify(payload) + "\n"); };
const error = (res, e, ctx = {}) => json(res, e.status ?? 500, {
  error: { code: e.code ?? "relay-internal-error", message: e.message, details: e.details ?? null },
  relay: { requestId: ctx.requestId ?? null, turnId: ctx.turnId ?? null, upstreamStatus: e.details?.upstreamStatus ?? null },
});

// 3. Forward and pass the provider response through unchanged - JSON as one body, SSE chunk by chunk.
async function proxy(upstream, res) {
  if (!upstream.ok) fail(502, "upstream-request-failed", `Upstream provider request failed (${upstream.status}).`, { upstreamStatus: upstream.status });
  const type = upstream.headers.get("content-type") ?? "application/json";
  res.writeHead(200, type.includes("text/event-stream")
    ? { "content-type": "text/event-stream; charset=utf-8", "cache-control": "no-cache, no-transform", connection: "keep-alive" }
    : { "content-type": type });
  for await (const chunk of upstream.body) res.write(chunk);
  res.end();
}

http.createServer(async (req, res) => {
  const origin = req.headers.origin;
  if (origin && origin !== ORIGIN) return error(res, new RelayError(403, "cors-origin-not-allowed", "Request Origin is not allowed by the Relay Server CORS policy."));
  res.setHeader("access-control-allow-origin", ORIGIN);
  res.setHeader("access-control-allow-methods", "POST, GET, OPTIONS");
  res.setHeader("access-control-allow-headers", req.headers["access-control-request-headers"] ?? "*");
  if (req.method === "OPTIONS") return res.writeHead(204).end();
  const path = new URL(req.url, "http://relay").pathname;
  if (req.method === "GET" && path === "/health") return json(res, 200, { ok: true, relayPath: RELAY_PATH });
  if (path !== RELAY_PATH) return error(res, new RelayError(404, "route-not-found", "Relay server route was not found.", { path }));
  if (req.method !== "POST") return error(res, new RelayError(405, "relay-method-unsupported", "Relay endpoint only accepts POST.", { method: req.method }));

  let ctx = {};
  try {
    const body = await readJson(req);
    ctx = { requestId: body?.tfAgentRelay?.requestId, turnId: body?.tfAgentRelay?.turnId };
    const { relay, transport } = validate(body);
    const target = resolve(relay, transport);
    const controller = new AbortController();
    res.on("close", () => controller.abort());
    const upstream = await fetch(target.url, {
      method: "POST", signal: controller.signal,
      headers: { "content-type": "application/json", ...target.headers },
      body: JSON.stringify(target.body),
    });
    await proxy(upstream, res);
    console.log(JSON.stringify({ event: "relay.completed", ...ctx, aiProvider: transport.aiProvider, model: target.body.model ?? transport.model, upstreamStatus: upstream.status }));
  } catch (e) {
    console.error(JSON.stringify({ event: "relay.failed", ...ctx, code: e.code ?? "relay-internal-error", message: e.message }));
    if (!res.headersSent) error(res, e, ctx); else res.end();
  }
}).listen(PORT, "127.0.0.1", () => console.log(`Relay: http://localhost:${PORT}${RELAY_PATH}`));

Read it top to bottom and you have read the whole relay pattern: the VENDORS object defines the policy, validate() checks the contract, resolve() applies the allowlist and selects the key, and proxy() streams the answer back. Everything else is plain HTTP plumbing.

VENDORS in the code is the list of provider routes configured on the relay. The contract page calls such a route a vendor: the name that tells apart several routes for the same provider family when they differ in credential or allowed models. With one route per provider, vendor and provider mean the same thing.

Set the variables for one provider in the server's environment and start it. The allowlist is the policy: a request for a model outside it is refused before any provider call is made. The commands below assume the bash shell on macOS or Linux.

export CLAUDE_API_KEY=sk-ant-your-test-key
export CLAUDE_ALLOWED_MODELS=claude-sonnet-5
node relay-server.mjs

On Windows PowerShell, set the same variables with the $env: syntax.

$env:CLAUDE_API_KEY = "sk-ant-your-test-key"
$env:CLAUDE_ALLOWED_MODELS = "claude-sonnet-5"
node relay-server.mjs

When the server starts, it prints the relay address to the console.

Relay: http://localhost:8787/ai-agent/relay

The request below checks the contract validation without calling a model.

curl -i http://localhost:8787/ai-agent/relay \n  -H 'Content-Type: application/json' \n  --data '{"tfAgentRelay":{},"tfAgentTransport":{}}'

The relay answers 400 with relay-request-invalid. The JSON body is formatted here for readability.

HTTP/1.1 400 Bad Request
content-type: application/json; charset=utf-8

{
  "error": {
    "code": "relay-request-invalid",
    "message": "Relay request body.tfAgentTransport.body must be a JSON object.",
    "details": null
  },
  "relay": { "requestId": null, "turnId": null, "upstreamStatus": null }
}

Switching the chat-ui to the relay

Replace only createMessage() in the quickstart chat-ui sample with the function below and remove the API key and model ID inputs from the page. The rest of the sample, getTools(), the tool loop, validation, approval, and tool.execute(), stays exactly as it is, and so does the expected result: the approved text appears in the document.

// Relay mode: the browser posts the envelope to your relay - no provider key in the page.
const RELAY_URL = "http://localhost:8787/ai-agent/relay";   // same-origin deployment: "/ai-agent/relay"
const MODEL = "claude-sonnet-5";                             // must be in the relay's allowlist
const sessionId = crypto.randomUUID();

async function createMessage({ system, messages, tools }) {
  const requestId = crypto.randomUUID();
  const res = await fetch(RELAY_URL, {
    method: "POST",
    headers: { "content-type": "application/json" },
    signal: AbortSignal.timeout(65000),
    body: JSON.stringify({
      tfAgentRelay: {
        requestId, turnId: requestId, agentId: "chat-ui", sessionId,
        aiProviderConfig: { connectionMode: "relay", aiProvider: "claude", model: MODEL },
      },
      tfAgentTransport: {
        adapterId: "claude-messages", aiProvider: "claude", model: MODEL, stream: false,
        body: { max_tokens: 8192, system, messages, tools },   // the same body direct mode sent to the provider
      },
    }),
  });
  if (!res.ok) {
    const { error } = await res.json().catch(() => ({ error: {} }));
    throw new Error(`Relay request failed (HTTP ${res.status}${error?.code ? `, ${error.code}` : ""}).`);
  }
  const data = await res.json();
  return { content: data.content, stop_reason: data.stop_reason };   // same shape as direct mode
}

This tfAgentRelay carries only what a relay checks: the identifiers it echoes in errors and logs, and the provider and model pair it compares with tfAgentTransport. The Playground sends the full snapshot, including the transcript and tool catalog, so that a relay can log and apply policy on meaning rather than on provider payloads. Both forms are valid; the fields are listed in Relay Server Protocol.

What production adds

The skeleton keeps the key off the page and enforces one policy, the model allowlist. Everything a shared or production deployment adds is also a relay concern, because the relay is the only place the request passes through before it becomes a bill.

• Authenticate the caller. Send your application's session token in a request header from createMessage() and reject requests without it before reading the body. CORS does not replace authentication.
• Authorize the document and tenant. Put the identifiers in a top-level field next to the two reserved keys and check them against the signed-in user.
• Limit cost. Enforce request-size, rate, and token limits per tenant, and pin the model on the server instead of trusting the requested one.
• Log without leaking. Record requestId, turnId, provider, model, status, and latency. Never log provider authorization headers or complete document content.
• Use HTTPS for both the application and the relay, and allow only the origins you operate.

For a local 403 with cors-origin-not-allowed, check that the page address matches RELAY_SERVER_CORS_ORIGIN; localhost and 127.0.0.1 are different origins. For 403 with relay-model-not-allowed, add the model to the allowlist or change MODEL. For 502, read relay.upstreamStatus in the error body: 401 means the key, 429 means the provider's rate limit. For a refused connection, check that the relay is running on the expected port.