A provider is the object the agent talks to the model through. You build one and pass it as provider to f.agent(); the agent calls it for the understand call and the speak call of every turn. All built-in providers implement the same AiProvider interface, so changing vendors is one constructor. There is no vendor SDK behind them: each is a thin binding over @providerkit/core, which speaks every vendor's REST API over fetch.
Every provider carries capabilities: ProviderCapabilities, five flags that describe what the implementation does. From each class in src/providers/.
Flag
Gemini
OpenAI
Anthropic
OpenRouter
DeepSeek
Z.ai
createOpenAICompatibleProvider default
supportsTools
yes
yes
yes
yes
yes
yes
yes
supportsNativeJsonSchema
yes
yes
no
yes
no
no
yes
supportsStreaming
yes
yes
yes
yes
yes
yes
yes
supportsStreamingToolCalls
yes
yes
yes
yes
yes
yes
yes
supportsPromptCaching
yes
yes
yes
yes
yes
yes
no
Anthropic has no native schema mode, so the schema is sent as an extra system block after the cached one; Z.ai is Anthropic-compatible and reports the same flag, and DeepSeek serves JSON mode but not a schema. In all three the schema reaches the model as prompt, which is why Gemini's own limit matters: on Gemini 2 a response schema and tools cannot ride the same call (400 "Function calling with a response mime type: 'application/json' is unsupported"), so those calls send the schema as prompt too. Gemini 3 takes both. FallbackAiProvider reports a flag as true only when every provider in its list does. createOpenAICompatibleProvider takes capabilities overrides merged over its defaults.
Options every vendor provider takes
The six vendor classes share these fields. Where a provider adds or renames one, its own section says so.
Field
Type
Default
Meaning
apiKey
string
required
The vendor key. An empty string throws at construction.
model
string
required (Z.ai: glm-5.3-flash)
The model id.
backupModels
string[]
[]
Tried in order on the same provider when a call fails with a kind that is backup-eligible in @providerkit/core, or with model.
fallbacks
Array<AiProvider | Provider>
none
Other providers tried after this one fails or is on cooldown. Takes this package's providers and @providerkit/core providers alike.
fallbackOptions
FallbackOptions<Provider>
none
Cooldowns per error kind and an onCooldown callback for fallbacks. Gemini, Anthropic and Z.ai only.
config
RequestConfig
none
Sampling defaults sent with every call: temperature, topP, maxTokens, stopSequences, effort.
retryConfig
{ timeout?: number; retries?: number }
{ timeout: 60000, retries: 3 }
Idle-stream deadline in ms and retries after the first attempt. See Retries.
fetchImpl
typeof fetch
global fetch
A replacement fetch, for tests that script the wire.
No native schema mode: a structured request carries the schema as a system block placed after the cached one, so a per-call schema does not invalidate the system prompt's cache.
OpenRouterProvider
ts
interface OpenRouterProviderOptions {
apiKey: string;
model: string;
backupModels?: string[];
fallbacks?: Array<AiProvider | Provider>;
/** Sent as `HTTP-Referer`, for OpenRouter's rankings. */
siteUrl?: string;
/** Sent as `X-Title`, for OpenRouter's rankings. */
siteName?: string;
/** Preferred upstream hosts, in order; keeps the prompt cache on one host. */
providerOrder?: string[];
jsonWithTools?: "response_format" | "prompt";
config?: RequestConfig;
retryConfig?: { timeout?: number; retries?: number };
fetchImpl?: typeof fetch;
}
ts
import { OpenRouterProvider } from "@falai/agent";
const openrouter = new OpenRouterProvider({
apiKey: process.env.OPENROUTER_API_KEY ?? "",
model: "anthropic/claude-sonnet-5",
providerOrder: ["anthropic"],
});
Base URL https://openrouter.ai/api, chat completions with json_schema.
The upstream host is pinned for you. OpenRouter's prompt cache lives on the upstream host's account, and default routing hops between hosts, so every hop is a cold cache. By default the model's own vendor is preferred — z-ai/glm-5.3-flash goes to z-ai, anthropic/claude-sonnet-5 to anthropic — with fallbacks on, so it is a preference and never a failed call. Measured 2026-09-21: unpinned, four calls with the same 2.9k-token prefix all landed on a host that reported cached: 0; pinned, the second call read 2,880 of 2,904 tokens from cache.
Pass providerOrder to choose the hosts yourself. There is no way to say it inside model: OpenRouter answers 400 "z-ai/glm-5.3-flash@novita is not a valid model ID".
The route changes the answers, so measure your own model here. Through this gateway, z-ai/glm-5.3-flash put "somos umas 30 pessoas" in the wrong band of a four-value enum on most attempts, in every JSON mode and routing tried, while the same model on ZaiProvider and deepseek-chat got it right every time. Over the 40-case understand eval on 2026-09-21 the same split holds: routing agreement 93% either way, but field agreement 86% (12/14) through OpenRouter against 100% (14/14) on ZaiProvider, and a median call three to five times slower. Prefer a model's own endpoint where you have one; run bun run eval:understand and bun run eval:live --only openrouter against the model you ship.
import { DeepSeekProvider } from "@falai/agent";
const deepseek = new DeepSeekProvider({ apiKey: process.env.DEEPSEEK_API_KEY ?? "", model: "deepseek-chat" });
Chat completions with json_object: DeepSeek answers a json_schema response format with 400 "This response_format type is unavailable now", so the endpoint guarantees JSON and the schema travels in the prompt. Reasoning arrives on reasoning_content and cache hits under prompt_cache_hit_tokens; both are read one layer down.
ZaiProvider
ts
interface ZaiProviderOptions {
apiKey: string;
/** Default "glm-5.3-flash". Bare ids: "glm-5.3-flash", not "z-ai/glm-5.3-flash". */
model?: string;
backupModels?: string[];
fallbacks?: Array<AiProvider | Provider>;
fallbackOptions?: FallbackOptions<Provider>;
/** Any endpoint speaking the Anthropic Messages dialect with the plan's auth. Default: the coding endpoint. */
baseUrl?: string;
config?: RequestConfig;
retryConfig?: { timeout?: number; retries?: number };
fetchImpl?: typeof fetch;
}
ts
import { ZaiProvider } from "@falai/agent";
const zai = new ZaiProvider({ apiKey: process.env.ZAI_API_KEY ?? "" }); // model defaults to glm-5.3-flash
The flat-rate Z.ai Coding Plan hosts GLM on an Anthropic-compatible endpoint. Model ids are bare, and thinking is off unless you ask for it: the endpoint reads an absent thinking field as thinking on, so @providerkit/core says "no" out loud for you. Measured 2026-09-21: a call with no config.effort sends thinking: { type: "disabled" }, the same body effort: "none" produces. Ask for thinking with config: { effort: "high" }.
FallbackAiProvider
Runs an ordered list of providers. Each call goes to the first provider not on cooldown; a failure puts that provider on cooldown for a time chosen by the error's kind and moves on to the next. Cooldowns are remembered across calls, so a key that hit a weekly quota is not tried again a second later.
ts
interface FallbackAiProviderOptions {
/** The ordered list of providers to try. The first is primary. */
providers: AiProvider[];
/** Cooldown per error kind in ms; null stops fallback for that kind. */
cooldownMs?: Partial<Record<ErrorKind, number | null>>;
/** Called when a provider enters cooldown. */
onCooldown?: (info: { candidate: AiProvider; error: unknown; kind: ErrorKind; retryAtMs: number }) => void;
}
class FallbackAiProvider implements AiProvider {
readonly name: string; // "fallback(zai->gemini)"
readonly capabilities: ProviderCapabilities;
readonly pool: FallbackPool<AiProvider>; // from @providerkit/core
constructor(options: FallbackAiProviderOptions);
}
ts
import { FallbackAiProvider, GeminiProvider, ZaiProvider } from "@falai/agent";
const provider = new FallbackAiProvider({
providers: [
new ZaiProvider({ apiKey: process.env.ZAI_API_KEY ?? "" }),
new GeminiProvider({ apiKey: process.env.GEMINI_API_KEY ?? "", model: "gemini-2.5-flash" }),
],
onCooldown: ({ candidate, kind, retryAtMs }) => console.warn(candidate.name, kind, new Date(retryAtMs)),
});
console.log(provider.name); // "fallback(zai->gemini)"
An empty providers list throws at construction. The difference from a provider's own fallbacks option: FallbackAiProvider works on whole AiProvider objects and keeps its pool on .pool; fallbacks folds the chain into one @providerkit/core provider inside the adapter.
createOpenAICompatibleProvider
Most endpoints that call themselves OpenAI-compatible (Azure OpenAI, Groq, Together, Fireworks, vLLM, LM Studio, Ollama, a gateway of your own) differ from OpenAI only in base URL, headers, and how they want structured output requested. This builds a provider from those settings alone.
ts
interface OpenAICompatibleOptions {
/** Names the provider in errors and logs: "azure", "ollama", "groq". */
name: string;
baseURL: string;
/** Local servers often ignore it; pass any non-empty string. */
apiKey: string;
model: string;
backupModels?: string[];
fallbacks?: Array<AiProvider | Provider>;
/** Merged over the defaults (all true except supportsPromptCaching). */
capabilities?: Partial<ProviderCapabilities>;
/** Extra request headers, e.g. Azure's `api-key`. */
defaultHeaders?: Record<string, string>;
/** Default "json_schema". */
structuredOutput?: "responses_parse" | "json_schema" | "json_object";
jsonWithTools?: "response_format" | "prompt";
config?: RequestConfig;
retryConfig?: { timeout?: number; retries?: number };
fetchImpl?: typeof fetch;
}
function createOpenAICompatibleProvider(options: OpenAICompatibleOptions): OpenAICompatibleProvider;
structuredOutput decides how a structured request is sent:
Mode
What goes out
Use when
responses_parse
OpenAI's Responses API, schema enforced natively
The endpoint is OpenAI itself. Most compatible servers do not implement it.
json_schema
Chat completions with a json_schema response format
The broadest enforced mode: DeepSeek, Groq, Together, Fireworks, vLLM. The default here.
json_object
Chat completions with plain JSON mode
Servers with no schema enforcement at all. The framework reads what comes back leniently.
Missing name, baseURL, apiKey or model throws at construction. For behaviour these settings do not cover, subclass OpenAICompatibleProvider.
OpenAICompatibleProvider
The abstract base every chat-completions provider extends: OpenAIProvider, OpenRouterProvider, DeepSeekProvider and the class behind createOpenAICompatibleProvider. A subclass names itself and its capabilities; the base picks the request shape from structuredOutput (default responses_parse) and hands everything else to ProviderAdapter.
The base class every built-in provider extends. @providerkit/core returns normalized stream chunks; this framework works in whole turns: a composed prompt plus history in, a parsed structured reply out. ProviderAdapter does that translation once, for every vendor, and adds retries, backup models and fallbacks around it. generateMessage is generateMessageStream drained, so both paths behave the same.
ts
interface ProviderAdapterInit {
/** The @providerkit/core provider this adapter drives. */
provider: Provider;
model: string;
/** Sampling defaults for every call. */
defaults?: RequestConfig;
/** Tried in order after the primary. */
backupModels?: string[];
fallbacks?: Array<AiProvider | Provider>;
fallbackOptions?: FallbackOptions<Provider>;
retryConfig?: { timeout?: number; retries?: number };
}
interface RequestConfig {
temperature?: number;
topP?: number;
maxTokens?: number;
stopSequences?: string[];
/** "none" | "low" | "medium" | "high" | "max". Absent: the model's own default, never sent. */
effort?: Effort;
}
interface RetryConfig {
/** Milliseconds a stream may stay silent before it counts as wedged. */
timeout: number;
/** Retries after the first attempt; 0 still makes one call. */
retries: number;
}
function resolveRetryConfig(input?: { timeout?: number; retries?: number }): RetryConfig;
abstract class ProviderAdapter implements AiProvider {
abstract readonly name: string;
abstract readonly capabilities: ProviderCapabilities;
protected readonly provider: Provider;
protected readonly primaryModel: string;
protected readonly backupModels: string[];
protected readonly retryConfig: RetryConfig;
get coreProvider(): Provider;
protected constructor(init: ProviderAdapterInit);
probeJsonWithTools(opts?: ProbeOptions): Promise<JsonWithToolsProbe>;
generateMessage<C, S>(input: GenerateMessageInput<C>): Promise<GenerateMessageOutput<S>>;
generateMessageStream<C, S>(input: GenerateMessageInput<C>): AsyncGenerator<GenerateMessageStreamChunk<S>>;
}
A subclass supplies a @providerkit/core provider and a name. Only @falai/agent symbols appear below, so the core provider arrives already built:
ts
import { ProviderAdapter, resolveRetryConfig } from "@falai/agent";
import type { ProviderAdapterInit, ProviderCapabilities } from "@falai/agent";
declare const core: ProviderAdapterInit["provider"]; // e.g. createOpenAIProvider(...) from @providerkit/core
class MyProvider extends ProviderAdapter {
readonly name = "mine";
readonly capabilities: ProviderCapabilities = {
supportsTools: true,
supportsNativeJsonSchema: true,
supportsStreaming: true,
supportsStreamingToolCalls: true,
supportsPromptCaching: false,
};
constructor() {
super({ provider: core, model: "my-model", backupModels: ["my-smaller-model"], retryConfig: { retries: 1 } });
}
}
console.log(resolveRetryConfig({ retries: 1 })); // { timeout: 60000, retries: 1 }
console.log(resolveRetryConfig({ timeout: 0 })); // { timeout: 60000, retries: 3 }: a 0 ms timeout would abort every call, so it is treated as unset
console.log(new MyProvider().name);
What the adapter does on every call, from src/providers/ProviderAdapter.ts:
Turns history plus the prompt into the provider's messages. The prompt is always the final user message.
Turns tools into tool definitions and parameters.jsonSchema into a JSON output request named parameters.schemaName (or structured_output). An empty schema means no JSON mode.
Merges defaults, then parameters.maxOutputTokens as maxTokens and parameters.reasoning.effort as effort.
Streams with a silence watchdog of retryConfig.timeout ms and retryConfig.retries + 1 attempts, then moves to the next backup model when the error kind allows.
Folds the chunks: text becomes delta chunks, tool call fragments are assembled and their arguments parsed, usage lands on metadata (tokensUsed, promptTokens, completionTokens, cachedInputTokens).
Parses the accumulated text leniently into structured. Text that looks like an envelope but did not parse is dropped rather than handed to the customer. A message that is blank after parsing, with no tool calls, throws Error: No response from <provider> — after the retries and backup models, so the caller's path (a deferred speak step, or the host replaying the turn) is what picks it up. A stream that produced no text and no tool call is caught earlier, as a ProviderError; reasoning alone is not an answer. When it stopped at maxTokens the kind is invalid, never retried and never sent to a backup model, and the message says how many tokens went to thinking: the output cap ran out before any answer (2048 of 2048 output tokens went to reasoning) — raise maxTokens or lower effort. Any other empty stream is overload. A stream that stops before the provider's end signal (no finish reason, no [DONE]) throws a ProviderError of kind network once what arrived is delivered, so half an answer is never read as a whole one.
Retries, backup models and fallbacks
Three layers, innermost first. All from src/providers/ProviderAdapter.ts and @providerkit/core.
Layer
Option
What triggers it
What it does
Retry
retryConfig
A transient error (timeout, network, overload, rate), or silence for timeout ms. Default timeout: 60000, retries: 3, so up to 4 attempts.
Same model, same provider. timeout: 0 is treated as unset; retries: 0 is honoured.
Backup model
backupModels
Retries exhausted with a backup-eligible kind, or kind === "model" (the endpoint does not serve that id).
Next model on the list, same provider.
Fallback
fallbacks (per provider) or FallbackAiProvider
The provider fails or is on cooldown.
Next provider. Cooldowns per kind; a server-supplied Retry-After always wins over the default interval.
The timeout bounds silence, not the whole call: a stream that keeps producing tokens is left alone, one that goes quiet for 60 s is cut and retried. Keep-alives count as life. The wait for the first chunk, and a stream that sends only keep-alives, are cut by @providerkit/core's progress clock: at 5 minutes, or at timeout when that is longer. A provider that rotates through models of its own, each watched on its own (an OpenCode Go chain), takes a long timeout so this watch never cuts the rotation short.
GenerateMessageInput.parameters.reasoning is on the interface for a custom caller, but the framework never sets it. The one way to reach the wire is config.effort on a provider, which the adapter sends with every call as the provider's default. Absent means the model's own dynamic thinking and is never sent; "none" is the only way to say do not think. It matters most under a small maxTokens, where thinking tokens and the answer share one budget: a turn that spends all of it thinking fails as invalid. Support varies per model; an unsupported level comes back as a 400 (kind: "invalid").
The AiProvider interface
What a provider must implement, from src/types/ai.ts. The built-in providers, FallbackAiProvider and the test mock all satisfy it.
Every call carries parameters.jsonSchema and a schemaName, so a custom provider can tell the calls apart and log them. From src/core/Understand.ts, src/core/Speak.ts and src/core/CompactionEngine.ts:
Call
Method
schemaName
tools
Once per
Understand
generateMessage
"understand"
never
message turn with something to judge: two or more flows to route between (counting the one on the floor), a mention flow, a when branch on the asking step, or a pending extract: 'anywhere' field on the flow on the floor or on a flow with message hints. Skipped when none of those is live.
Speak
generateMessage on turn(), generateMessageStream on turnStream()
"speak"
on rounds where tools are offered (maxToolLoops, default 5)
talk step or idle answer, plus one call per tool round
Compaction summary
generateMessage
none, and jsonSchema is {}
never
turn whose history crossed the compaction threshold, before both calls above
The framework reads structured first. When it is missing it looks for a JSON object inside message, so a provider that only returns text still works as long as the text is the envelope. A speak reply whose message is blank after the last tool round is treated like a failed call: the step is deferred (see Errors).
A custom provider
The shortest useful one wraps another provider and logs which call it is. The two methods are generic; declare the type parameters and pass them through.
ts
import { falai, GeminiProvider } from "@falai/agent";
import type { AiProvider, GenerateMessageInput } from "@falai/agent";
function logged(inner: AiProvider): AiProvider {
const tag = (input: GenerateMessageInput<unknown>): string => input.parameters?.schemaName ?? "unnamed";
return {
name: `logged(${inner.name})`,
capabilities: inner.capabilities,
generateMessage<C, S>(input: GenerateMessageInput<C>) {
console.log(`[${tag(input)}] ${input.prompt.length} chars, ${input.tools?.length ?? 0} tools`);
return inner.generateMessage<C, S>(input);
},
generateMessageStream<C, S>(input: GenerateMessageInput<C>) {
console.log(`[${tag(input)}] streaming`);
return inner.generateMessageStream<C, S>(input);
},
};
}
const f = falai().fields({ nome: { type: "string", ask: "Pergunte o nome." } });
const agent = f.agent({
name: "Ana",
provider: logged(new GeminiProvider({ apiKey: process.env.GEMINI_API_KEY ?? "", model: "gemini-2.5-flash" })),
flows: [f.flow({ id: "oi", name: "Oi", on: [{ message: [] }], steps: [{ id: "nome", collect: ["nome"] }] })],
});
const r = await agent.turn({ sessionId: "s1", message: "oi" });
console.log(r.llmCalls); // logs "[speak] …" once: one flow with no routing hints and nobody on the floor, so no understand call
When the model narrates a tool instead of calling it
Every turn the framework sends pins the model's output to a schema, because that is how message and the step's fields come back. Some models cannot emit a tool call while pinned that way. They do not report it: the model writes "deixa eu ver aqui" and stops. On the wire the call succeeded; in the product the agent never uses its tools, and no instruction fixes it.
jsonWithTools on GeminiProvider, OpenRouterProvider, DeepSeekProvider, createOpenAICompatibleProvider and OpenAICompatibleProviderInit decides how the schema is sent on calls that also carry tools:
"response_format" sends both, which every wire documents and most models honour.
"prompt" leaves the response format off those calls and puts the schema in the prompt instead. Calls without tools are untouched.
The answer belongs to the model, not the endpoint, and it goes both ways. The code's own measurements, sampled on 2026-09-07. Each cell is the samples where the model called its tool:
Model
Response format
Prompt
z-ai/glm-5.3-flash (OpenRouter)
0/10
8/8
qwen3.8-flash (OpenRouter)
10/10
1/6
DeepSeek's flash models
0/5
not sampled
gemini-3.8-flash
3/10
not sampled
gemini-3.5-flash, gemini-3.5-flash-lite
every call
not sampled
Do not pick from that list. Ask the model you ship.
The probe
Every ProviderAdapter subclass has probeJsonWithTools(opts?). It asks the bound model, on the wire, both ways, and reports which shape called the tool on every sample.
The probe asks the adapter's own model, never its fallbacks, one call at a time. A rate-limited fallback cannot make a working primary look broken, and a primary that fails the probe throws. A FallbackAiProvider has no probe of its own: probe each adapter you put in it.
ts
interface JsonWithToolsProbe {
/** The shape to configure, or null when neither called the tool on every sample. */
use: "response_format" | "prompt" | null;
/** Samples that produced a tool call, per shape. */
calls: Record<"response_format" | "prompt", number>;
samples: number;
}
interface ProbeOptions {
/** Samples per shape. Default 3. */
samples?: number;
/** Probe a model other than the bound one. */
model?: string;
signal?: AbortSignal;
}
ts
import { OpenRouterProvider } from "@falai/agent";
const apiKey = process.env.OPENROUTER_API_KEY ?? "";
const model = "z-ai/glm-5.3-flash";
const probe = await new OpenRouterProvider({ apiKey, model }).probeJsonWithTools();
console.log(probe.calls); // e.g. { response_format: 0, prompt: 3 }
if (!probe.use) throw new Error(`${model} cannot call a tool while pinned to a schema; pick another model`);
const provider = new OpenRouterProvider({ apiKey, model, jsonWithTools: probe.use });
console.log(provider.name);
Run it once, at boot. It costs samples × 2 short calls (6 by default), and errors propagate: a probe that swallowed a bad key would report "this model cannot call tools", which is worse to believe than "the call failed". Log calls, not just use; 0/3 and 3/3 is what makes the next model swap's regression obvious. The probe catches the structural failure, a model that cannot emit the call on the easiest question there is; a model that is merely unreliable passes it.