Skip to main content

Baseten integration

View Markdown

Baseten serves open-weight models such as gpt-oss, DeepSeek, GLM, and Kimi, both on its shared Model APIs and on dedicated deployments, through an API compatible with the OpenAI Chat Completions API.

Temporal's OpenAI Agents SDK integration runs each model call as an Activity. A call to Baseten gets retries with backoff, a timeout, and a durable record in Event History, and the agent loop around it survives Worker restarts. Because Baseten speaks the OpenAI Chat Completions API, the integration's model provider hook is all you need: the Worker points an OpenAI client at Baseten's base URL and names a model, and the Workflow code does not change.

This integration is a sample pattern rather than a plugin. Code snippets in this guide are taken from the OpenAI Agents SDK model provider samples, which run the same agent on Baseten, Fireworks AI, or Together AI.

Prerequisites​

Install​

uv add temporalio-openai-agents

The package installs the Temporal Python SDK, the OpenAI Agents SDK, and the openai client. The Baseten CLI and the truss package are for deploying models to Baseten; calling Model APIs needs neither.

Configure the model provider​

The sample describes each provider as a base URL, the environment variable that holds its API key, and a default model:

openai_agents/model_providers/openai_compatible_providers.py

BASETEN = OpenAICompatibleProvider(
    name="baseten",
    base_url="https://inference.baseten.co/v1",
    api_key_env="BASETEN_API_KEY",
    model="openai/gpt-oss-120b",
)

Model APIs name models the way Hugging Face does, as <organization>/<model>, so the default here is openai/gpt-oss-120b: an open-weight model under the Apache 2.0 license that Baseten serves on Model APIs, where tool calling is supported for every model. Its model library page and GET https://inference.baseten.co/v1/models describe the catalog. A dedicated deployment has its own base URL, https://model-<id>.api.baseten.co/environments/production/sync/v1, takes the same API key, and expects the model name that the deployment serves.

A ModelProvider turns that description into the Chat Completions model that the Agents SDK calls:

openai_agents/model_providers/openai_compatible_providers.py

class OpenAICompatibleModelProvider(ModelProvider):
    """OpenAI Agents SDK model provider for an OpenAI-compatible endpoint.

    Client retries are off: the Temporal plugin runs each model call as an
    Activity, and the Activity's Retry Policy owns the retries. An agent that
    names no model gets the provider's default; an agent can name any other
    model the provider serves.
    """

    def __init__(self, provider: OpenAICompatibleProvider) -> None:
        self._provider = provider
        self._client = AsyncOpenAI(
            base_url=provider.base_url,
            api_key=os.environ[provider.api_key_env],
            max_retries=0,
        )

    def get_model(self, model_name: str | None) -> Model:
        # Chat Completions is the API all of these providers serve. The Agents
        # SDK defaults to the Responses API, which not every provider offers.
        return OpenAIChatCompletionsModel(
            model=model_name or self._provider.model,
            openai_client=self._client,
        )

Two details matter here. Client retries are off (max_retries=0), so every attempt is a visible Activity attempt governed by the Activity's Retry Policy rather than a hidden retry inside the HTTP client. And the provider builds an OpenAIChatCompletionsModel rather than the Agents SDK's default Responses API model, which Baseten does not serve: Model APIs expose Chat Completions and, in beta, the Anthropic Messages API.

Configure the Worker​

Pass the provider to the plugin. A custom model provider must set a model Activity timeout; the sample uses 60 seconds. Model APIs need no deployment of your own to warm up; a dedicated deployment that has scaled to zero parks a request until a replica starts, for up to 1200 seconds by default, so cover that cold start in the timeout or keep min_replica above zero. The Heartbeat Timeout lets Temporal notice a Worker that died mid-call within seconds instead of after the full Start-To-Close Timeout; the plugin heartbeats on its own.

openai_agents/model_providers/run_openai_compatible_worker.py

client = await Client.connect(
    "localhost:7233",
    plugins=[
        OpenAIAgentsPlugin(
            model_params=ModelActivityParameters(
                # A custom model provider must set a timeout. Raise it for
                # slow models or deployments that scale up from zero.
                start_to_close_timeout=timedelta(seconds=60),
                # The plugin heartbeats, so a Worker that dies mid-call is
                # noticed in seconds rather than after the full timeout.
                heartbeat_timeout=timedelta(seconds=10),
            ),
            model_provider=OpenAICompatibleModelProvider(PROVIDERS[provider_name]),
        ),
    ],
)

worker = Worker(
    client,
    task_queue="openai-agents-model-providers-task-queue",
    workflows=[OpenAICompatibleAgentWorkflow],
)
await worker.run()

The Client that starts the Workflow also needs OpenAIAgentsPlugin(), for its Data Converter, but no model provider: only the Worker talks to Baseten.

Write the Workflow​

The Workflow is ordinary Agents SDK code. It names no model, so the Worker's provider supplies its default; set model= on the Agent to any model Baseten serves to override it.

openai_agents/model_providers/workflows/openai_compatible_workflow.py

@workflow.defn
class OpenAICompatibleAgentWorkflow:
    @workflow.run
    async def run(self, prompt: str) -> str:
        # Tools that run inside the Workflow must be async: the Agents SDK runs
        # sync tools in a thread, which the Workflow sandbox does not allow.
        @function_tool
        async def get_weather(city: str) -> str:
            workflow.logger.debug(f"Getting weather for {city}")
            return f"The weather in {city} is sunny."

        # No model is named here, so the Worker's model provider supplies its
        # default. Set model= to any model the provider serves to override it.
        # Start from empty model settings: an agent that names no model gets
        # the Agents SDK's defaults for OpenAI's default model (a reasoning
        # effort and `verbosity`), which other providers reject.
        agent = Agent(
            name="Assistant",
            instructions="You only respond in haikus. When asked about the weather always use the tool to get the current weather.",
            model_settings=ModelSettings(),
            tools=[get_weather],
        )

        result = await Runner.run(agent, prompt)
        return result.final_output

The empty ModelSettings() matters. An Agent that names no model gets the Agents SDK's default settings for OpenAI's current default model, which include verbosity and a reasoning effort, and the plugin forwards them to the provider. Fireworks AI, for example, rejects verbosity with a 400. An agent that names its model explicitly does not get those defaults.

Tools that run inside the Workflow must be async: the Agents SDK runs sync tools in a thread, which the Workflow sandbox does not allow. Tools that do I/O belong in Activities; see Activity-backed tools.

Run the sample​

Start the Worker with Baseten selected:

export BASETEN_API_KEY="your_api_key"
uv run openai_agents/model_providers/run_openai_compatible_worker.py --provider baseten

Then start the Workflow in a second terminal:

uv run openai_agents/model_providers/run_openai_compatible_workflow.py

Retries, rate limits, and timeouts​

The plugin classifies Baseten's HTTP errors. Status 408, 409, 429, and any 5xx retry under the Activity's Retry Policy, and a Retry-After header becomes the next retry delay. Connection errors and client-side timeouts retry as well. Other 4xx responses, such as an invalid API key or an unknown model name, fail the Activity without retrying. Each attempt, its failure, and the final result are in Event History.

Baseten enforces requests-per-minute and tokens-per-minute limits that depend on your account tier and answers 429 when either is exceeded; the x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens headers show which one hit zero. A 529 means Baseten had no serving capacity for the request even within your limits. Both retry under the Activity's Retry Policy. Baseten's error reference asks for exponential backoff and mentions a Retry-After header only as a possibility on 529, so the Retry Policy's backoff, not the provider, usually sets the pace. Sustained 429 responses mean the limit itself needs to go up, through Baseten's rate limit increase form.

What works through Baseten​

Function tools, Activity-backed tools, handoffs, agents as tools, and MCP servers all work unchanged: they run in the Workflow or in Activities, not at the provider. OpenAI-hosted tools such as web search, file search, code interpreter, and hosted MCP are Responses API features of OpenAI's own service, so they are not available through the Chat Completions API that the sample uses. The sample disables the Agents SDK's hosted tracing, which exports to OpenAI; the OpenTelemetry tracing path works as usual.

Baseten notes​

  • A dedicated deployment is called at https://model-<id>.api.baseten.co/environments/production/sync/v1 with the same Authorization: Bearer key; the legacy Api-Key scheme is still accepted. The model name must match what the deployment serves. A deployment whose min_replica is zero releases its replicas when idle, and the first request after that waits for a replica to start; see Baseten's request lifecycle page.
  • GET https://inference.baseten.co/v1/models lists the Model APIs catalog with context windows, pricing, and supported features, as does baseten model-api list from the Baseten CLI. Baseten notes that the catalog can report different limits from the ones the endpoint enforces.
  • gpt-oss-120b reasons by default at medium effort. The reasoning arrives in reasoning_content on the assistant message, and its tokens count as output tokens. Set ModelSettings(reasoning=Reasoning(effort="low")) on the agent to change the effort; Baseten's reasoning page lists the accepted values per model.
  • Tool calling, structured outputs, and JSON mode are available on every Model API through tools and response_format; see Baseten's function calling and structured outputs guides. Only one completion per request is supported, so leave n at its default.
  • Baseten reuses cached input tokens across requests that share an x-session-affinity header. An agent run resends the growing conversation on every turn, so sending the Workflow ID as the session, through ModelSettings(extra_headers=...), lets later turns reuse the cache. Baseten recommends at most about 60 requests per minute per session.
  • Two Baseten features sit outside this pattern: async inference, which delivers results to a webhook and works only on dedicated deployments, and server-side web search tools on Model APIs, which are in preview.

Samples​