Skip to main content

Together AI integration

View Markdown

Together AI serves open-weight models such as gpt-oss, DeepSeek, Qwen, and GLM on a serverless per-token API and on dedicated endpoints through an API compatible with the OpenAI Chat Completions API.

Temporal's OpenAI Agents SDK integration runs each model call as an Activity. A call to Together AI gets retries with backoff, a timeout, and a durable record in Event History, and the agent loop around it survives Worker restarts. Because Together AI speaks the OpenAI Chat Completions API, the integration's model provider hook is all you need: the Worker points an OpenAI client at Together AI's base URL and names a model, and the Workflow code does not change.

This integration is a sample pattern rather than a plugin. Code snippets in this guide are taken from the OpenAI Agents SDK model provider samples, which run the same agent on Baseten, Fireworks AI, or Together AI.

Prerequisites​

Install​

uv add temporalio-openai-agents

The package installs the Temporal Python SDK, the OpenAI Agents SDK, and the openai client. Together AI's own together package is not needed; the sample talks to Together AI through the openai client.

Configure the model provider​

The sample describes each provider as a base URL, the environment variable that holds its API key, and a default model:

openai_agents/model_providers/openai_compatible_providers.py

TOGETHER = OpenAICompatibleProvider(
    name="together",
    base_url="https://api.together.ai/v1",
    api_key_env="TOGETHER_API_KEY",
    model="openai/gpt-oss-120b",
)

Together AI model IDs follow the <organization>/<model-name> convention, for example openai/gpt-oss-120b; OpenAI's own model names return a 404. The serverless models catalog lists each model's ID and whether it supports function calling and structured outputs. The sample's default, openai/gpt-oss-120b, is an open-weight reasoning model that supports both. A dedicated endpoint is addressed the same way: its endpoint string, <project-slug>/<endpoint-name>, goes in the model field, and requests go to https://api-inference.together.ai/v1 instead of the serverless base URL.

A ModelProvider turns that description into the Chat Completions model that the Agents SDK calls:

openai_agents/model_providers/openai_compatible_providers.py

class OpenAICompatibleModelProvider(ModelProvider):
    """OpenAI Agents SDK model provider for an OpenAI-compatible endpoint.

    Client retries are off: the Temporal plugin runs each model call as an
    Activity, and the Activity's Retry Policy owns the retries. An agent that
    names no model gets the provider's default; an agent can name any other
    model the provider serves.
    """

    def __init__(self, provider: OpenAICompatibleProvider) -> None:
        self._provider = provider
        self._client = AsyncOpenAI(
            base_url=provider.base_url,
            api_key=os.environ[provider.api_key_env],
            max_retries=0,
        )

    def get_model(self, model_name: str | None) -> Model:
        # Chat Completions is the API all of these providers serve. The Agents
        # SDK defaults to the Responses API, which not every provider offers.
        return OpenAIChatCompletionsModel(
            model=model_name or self._provider.model,
            openai_client=self._client,
        )

Two details matter here. Client retries are off (max_retries=0), so every attempt is a visible Activity attempt governed by the Activity's Retry Policy rather than a hidden retry inside the HTTP client. And the provider builds an OpenAIChatCompletionsModel rather than the Agents SDK's default Responses API model, which Together AI's OpenAI compatibility guide does not cover.

Configure the Worker​

Pass the provider to the plugin. A custom model provider must set a model Activity timeout; the sample uses 60 seconds. Serverless performance on Together AI is best-effort, and a dedicated endpoint that was stopped or scaled down pays a cold start on its next request, so size the timeout for the slowest call you expect rather than the median. The Heartbeat Timeout lets Temporal notice a Worker that died mid-call within seconds instead of after the full Start-To-Close Timeout; the plugin heartbeats on its own.

openai_agents/model_providers/run_openai_compatible_worker.py

client = await Client.connect(
    "localhost:7233",
    plugins=[
        OpenAIAgentsPlugin(
            model_params=ModelActivityParameters(
                # A custom model provider must set a timeout. Raise it for
                # slow models or deployments that scale up from zero.
                start_to_close_timeout=timedelta(seconds=60),
                # The plugin heartbeats, so a Worker that dies mid-call is
                # noticed in seconds rather than after the full timeout.
                heartbeat_timeout=timedelta(seconds=10),
            ),
            model_provider=OpenAICompatibleModelProvider(PROVIDERS[provider_name]),
        ),
    ],
)

worker = Worker(
    client,
    task_queue="openai-agents-model-providers-task-queue",
    workflows=[OpenAICompatibleAgentWorkflow],
)
await worker.run()

The Client that starts the Workflow also needs OpenAIAgentsPlugin(), for its Data Converter, but no model provider: only the Worker talks to Together AI.

Write the Workflow​

The Workflow is ordinary Agents SDK code. It names no model, so the Worker's provider supplies its default; set model= on the Agent to any model Together AI serves to override it.

openai_agents/model_providers/workflows/openai_compatible_workflow.py

@workflow.defn
class OpenAICompatibleAgentWorkflow:
    @workflow.run
    async def run(self, prompt: str) -> str:
        # Tools that run inside the Workflow must be async: the Agents SDK runs
        # sync tools in a thread, which the Workflow sandbox does not allow.
        @function_tool
        async def get_weather(city: str) -> str:
            workflow.logger.debug(f"Getting weather for {city}")
            return f"The weather in {city} is sunny."

        # No model is named here, so the Worker's model provider supplies its
        # default. Set model= to any model the provider serves to override it.
        # Start from empty model settings: an agent that names no model gets
        # the Agents SDK's defaults for OpenAI's default model (a reasoning
        # effort and `verbosity`), which other providers reject.
        agent = Agent(
            name="Assistant",
            instructions="You only respond in haikus. When asked about the weather always use the tool to get the current weather.",
            model_settings=ModelSettings(),
            tools=[get_weather],
        )

        result = await Runner.run(agent, prompt)
        return result.final_output

The empty ModelSettings() matters. An Agent that names no model gets the Agents SDK's default settings for OpenAI's current default model, which include verbosity and a reasoning effort, and the plugin forwards them to the provider. Fireworks AI, for example, rejects verbosity with a 400. An agent that names its model explicitly does not get those defaults.

Tools that run inside the Workflow must be async: the Agents SDK runs sync tools in a thread, which the Workflow sandbox does not allow. Tools that do I/O belong in Activities; see Activity-backed tools.

Run the sample​

Start the Worker with Together AI selected:

export TOGETHER_API_KEY="your_api_key"
uv run openai_agents/model_providers/run_openai_compatible_worker.py --provider together

Then start the Workflow in a second terminal:

uv run openai_agents/model_providers/run_openai_compatible_workflow.py

Retries, rate limits, and timeouts​

The plugin classifies Together AI's HTTP errors. Status 408, 409, 429, and any 5xx retry under the Activity's Retry Policy, and a Retry-After header becomes the next retry delay. Connection errors and client-side timeouts retry as well. Other 4xx responses, such as an invalid API key or an unknown model name, fail the Activity without retrying. Each attempt, its failure, and the final result are in Event History.

Under high demand, Together AI's serverless API answers 429 Too Many Requests or 503 Service Unavailable and asks clients to retry with exponential backoff, which is what the default Activity Retry Policy does; Together AI's documentation does not describe a Retry-After header. Its error codes page lists 500, 503, 504, 524, and 529 as retryable and the rest as request, credential, or billing problems to fix first, including 402 when the account reaches its monthly spending limit and 403 when the prompt plus max_tokens exceeds the model's context length. Serverless performance is best-effort; see rate limits for the current policy and provisioned throughput for reserved capacity with a throughput and reliability commitment.

What works through Together AI​

Function tools, Activity-backed tools, handoffs, agents as tools, and MCP servers all work unchanged: they run in the Workflow or in Activities, not at the provider. OpenAI-hosted tools such as web search, file search, code interpreter, and hosted MCP are Responses API features of OpenAI's own service, so they are not available through the Chat Completions API that the sample uses. The sample disables the Agents SDK's hosted tracing, which exports to OpenAI; the OpenTelemetry tracing path works as usual.

Together AI notes​

  • Dedicated endpoints are served at https://api-inference.together.ai/v1, not the serverless base URL. Deploy a model, wait for the deployment to reach DEPLOYMENT_STATE_READY (first-time provisioning takes minutes), then use the endpoint string as the model name. A running deployment keeps at least one replica; stopping it ends the cost and makes the next request pay a cold start. See autoscaling for the replica bounds.
  • Regional inference: https://us-api.together.ai/v1 keeps serverless processing in the United States and https://ca-api.together.ai/v1 in Canada, each with its own model list. See geo-specific inference.
  • openai/gpt-oss-120b is a reasoning model. Together AI accepts reasoning_effort (low, medium, or high; the default is medium), which the Agents SDK sends when an agent sets ModelSettings(reasoning=Reasoning(effort="low")). Together AI returns the model's chain of thought in a reasoning field, which the Agents SDK's Chat Completions model does not read, so prior reasoning is not passed back between tool calls. See reasoning.
  • Structured outputs: an agent's output_type becomes a response_format JSON schema, which Together AI supports on models marked for it in the catalog. Together AI recommends also describing the schema in the prompt, and a response that stops with finish_reason of length leaves the JSON truncated, so give structured answers enough max_tokens.
  • Model discovery: the openai client's models.list() works against Together AI, and the serverless catalog marks function calling and structured outputs per model. Together AI retires serverless models over time (for example, openai/gpt-oss-20b in September 2026), so check the changelog before pinning a model in production.

Samples​