Skip to main content

Use Teale with OpenAI SDK

Teale exposes an OpenAI-compatible API at http://localhost:11435/v1. Any application or library that works with the OpenAI API works with Teale --- no code changes beyond the base URL.

Prerequisites​

  • Teale running with a model loaded (teale up)
  • The OpenAI SDK for your language installed

Python​

Install the OpenAI Python package:

pip install openai

Streaming completion​

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11435/v1",
    api_key="not-needed",
)

response = client.chat.completions.create(
    model="llama-3.1-8b-instruct-4bit",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

Non-streaming completion​

response = client.chat.completions.create(
    model="llama-3.1-8b-instruct-4bit",
    messages=[{"role": "user", "content": "Explain quicksort in two sentences."}],
)

print(response.choices[0].message.content)

Node.js​

Install the OpenAI Node package:

npm install openai

Streaming completion​

import OpenAI from 'openai';

const client = new OpenAI({
    baseURL: 'http://localhost:11435/v1',
    apiKey: 'not-needed',
});

const stream = await client.chat.completions.create({
    model: 'llama-3.1-8b-instruct-4bit',
    messages: [{ role: 'user', content: 'Hello!' }],
    stream: true,
});

for await (const chunk of stream) {
    process.stdout.write(chunk.choices[0]?.delta?.content || '');
}

Non-streaming completion​

const response = await client.chat.completions.create({
    model: 'llama-3.1-8b-instruct-4bit',
    messages: [{ role: 'user', content: 'Explain quicksort in two sentences.' }],
});

console.log(response.choices[0].message.content);

List available models​

Use the standard models endpoint to see what is loaded:

curl http://localhost:11435/v1/models | jq

Or via the SDK:

models = client.models.list()
for model in models:
    print(model.id)

Compatible libraries and tools​

Teale's OpenAI-compatible API works with any client that supports custom base URLs:

Tool / LibraryConfiguration
LangChainSet openai_api_base="http://localhost:11435/v1"
LlamaIndexSet api_base="http://localhost:11435/v1"
Open WebUIAdd connection with URL http://localhost:11435
Vercel AI SDKUse createOpenAI({ baseURL: "http://localhost:11435/v1" })

See IDE Integration for editor-specific setup.

Authentication​

When Teale runs locally with network access disabled (the default), no API key is needed. Pass any string as api_key to satisfy SDK requirements.

If you have enabled network access and generated API keys, pass the key as a Bearer token:

client = OpenAI(
    base_url="http://localhost:11435/v1",
    api_key="your-api-key-here",
)

See API Key Management for details.

Next steps​