Skip to main content

AI Models & Chat

OpenChat uses OpenRouter as a unified gateway to access 100+ AI models from providers like OpenAI, Anthropic, Google, Meta, and more.

Overview

OpenChat supports two modes for AI access:
ModeDescriptionCost
OSSChat CloudUses server’s OpenRouter keyFree (10¢/day limit)
BYOKBring Your Own KeyYour OpenRouter credits

Dual Provider System

OSSChat Cloud (Default)

The free tier uses the server’s OpenRouter API key with daily limits:
// Automatically used when no personal key is set
const provider = "osschat";
const apiKey = process.env.OPENROUTER_API_KEY; // Server's key
  • Daily limit: ~10¢ worth of usage
  • Resets: Daily at midnight UTC
  • Models: All OpenRouter models available

BYOK (Bring Your Own Key)

Users can connect their own OpenRouter account for unlimited access:
// User's personal API key
const provider = "openrouter";
const apiKey = userStore.openrouterApiKey; // User's key
  • No limits: Uses your own OpenRouter credits
  • All models: Including premium models
  • Privacy: API key stored locally (encrypted if saved to Convex)

Connecting OpenRouter (BYOK)

1

Open Settings

Click your profile in the sidebar, then click Settings.
2

Go to Providers Tab

Select the Providers tab to see connection options.
3

Connect OpenRouter

Click Connect OpenRouter. You’ll be redirected to OpenRouter to authorize.
OpenChat uses OAuth PKCE for secure key exchange. Your key is never exposed in URLs.
4

Authorize

On OpenRouter, authorize the application. You’ll be redirected back to OpenChat.
5

Start Using

Your models are now available! The provider switches to “Personal OpenRouter” automatically.

Model Selection

Using the Model Selector

Click the model name in the chat input to open the model selector:
  • Search: Type to filter models
  • Favorites: Star models for quick access
  • Details: See context window, pricing, capabilities

Model Capabilities

Models vary in their capabilities:
CapabilityDescriptionExample Models
ReasoningExtended thinking for complex tasksClaude 3.5, o1, DeepSeek R1
VisionProcess imagesClaude 3.5, GPT-4o, Gemini
Long ContextLarge context windows (100K+)Claude 3.5, Gemini Pro
FastLow latency responsesGPT-4o-mini, Claude Haiku

Reasoning Mode

For supported models, enable reasoning mode to see the AI’s thinking process:
// Configure reasoning effort
const reasoningEffort = "medium"; // none, low, medium, high

// Sent to OpenRouter
providerOptions: {
  openrouter: {
    reasoning: {
      effort: reasoningEffort
    }
  }
}
Reasoning appears in a collapsible “Chain of Thought” section above the response.

Chat API

Endpoint

POST /api/chat

Request Body

{
  messages: UIMessage[],      // Conversation history
  model: string,              // e.g., "anthropic/claude-3.5-sonnet"
  provider: "osschat" | "openrouter",
  apiKey?: string,            // Required for "openrouter" provider
  enableWebSearch?: boolean,  // Enable web search tool
  reasoningEffort?: "none" | "low" | "medium" | "high",
  maxSteps?: number           // Max tool call iterations (default: 5)
}

Response

Returns a Server-Sent Events stream compatible with AI SDK 5’s useChat hook:
// Stream events
data: {"type":"text","content":"Hello"}
data: {"type":"text","content":" there!"}
data: {"type":"finish","usage":{"promptTokens":10,"completionTokens":5}}

Example

import { useChat } from "ai/react";

function ChatComponent() {
  const { messages, input, handleInputChange, handleSubmit } = useChat({
    api: "/api/chat",
    body: {
      model: "anthropic/claude-3.5-sonnet",
      provider: "osschat",
    },
  });
  
  return (
    <form onSubmit={handleSubmit}>
      <input value={input} onChange={handleInputChange} />
      <button type="submit">Send</button>
    </form>
  );
}

Streaming Architecture

OpenChat uses AI SDK 5 for streaming:

Cancellation Support

Users can cancel in-progress responses:
// Client
const abortController = new AbortController();
fetch("/api/chat", { signal: abortController.signal });

// To cancel
abortController.abort();

// Server
const abortSignal = request.signal;
streamText({ model, messages, abortSignal });
OpenChat can search the web for current information:
// Enable web search
const response = await fetch("/api/chat", {
  method: "POST",
  body: JSON.stringify({
    messages,
    model: "anthropic/claude-3.5-sonnet",
    enableWebSearch: true,  // Enable
    maxSteps: 5             // Allow tool iterations
  })
});
Web search uses the Valyu AI SDK and requires VALYU_API_KEY to be configured.

Token Usage & Cost Tracking

Usage Display

Token usage is tracked and displayed per message:
// Response includes usage metadata
{
  inputTokens: 1234,
  outputTokens: 567,
  totalTokens: 1801
}

Cost Calculation

For OSSChat Cloud, costs are tracked against the daily limit:
// Provider store tracks usage
const { dailyUsageCents, dailyLimitCents } = useProviderStore();

// Display remaining budget
const remaining = dailyLimitCents - dailyUsageCents; // e.g., 7¢ remaining

File Attachments

Upload images and documents to include in messages:
// Generate upload URL
const uploadUrl = await convex.mutation(api.files.generateUploadUrl, {
  userId,
  chatId
});

// Upload file
await fetch(uploadUrl, {
  method: "POST",
  body: file
});

// Save metadata
const { fileId, url } = await convex.mutation(api.files.saveFileMetadata, {
  userId,
  chatId,
  storageId,
  filename: file.name,
  contentType: file.type,
  size: file.size
});

Limits

LimitValue
Max file size10 MB
Files per user150 total
Supported typesImages (jpg, png, gif, webp), PDF

Prompt Templates

Create custom slash commands for frequently used prompts:
// Create template
await convex.mutation(api.promptTemplates.create, {
  name: "Code Review",
  command: "/review",
  template: "Review this code for bugs and improvements:\n\n$ARGUMENTS",
  category: "coding"
});

// Use in chat
/review function add(a, b) { return a + b; }

Template Variables

VariableDescription
$ARGUMENTSEverything after the command
$1, $2, …Positional arguments

Configuration

Environment Variables

# Server-side
OPENROUTER_API_KEY=sk-or-v1-...  # For OSSChat Cloud
VALYU_API_KEY=...                 # For web search

# Client-side (if needed)
# None - API keys should never be on client

Rate Limits

The chat API is rate-limited to prevent abuse:
OperationRateBurst
Message send30/min10
Stream upsert200/min50

Troubleshooting

Check:
  1. Model is available (some are intermittently down)
  2. API key is valid (try regenerating)
  3. You have credits (for BYOK) or daily budget (for OSSChat)
This can happen if:
  1. Network connection drops
  2. Model reaches max tokens
  3. Rate limit exceeded
Try sending a shorter message or switching models.
Verify:
  1. VALYU_API_KEY is set on the server
  2. enableWebSearch: true in request
  3. Model supports tool use
For OSSChat Cloud, the 10¢ daily limit resets at midnight UTC. Options:
  1. Wait for reset
  2. Connect your own OpenRouter key (BYOK)

Next Steps