promptfoo/site/docs/providers/ai21.md

AI21 Labs

The AI21 Labs API offers access to AI21 models such as jamba-1.5-mini and jamba-1.5-large.

API Key

To use AI21 Labs, you need to set the AI21_API_KEY environment variable, or specify the apiKey in the provider configuration.

Example of setting the environment variable:

export AI21_API_KEY=your_api_key_here

Model Selection

You can specify which AI21 model to use in your configuration. Currently, the following models are available:

1. jamba-1.5-mini
2. jamba-1.5-large

Here's an example config that compares AI21 models:

providers:
  - ai21:jamba-1.5-mini
  - ai21:jamba-1.5-large

Options

The AI21 provider supports several options to customize the behavior of the model. These include:

• temperature: Controls the randomness of the output.
• top_p: Controls nucleus sampling, affecting the randomness of the output.
• max_tokens: The maximum length of the generated text.
• response_format: Set to { type: 'json_object' } for JSON output or { type: 'text' } for text output.
• apiKeyEnvar: An environment variable that contains the API key.
• apiBaseUrl: The base URL of the AI21 API.

Example Configuration

Here's an example configuration for the AI21 provider:

providers:
  - ai21:jamba-1.5-mini
    config:
      apiKey: your_api_key_here
      temperature: 0.1
      top_p: 1
      max_tokens: 1024
      response_format: { type: 'json_object' }

This configuration uses the jamba-1.5-mini model with a temperature of 0.1, top-p sampling with a value of 1, a maximum output length of 1024 tokens, JSON-formatted output, and a custom cost of $0.2 per 1M input tokens.

Cost

The cost of using AI21 models depends on the model and the number of input and output tokens. Here are the costs for the available models:

• jamba-1.5-mini: $0.2 per 1M input tokens, $0.4 per 1M output tokens
• jamba-1.5-large: $2 per 1M input tokens, $8 per 1M output tokens

You can set the cost option in the provider configuration to specify a custom cost for the model.

Supported environment variables

These AI21-related environment variables are supported:

Variable	Description
AI21_API_BASE_URL	The base URL (protocol + hostname + port) to use for the AI21 API.
AI21_API_KEY	AI21 API key.