promptfoo/site/docs/providers/index.md

LLM Providers

Providers in promptfoo are the interfaces to various language models and AI services. This guide will help you understand how to configure and use providers in your promptfoo evaluations.

Quick Start

Here's a basic example of configuring providers in your promptfoo YAML config:

providers:
  - openai:gpt-4o-mini
  - anthropic:messages:claude-3-5-sonnet-20241022
  - vertex:gemini-pro

Available Providers

Api Providers	Description	Syntax & Example
OpenAI	GPT models including GPT-4 and GPT-3.5	openai:o1-preview
Anthropic	Claude models	anthropic:messages:claude-3-5-sonnet-20241022
HTTP	Generic HTTP-based providers	https://api.example.com/v1/chat/completions
Javascript	Custom - JavaScript file	file://path/to/custom_provider.js
Python	Custom - Python file	file://path/to/custom_provider.py
Shell Command	Custom - script-based providers	exec: python chain.py
AI21 Labs	Jurassic and Jamba models	ai21:jamba-1.5-mini
AWS Bedrock	AWS-hosted models from various providers	bedrock:us.meta.llama3-2-90b-instruct-v1:0
Azure OpenAI	Azure-hosted OpenAI models	azureopenai:gpt-4o-custom-deployment-name
Cloudflare AI	Cloudflare's AI platform	cloudflare-ai:@cf/meta/llama-3-8b-instruct
Cohere	Cohere's language models	cohere:command
F5	OpenAI-compatible AI Gateway interface	f5:path-name
fal.ai	Image Generation Provider	fal:image:fal-ai/fast-sdxl
GitHub	GitHub AI Gateway	github:gpt-4o-mini
Google AI Studio (PaLM)	Gemini and PaLM models	google:gemini-pro
Google Vertex AI	Google Cloud's AI platform	vertex:gemini-pro
Groq	High-performance inference API	groq:llama3-70b-8192-tool-use-preview
Hugging Face	Access thousands of models	huggingface:text-generation:gpt2
IBM BAM	IBM's foundation models	bam:chat:ibm/granite-13b-chat-v2
LiteLLM	Unified interface for multiple providers	Compatible with OpenAI syntax
Mistral AI	Mistral's language models	mistral:open-mistral-nemo
OpenLLM	BentoML's model serving framework	Compatible with OpenAI syntax
OpenRouter	Unified API for multiple providers	openrouter:mistral/7b-instruct
Perplexity AI	Specialized in question-answering	Compatible with OpenAI syntax
Replicate	Various hosted models	replicate:stability-ai/sdxl
Together AI	Various hosted models	Compatible with OpenAI syntax
Voyage AI	Specialized embedding models	voyage:voyage-3
vLLM	Local	Compatible with OpenAI syntax
Ollama	Local	ollama:llama3.2:latest
LocalAI	Local	localai:gpt4all-j
Llamafile	OpenAI-compatible llamafile server	Uses OpenAI provider with custom endpoint
llama.cpp	Local	llama:7b
Text Generation WebUI	Gradio WebUI	Compatible with OpenAI syntax
WebSocket	WebSocket-based providers	ws://example.com/ws
Webhook	Custom - Webhook integration	webhook:http://example.com/webhook
Echo	Custom - For testing purposes	echo
Manual Input	Custom - CLI manual entry	promptfoo:manual-input
Go	Custom - Go file	file://path/to/your/script.go
Web Browser	Custom - Automate web browser interactions	browser
Sequence	Custom - Multi-prompt sequencing	sequence with config.inputs array
Simulated User	Custom - Conversation simulator	promptfoo:simulated-user
WatsonX	IBM's WatsonX	watsonx:ibm/granite-13b-chat-v2
X.AI	X.AI's models	xai:grok-2
Adaline Gateway	Unified interface for multiple providers	Compatible with OpenAI syntax

Provider Syntax

Providers are specified using various syntax options:

1. Simple string format:

   provider_name:model_name
   

   Example: openai:gpt-4o-mini or anthropic:claude-3-sonnet-20240229

2. Object format with configuration:

   - id: provider_name:model_name
     config:
       option1: value1
       option2: value2
   

   Example:

   - id: openai:gpt-4o-mini
     config:
       temperature: 0.7
       max_tokens: 150
   

3. File-based configuration:

   - file://path/to/provider_config.yaml
   

Configuring Providers

Most providers use environment variables for authentication:

export OPENAI_API_KEY=your_api_key_here
export ANTHROPIC_API_KEY=your_api_key_here

You can also specify API keys in your configuration file:

providers:
  - id: openai:gpt-4o-mini
    config:
      apiKey: your_api_key_here

Custom Integrations

promptfoo supports several types of custom integrations:

1. File-based providers:

   providers:
     - file://path/to/provider_config.yaml
   

2. JavaScript providers:

   providers:
     - file://path/to/custom_provider.js
   

3. Python providers:

   providers:
     - id: file://path/to/custom_provider.py
   

4. HTTP/HTTPS API:

   providers:
     - id: https://api.example.com/v1/chat/completions
       config:
         headers:
           Authorization: 'Bearer your_api_key'
   

5. WebSocket:

   providers:
     - id: ws://example.com/ws
       config:
         messageTemplate: '{"prompt": "{{prompt}}"}'
   

6. Custom scripts:

   providers:
     - 'exec: python chain.py'
   

Common Configuration Options

Many providers support these common configuration options:

• temperature: Controls randomness (0.0 to 1.0)
• max_tokens: Maximum number of tokens to generate
• top_p: Nucleus sampling parameter
• frequency_penalty: Penalizes frequent tokens
• presence_penalty: Penalizes new tokens based on presence in text
• stop: Sequences where the API will stop generating further tokens

Example:

providers:
  - id: openai:gpt-4o-mini
    config:
      temperature: 0.7
      max_tokens: 150
      top_p: 0.9
      frequency_penalty: 0.5
      presence_penalty: 0.5
      stop: ["\n", 'Human:', 'AI:']