promptfoo/examples/redteam-chatbot

Redteaming a Multi-turn Chatbot

Introduction

This example demonstrates how to test a stateless chatbot for security vulnerabilities using promptfoo's multi-turn strategies. It includes a Node.js Express server that accepts a conversation history in OpenAI format and returns a response in the same format. It leverages promptfoo's goat and crescendo strategies for multi-turn redteaming. You can learn more about configuring these strategies here.

Setup

Installation

1. Install dependencies:

npm install

2. Set your OpenAI API key:

export OPENAI_API_KEY=your-api-key-here

3. Start the server:

npm start

Running Tests

# Generate test cases
promptfoo redteam generate

# Execute evaluation
promptfoo redteam eval

# View results
promptfoo view

Node.js Webserver Example Usage

Single Message Request

curl -X POST http://localhost:2345/chat \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer your-token-here" \
    -d '{
        "api_provider": "openai",
        "chat_history": [
            {"role": "user", "content": "Tell me about your turboencabulator models"}
        ]
    }'

Multi-turn Conversation

curl -X POST http://localhost:2345/chat \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer your-token-here" \
    -d '{
        "api_provider": "openai",
        "chat_history": [
            {"role": "user", "content": "Tell me about your turboencabulator models"},
            {"role": "assistant", "content": "TurboTech offers several turboencabulator models..."},
            {"role": "user", "content": "What maintenance does it require?"}
        ]
    }'