Connect to our Discord channel
Michael
abf1612226
Models
DVC
You can run this example with:
Annotations
npx promptfoo@latest init --example claude-vs-gpt-image
This example compares an image analysis task using:
GPT-4.1 and Claude have different prompt formats. We use custom provider functions in Python and JavaScript to dynamically format the prompt based on context about the provider. The responses are scored using llm-rubric with a vision-capable OpenAI model.
To get started, set your environment variables:
OPENAI_API_KEYANTHROPIC_API_KEYAWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEYIf you do not have access to all of these providers, simply comment out the providers you do not have access to in promptfooconfig.yaml.
Then run:
npx promptfoo@latest eval
Afterwards, you can view the results by running:
npx promptfoo@latest view
Press p or to see the previous file or, n or to see the next file
Browsing data directories saved to S3 is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!
promptfoo is now integrated with AWS S3!
Are you sure you want to delete this access key?
Browsing data directories saved to Google Cloud Storage is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!
promptfoo is now integrated with Google Cloud Storage!
Are you sure you want to delete this access key?
Browsing data directories saved to Azure Cloud Storage is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!
promptfoo is now integrated with Azure Cloud Storage!
Are you sure you want to delete this access key?
Browsing data directories saved to S3 compatible storage is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!
promptfoo is now integrated with your S3 compatible storage!
Are you sure you want to delete this access key?