nirbarazida/promptfoo

Will b9906a24f1

feat(redteam): mp plugin (#3785)

3 months ago

Commits

adaline-gateway

95727630df

chore(providers): make adaline a peer dependency (#2833)

6 months ago

amazon-bedrock

e19b816373

feat(providers): Add support for Amazon nova-sonic (#3713)

3 months ago

amazon-sagemaker

fcf6ac48f9

feat(providers): add support for Amazon SageMaker (#3413)

5 months ago

assertion-scoring-override

42ab66b096

feat(assertions): add custom assertion scoring functions (#3142)

6 months ago

assistant-cli

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

azure-openai

c6024f512f

chore(providers): improve support for azure reasoning models and update documentation (#3332)

5 months ago

azure-openai-assistant

2e0e636bfb

docs(azure-example): update assistant prompts and test cases (#3529)

4 months ago

bedrock-nova-sonic

e19b816373

feat(providers): Add support for Amazon nova-sonic (#3713)

3 months ago

cerebras

3727ff5c37

feat(providers): add cerebras provider (#3814)

3 months ago

claude-thinking

02af9d6e5b

refactor(providers): split anthropic provider into modular components (#3406)

5 months ago

claude-vision

f14f76e927

fix(providers): add regional model support to bedrock (#2354)

8 months ago

claude-vs-gpt

02af9d6e5b

refactor(providers): split anthropic provider into modular components (#3406)

5 months ago

claude-vs-gpt-image

02af9d6e5b

refactor(providers): split anthropic provider into modular components (#3406)

5 months ago

cloudflare-ai

2ac593c9cb

chore: Fix formatting and add style check to CI (#872)

1 year ago

cohere

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

cohere-benchmark

02af9d6e5b

refactor(providers): split anthropic provider into modular components (#3406)

5 months ago

colab-notebook

6bbaf876e7

docs(press): add dedicated press page (#2990)

6 months ago

csv-metadata

6498390323

chore: Update csv docs and trim whitespace for keys in CSV test files (#3571)

4 months ago

custom-grader-csv

eb47d2d8a8

fix: support javascript files in csv assertions file:// protocol (#2723)

7 months ago

custom-grading-prompt

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

custom-prompt-function

0c58ee772e

feat(prompts): add support for loading prompts from CSV files (#3542)

4 months ago

custom-provider

e41f885740

feat: support typescript custom providers (#2285)

8 months ago

custom-provider-embeddings

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

custom-provider-mjs

e41f885740

feat: support typescript custom providers (#2285)

8 months ago

custom-provider-typescript

e41f885740

feat: support typescript custom providers (#2285)

8 months ago

cyberseceval

866a6876d2

chore: add english language cyberseceval (#2561)

7 months ago

dbrx-benchmark

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

deepseek-r1-vs-openai-o1

7ef3b75451

chore(testCases): add HuggingFace Hub token support for dataset access (#3063)

6 months ago

docker-code-generation-sandbox

02af9d6e5b

refactor(providers): split anthropic provider into modular components (#3406)

5 months ago

donotanswer

2ef5c77602

docs: donotanswer example (#3780)

4 months ago

dynamic-var

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

errors-vs-failures

8afa0ffe8b

chore: separate errors from assert failures (#2214)

8 months ago

extension-api

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

external-provider-config

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

f-score

4426dc8290

docs: add F-score example (#2198)

9 months ago

function-tools-callback

f2d23ac32f

fix(providers): openai function tool callbacks handling (#1648)

11 months ago

g-eval

e822ec6a6e

feat: add G-Eval assertion (#2436)

8 months ago

gemma-vs-llama

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

gemma-vs-mistral

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

getting-started

0e6f688c63

docs: improve getting started guide (#3065)

6 months ago

github-action

2ac593c9cb

chore: Fix formatting and add style check to CI (#872)

1 year ago

golang-provider

b9357683eb

fix(go-provider): resolve CallApi redeclaration issue (#3414)

5 months ago

google-aistudio-gemini

2b5d4878b8

chore: Increase max outuput tokens for `google:gemini-2.5-pro-exp-03-25` to 2048 in Gemini example (#3753)

4 months ago

google-aistudio-tools

3b8baeffcb

feat(providers): support Google Search grounding (#3800)

3 months ago

google-live

f9b77ee9ff

fix(providers): output json rather than string from google live provider (#3703)

4 months ago

google-sheets

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

google-vertex

3b8baeffcb

feat(providers): support Google Search grounding (#3800)

3 months ago

google-vertex-tools

bdca86b2e1

fix: google vertex ais output format (#3660)

4 months ago

google-vertex-tools-callback

e4b232af24

feat(providers): add support for execution of function/tool callbacks in Vertex provider (#3215)

5 months ago

gpt-4o-temperature-comparison

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

gpt-4o-vs-4o-mini

e5122a89d1

docs(examples): Improves instructions for running 4o vs. 4o mini example (#3474)

5 months ago

groq

f9b389314c

chore(providers): update Groq documentation with latest models (#2733)

7 months ago

harmbench

6bbaf876e7

docs(press): add dedicated press page (#2990)

6 months ago

headless-browser

841a40dcda

feat(providers): add transformRequest to HTTP provider. Rename responseParser to transformResponse (#2228)

8 months ago

http-provider

ded1f3ef7c

feat: add HTTP provider configuration generator (#2409)

8 months ago

http-provider-auth-signature

482cae38b7

chore: improve http signature setup (#2779)

7 months ago

huggingface-dataset

f77a390ef9

feat: add support for huggingface datasets (#2497)

8 months ago

huggingface-dataset-factuality

3e6b2eeb3c

feat(assertions): update factuality grading prompt to improve compatibility across many different providers (#3408)

5 months ago

huggingface-hate-speech-detection

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

huggingface-inference-endpoint

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

huggingface-pii

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

huggingface-similarity

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

ibm-granite-vs-llama

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

image-classification

e1b44fbd73

chore(examples): modernize image classification example (#2197)

9 months ago

javascript-assert-external

1ca234eb25

feat(assertions): add support for specifying function names in external assertions (#2548)

7 months ago

javascript-test-cases

ab2de8cce4

feat(testcases): add support for loading dynamic test cases from Python and Javascript/Typescript files (#2993)

6 months ago

jest-integration

5fb89d63f2

chore(deps): update vitest to resolve CVE issues (#3016)

6 months ago

js-config

2a5d88c4b7

chore(openai): update model list (#1219)

1 year ago

json-output

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

jsonl-test-cases

b0d1e2985a

fix: Support for jsonl test cases (#3533)

4 months ago

lambdalabs

598e71c089

feat(providers): add Lambda Labs integration (#3601)

4 months ago

langchain-python

305a5e62be

chore(deps): bump langchain from 0.1.14 to 0.3.0 in /examples/langchain-python (#2035)

9 months ago

llama-cpp

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

llama-gpt-comparison

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

lm-studio

841a40dcda

feat(providers): add transformRequest to HTTP provider. Rename responseParser to transformResponse (#2228)

8 months ago

mistral

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

mistral-llama-comparison

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

moderation

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

multiple-configs

98b8159de3

docs: update examples and docs to use gpt-4o and gpt-4o-mini models (#1323)

1 year ago

multiple-translations

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

multiple-translations-scenarios

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

multiple-turn-conversation

a397e11e98

feat(eval): support more than one multi-turn conversations in the same eval with conversationId meta field (#2360)

8 months ago

multishot

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

named-metrics

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

node-module-package

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

node-package

21f4d639b2

docs: update model references from gpt-3.5-turbo to gpt-4o-mini (#1655)

11 months ago

node-package-typescript

ad33256d02

chore: enforce consistent type imports (#1341)

1 year ago

nunjucks-custom-filters

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

ollama-comparison

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

openai-audio

90f99d8a7b

feat(providers): add support for gpt-4o-audio-preview (#3302)

5 months ago

openai-chat-history

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

openai-dalle-images

eecb2d4327

chore(providers): refactor OpenAI image provider to remove OpenAI Node SDK dependency (#3245)

5 months ago

openai-eval-factuality

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

openai-function-call

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

openai-multiline-yaml

76797692e5

feat(schema): add yaml schema validation to config files (#1990)

10 months ago

openai-realtime

50676d0d5f

feat(providers): add support for openai realtime api (#3383)

5 months ago

openai-responses

63dea63a93

feat(providers): add support for o3 and o4-mini reasoning model (#3727)

4 months ago

openai-structured-output

cbebe2026f

chore(providers): add openai gpt-4o-2024-11-20 to supported models (#2203)

9 months ago

openai-tools-call

63dea63a93

feat(providers): add support for o3 and o4-mini reasoning model (#3727)

4 months ago

openai-vision

63dea63a93

feat(providers): add support for o3 and o4-mini reasoning model (#3727)

4 months ago

Comments

Use AWS S3 as storage!

Specify your S3 bucket

Access key (If needed)

Congratulations!

Use Google Cloud Storage!

Specify your Google Storage bucket

Service Account Key

Congratulations!

Use Azure Cloud Storage!

Specify your Azure Storage bucket

Access key (If needed)

Congratulations!

Use any S3 compatible storage!

Specify your S3 bucket

Access key (If needed)

Congratulations!

nirbarazida / promptfoo mirror of https://github.com/promptfoo/promptfoo

Comments

Use AWS S3 as storage!

Specify your S3 bucket

Access key (If needed)

Congratulations!

Use Google Cloud Storage!

Specify your Google Storage bucket

Service Account Key

Congratulations!

Use Azure Cloud Storage!

Specify your Azure Storage bucket

Access key (If needed)

Congratulations!

Use any S3 compatible storage!

Specify your S3 bucket

Access key (If needed)

Congratulations!

nirbarazida
/
promptfoo
mirror of https://github.com/promptfoo/promptfoo