Skip to main content

Create Test Sets Programmatically

tip

For a complete guide on managing test sets with the SDK (including listing, retrieving, and updating), see Managing Testsets.

Overview​

Creating test sets programmatically allows you to automate test set generation, integrate with your CI/CD pipeline, or dynamically generate test cases from existing data sources.

Creating via API​

You can create a versioned testset using the simple testset API. Find the API endpoint reference here.

Here's an example of such a call:

HTTP Request:

POST /preview/simple/testsets/

Request Body:

{
  "testset": {
    "slug": "countries-capitals",
    "name": "countries_capitals",
    "data": {
      "testcases": [
        {"data": {"country": "France", "capital": "Paris"}},
        {"data": {"country": "Germany", "capital": "Berlin"}}
      ]
    }
  }
}

Example with curl​

curl -X POST "https://cloud.agenta.ai/api/preview/simple/testsets/" \
  -H "Content-Type: application/json" \
  -H "Authorization: ApiKey YOUR_API_KEY" \
  -d '{
    "testset": {
      "slug": "my-test-set",
      "name": "my_test_set",
      "data": {
        "testcases": [
          {"data": {"input": "Hello", "expected": "Hi there!"}},
          {"data": {"input": "How are you?", "expected": "I am doing well!"}}
        ]
      }
    }
  }'

Creating via SDK​

import asyncio
import agenta as ag

ag.init(host="https://cloud.agenta.ai", api_key="your-api-key")

async def main():
    # Create test set data
    csvdata = [
        {"country": "France", "capital": "Paris"},
        {"country": "Germany", "capital": "Berlin"},
        {"country": "Spain", "capital": "Madrid"},
    ]

    # Create the testset (returns a TestsetRevision)
    testset = await ag.testsets.acreate(
        name="countries_capitals",
        data=csvdata,
    )

    testset_revision_id = testset.id
    print(f"Created testset revision with ID: {testset_revision_id}")

asyncio.run(main())

Next steps​