Test the preview with AI

Goal

When a preview is deployed, the work item offers Test the preview with AI with the preview's address filled in. Claude Code reads the work item, writes Playwright tests for its acceptance criteria and runs them against the preview. Then the workflow runs the tests once more itself and reports the count. If they pass, release managers get Promote to production. If they fail, the work item offers Run agent again with the failing tests already written in.

Pass or fail comes from that last test run, not from what the AI says. The tests are saved with the run, so anyone can read what was checked.

What you need

  • Cuelight installed and GitHub connected
  • A repository secret ANTHROPIC_API_KEY
  • Playwright in the repository (npm init playwright@latest), with testDir: './tests' and use: { baseURL: process.env.BASE_URL } in playwright.config.ts
  • The deploy-preview and promote-production actions from Approve after an agent's pull request, and run-agent-again from Run Claude Code or Codex from Jira. Cuelight refuses a report that offers an action you haven't defined.

Actions

[
  {
    "id": "ai-tests",
    "label": "Test the preview with AI",
    "type": "github_workflow",
    "availability": "next-only",
    "github": {
      "workflow": "ai-tests.yml",
      "inputs": {
        "issue_key": "{{issue.key}}",
        "summary": "{{issue.summary}}",
        "description": "{{issue.fields.description}}",
        "url": "{{input.url}}"
      }
    },
    "form": [
      { "name": "url", "label": "Preview URL", "type": "text", "required": true, "maxLength": 300 }
    ],
    "allow": { "groups": ["qa-team", "developers"] }
  }
]

The button is next-only: it appears when a preview deploy offers it. In deploy.yml, have a successful preview offer the tests, with the address your deploy script printed, instead of Promote:

case "$ENVIRONMENT" in
preview) echo "next=$(jq -nc --arg url "$PREVIEW_URL" '[{action: "ai-tests", inputs: {url: $url}}]')" >>"$GITHUB_OUTPUT" ;;
staging) echo "next=rollback-staging,promote-production" >>"$GITHUB_OUTPUT" ;;
production) echo "next=rollback-production" >>"$GITHUB_OUTPUT" ;;
esac

Workflow

ai-tests.yml:

name: AI tests
on:
  workflow_dispatch:
    inputs:
      issue_key: { type: string, required: true }
      summary: { type: string, required: true }
      description: { type: string, required: false }
      url: { type: string, required: true }
      cuelight_callback_url: { type: string, required: false }
      cuelight_token: { type: string, required: false }

concurrency:
  group: ai-tests-${{ inputs.issue_key }}

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - name: Check the inputs
        env:
          ISSUE_KEY: ${{ inputs.issue_key }}
          BASE_URL: ${{ inputs.url }}
        run: |
          [[ "$ISSUE_KEY" =~ ^[A-Z][A-Z0-9]+-[0-9]+$ ]] || exit 1
          # Only your own previews. Change the pattern to match their addresses.
          [[ "$BASE_URL" =~ ^https://[a-z0-9-]+\.preview\.example\.com/?$ ]] || {
            echo "Not a preview address: $BASE_URL" >&2
            exit 1
          }
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      - uses: actions/setup-node@v6
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Write and try the tests with Claude Code
        uses: anthropics/claude-code-action@v1
        with:
          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
          github_token: ${{ github.token }}
          settings: |
            { "env": { "BASE_URL": ${{ toJSON(inputs.url) }} } }
          claude_args: >-
            --max-turns 40
            --allowedTools "Read,Glob,Grep,Edit(tests/ai/**),Bash(npx playwright test *)"
          prompt: |
            Test Jira work item ${{ inputs.issue_key }}: ${{ inputs.summary }}

            Description, as plain text and cut at 1000 characters:
            ${{ inputs.description }}

            The change is deployed at BASE_URL, and playwright.config.ts uses it as baseURL.
            Write Playwright tests in tests/ai/, one test per acceptance criterion, and run them
            with npx playwright test tests/ai. Fix mistakes in your tests, but never weaken a test
            to make it pass: if the preview doesn't meet a criterion, leave that test failing.
      - name: Run the tests
        id: run
        continue-on-error: true
        env:
          BASE_URL: ${{ inputs.url }}
          PLAYWRIGHT_JSON_OUTPUT_FILE: ${{ runner.temp }}/results.json
        run: npx playwright test tests/ai --reporter=line,json
      - name: Count the results
        id: count
        env:
          BASE_URL: ${{ inputs.url }}
          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
        run: |
          results="$RUNNER_TEMP/results.json"
          passed="$(jq '.stats.expected // 0' "$results" 2>/dev/null || echo 0)"
          failed="$(jq '.stats.unexpected // 0' "$results" 2>/dev/null || echo 0)"
          again="$(jq -nc --arg url "$BASE_URL" '{action: "ai-tests", inputs: {url: $url}}')"
          if [ "$failed" -eq 0 ] && [ "$passed" -gt 0 ]; then
            status=success message="AI tests passed on the preview: $passed passed"
            next=promote-production
          elif [ "$failed" -gt 0 ]; then
            names="$(jq -r '[.. | .specs? // empty | .[] | select(.ok | not) | .title] | join("; ")' "$results")"
            feedback="$(printf 'These AI tests failed on the preview: %s. Details: %s' "$names" "$RUN_URL" | head -c 900)"
            status=failure message="AI tests failed on the preview: $failed of $((passed + failed))"
            next="$(jq -nc --arg f "$feedback" --argjson again "$again" \
              '[{action: "run-agent-again", inputs: {feedback: $f}}, $again]')"
          else
            status=failure message="No AI tests ran. The run log says why." next="[$again]"
          fi
          { echo "status=$status"; echo "message=$message"; echo "next=$next"; } >>"$GITHUB_OUTPUT"
          cp "$results" tests/ai/ 2>/dev/null || true
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: ai-tests
          path: tests/ai/
          if-no-files-found: ignore
      - name: Hand back the result
        if: always()
        uses: huseyinbuyukdere/cuelight-report@v1
        with:
          status: ${{ steps.count.outputs.status || 'failure' }}
          message: ${{ steps.count.outputs.message || 'The AI tests did not run' }}
          url: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
          next: ${{ steps.count.outputs.next }}

toJSON puts the address into the settings as a JSON string, so Claude Code's own test runs see it as BASE_URL. Edit(tests/ai/**) lets it create and change files only in that folder.

What people see

  1. QA presses Deploy preview for PR #318 on SHOP-142. When the preview is live, the panel offers Test the preview with AI, "Suggested by deploy.yml", with the address filled in.
  2. Ayşe presses it. A few minutes later the panel says "AI tests passed on the preview: 4 passed" and offers Promote to production, which only release managers can press.
  3. If a test fails, the panel says "AI tests failed on the preview: 1 of 4" and offers Run agent again, with the failing test's name and the run's link already in the form, and Test the preview with AI for after the fix.

Things to know

  • The tests are code the AI wrote and ran. It runs them on your runner. Keep allow.groups narrow, give this workflow an Anthropic key of its own with a spending limit, and keep other secrets out of it.
  • A pass means these tests passed. Read them in the run's ai-tests artifact, which the work item links to. They're as good as the acceptance criteria they came from.
  • Only your previews. The form takes any text, so the first step stops the run for an address that isn't one of your previews.
  • The description arrives as plain text, cut at 1000 characters. Put the acceptance criteria near the top, or have the workflow read the work item from Jira, as in Write the Jira description with AI.

Variations

  • Keep Promote next to the tests. Offer promote-production with ai-tests after the preview if release managers may ship without them.
  • Run your own suite too. Run npx playwright test without tests/ai in the last test step to count the tests you already have.
  • Keep the good tests. Open a pull request with tests/ai/ from a separate job when they pass, so they become part of your suite.