Test the preview with AI
Goal
When a preview is deployed, the work item offers Test the preview with AI with the preview's address filled in. Claude Code reads the work item, writes Playwright tests for its acceptance criteria and runs them against the preview. Then the workflow runs the tests once more itself and reports the count. If they pass, release managers get Promote to production. If they fail, the work item offers Run agent again with the failing tests already written in.
Pass or fail comes from that last test run, not from what the AI says. The tests are saved with the run, so anyone can read what was checked.
What you need
- Cuelight installed and GitHub connected
- A repository secret
ANTHROPIC_API_KEY - Playwright in the repository (
npm init playwright@latest), withtestDir: './tests'anduse: { baseURL: process.env.BASE_URL }inplaywright.config.ts - The
deploy-previewandpromote-productionactions from Approve after an agent's pull request, andrun-agent-againfrom Run Claude Code or Codex from Jira. Cuelight refuses a report that offers an action you haven't defined.
Actions
[
{
"id": "ai-tests",
"label": "Test the preview with AI",
"type": "github_workflow",
"availability": "next-only",
"github": {
"workflow": "ai-tests.yml",
"inputs": {
"issue_key": "{{issue.key}}",
"summary": "{{issue.summary}}",
"description": "{{issue.fields.description}}",
"url": "{{input.url}}"
}
},
"form": [
{ "name": "url", "label": "Preview URL", "type": "text", "required": true, "maxLength": 300 }
],
"allow": { "groups": ["qa-team", "developers"] }
}
]The button is next-only: it appears when a preview deploy offers it. In deploy.yml, have a
successful preview offer the tests, with the address your deploy script printed, instead of Promote:
case "$ENVIRONMENT" in
preview) echo "next=$(jq -nc --arg url "$PREVIEW_URL" '[{action: "ai-tests", inputs: {url: $url}}]')" >>"$GITHUB_OUTPUT" ;;
staging) echo "next=rollback-staging,promote-production" >>"$GITHUB_OUTPUT" ;;
production) echo "next=rollback-production" >>"$GITHUB_OUTPUT" ;;
esacWorkflow
ai-tests.yml:
name: AI tests
on:
workflow_dispatch:
inputs:
issue_key: { type: string, required: true }
summary: { type: string, required: true }
description: { type: string, required: false }
url: { type: string, required: true }
cuelight_callback_url: { type: string, required: false }
cuelight_token: { type: string, required: false }
concurrency:
group: ai-tests-${{ inputs.issue_key }}
permissions:
contents: read
jobs:
test:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Check the inputs
env:
ISSUE_KEY: ${{ inputs.issue_key }}
BASE_URL: ${{ inputs.url }}
run: |
[[ "$ISSUE_KEY" =~ ^[A-Z][A-Z0-9]+-[0-9]+$ ]] || exit 1
# Only your own previews. Change the pattern to match their addresses.
[[ "$BASE_URL" =~ ^https://[a-z0-9-]+\.preview\.example\.com/?$ ]] || {
echo "Not a preview address: $BASE_URL" >&2
exit 1
}
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: actions/setup-node@v6
with:
node-version: 22
cache: npm
- run: npm ci
- run: npx playwright install --with-deps chromium
- name: Write and try the tests with Claude Code
uses: anthropics/claude-code-action@v1
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
github_token: ${{ github.token }}
settings: |
{ "env": { "BASE_URL": ${{ toJSON(inputs.url) }} } }
claude_args: >-
--max-turns 40
--allowedTools "Read,Glob,Grep,Edit(tests/ai/**),Bash(npx playwright test *)"
prompt: |
Test Jira work item ${{ inputs.issue_key }}: ${{ inputs.summary }}
Description, as plain text and cut at 1000 characters:
${{ inputs.description }}
The change is deployed at BASE_URL, and playwright.config.ts uses it as baseURL.
Write Playwright tests in tests/ai/, one test per acceptance criterion, and run them
with npx playwright test tests/ai. Fix mistakes in your tests, but never weaken a test
to make it pass: if the preview doesn't meet a criterion, leave that test failing.
- name: Run the tests
id: run
continue-on-error: true
env:
BASE_URL: ${{ inputs.url }}
PLAYWRIGHT_JSON_OUTPUT_FILE: ${{ runner.temp }}/results.json
run: npx playwright test tests/ai --reporter=line,json
- name: Count the results
id: count
env:
BASE_URL: ${{ inputs.url }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
results="$RUNNER_TEMP/results.json"
passed="$(jq '.stats.expected // 0' "$results" 2>/dev/null || echo 0)"
failed="$(jq '.stats.unexpected // 0' "$results" 2>/dev/null || echo 0)"
again="$(jq -nc --arg url "$BASE_URL" '{action: "ai-tests", inputs: {url: $url}}')"
if [ "$failed" -eq 0 ] && [ "$passed" -gt 0 ]; then
status=success message="AI tests passed on the preview: $passed passed"
next=promote-production
elif [ "$failed" -gt 0 ]; then
names="$(jq -r '[.. | .specs? // empty | .[] | select(.ok | not) | .title] | join("; ")' "$results")"
feedback="$(printf 'These AI tests failed on the preview: %s. Details: %s' "$names" "$RUN_URL" | head -c 900)"
status=failure message="AI tests failed on the preview: $failed of $((passed + failed))"
next="$(jq -nc --arg f "$feedback" --argjson again "$again" \
'[{action: "run-agent-again", inputs: {feedback: $f}}, $again]')"
else
status=failure message="No AI tests ran. The run log says why." next="[$again]"
fi
{ echo "status=$status"; echo "message=$message"; echo "next=$next"; } >>"$GITHUB_OUTPUT"
cp "$results" tests/ai/ 2>/dev/null || true
- uses: actions/upload-artifact@v7
if: always()
with:
name: ai-tests
path: tests/ai/
if-no-files-found: ignore
- name: Hand back the result
if: always()
uses: huseyinbuyukdere/cuelight-report@v1
with:
status: ${{ steps.count.outputs.status || 'failure' }}
message: ${{ steps.count.outputs.message || 'The AI tests did not run' }}
url: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
next: ${{ steps.count.outputs.next }}toJSON puts the address into the settings as a JSON string, so Claude Code's own test runs see
it as BASE_URL. Edit(tests/ai/**) lets it create and change files only in that folder.
What people see
- QA presses Deploy preview for PR #318 on SHOP-142. When the preview is live, the panel offers Test the preview with AI, "Suggested by deploy.yml", with the address filled in.
- Ayşe presses it. A few minutes later the panel says "AI tests passed on the preview: 4 passed" and offers Promote to production, which only release managers can press.
- If a test fails, the panel says "AI tests failed on the preview: 1 of 4" and offers Run agent again, with the failing test's name and the run's link already in the form, and Test the preview with AI for after the fix.
Things to know
- The tests are code the AI wrote and ran. It runs them on your runner. Keep
allow.groupsnarrow, give this workflow an Anthropic key of its own with a spending limit, and keep other secrets out of it. - A pass means these tests passed. Read them in the run's
ai-testsartifact, which the work item links to. They're as good as the acceptance criteria they came from. - Only your previews. The form takes any text, so the first step stops the run for an address that isn't one of your previews.
- The description arrives as plain text, cut at 1000 characters. Put the acceptance criteria near the top, or have the workflow read the work item from Jira, as in Write the Jira description with AI.
Variations
- Keep Promote next to the tests. Offer
promote-productionwithai-testsafter the preview if release managers may ship without them. - Run your own suite too. Run
npx playwright testwithouttests/aiin the last test step to count the tests you already have. - Keep the good tests. Open a pull request with
tests/ai/from a separate job when they pass, so they become part of your suite.