> ## Documentation Index
> Fetch the complete documentation index at: https://docs.devhelm.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Test webhooks and email in CI

> Run webhook and email tests in GitHub Actions with one inbox per job, a path and address per test, teardown, and a shared wait budget

Give each CI job its own webhook inbox, give each test its own path and email address, and delete what the job created before it exits.

> [accordion] Prerequisites
>
> * A test suite in Vitest, Playwright, or pytest
> * A system under test whose webhook URL and recipient address you can set per test
> * A DevHelm API key and workspace ID:
>
> Export an API key. When the organization has one workspace, leave the workspace unset: the API uses that workspace, and the CLI and SDKs omit `x-phelm-workspace-id`. When it has more than one, export the workspace ID as well.
>
> ```bash theme={null}
> export DEVHELM_API_TOKEN=dh_live_xxxxxxxx
> # Only when the organization has more than one workspace:
> export DEVHELM_WORKSPACE_ID=<workspace-id>
> ```
>
> The CLI and Python SDK read `DEVHELM_API_TOKEN`. The TypeScript SDK takes the key in its constructor and reads `DEVHELM_WORKSPACE_ID` when it is set. With curl, send the workspace header only when you set the variable:
>
> ```bash theme={null}
> -H "Authorization: Bearer $DEVHELM_API_TOKEN" \
> -H "x-phelm-workspace-id: $DEVHELM_WORKSPACE_ID"
> ```

## Plan the isolation

Parallel jobs that share an inbox see each other's events, and an old event can satisfy a new test's wait. Isolate at the level each resource allows:

| Resource | One per | Why |
| - | - | - |
| Webhook inbox | CI job | Inboxes are capped per organization: 3 on Free, 10 on Developer |
| Path on the inbox URL | Test | Wait matches the path with `pathPrefix` |
| Email address | Test | Any local part is an inbox, and addresses have no cap |
| Email domain | Organization | Every run shares the assigned domain. Free and Developer allow one |

Inboxes in use at once are matrix shards × concurrent workflow runs, plus any that developers keep. Over the cap, the create step fails with `403 INBOUND_WEBHOOK_INBOX_LIMIT_REACHED`.

## Store the secrets

Create an API key used only by CI, so you can revoke it on its own (see [API keys](https://docs.devhelm.io/platform/api-keys.md)). Add it and the workspace ID as repository secrets:

```bash theme={null}
gh secret set DEVHELM_API_TOKEN
gh secret set DEVHELM_WORKSPACE_ID
```

Set `DEVHELM_WORKSPACE_ID` when the organization has more than one workspace, and pass it on every step that calls DevHelm. With one workspace, leave it unset.

> [!WARNING]
>
> GitHub does not pass secrets to `pull_request` runs from forks or Dependabot. The workflow below skips the job on fork PRs. Do not switch to `pull_request_target` to get secrets into those runs: checking out the fork's code there lets that code read the key.

## The workflow

Create `.github/workflows/e2e.yml`:

```yaml theme={null}
name: e2e
on:
  push:
    branches: [main]
  pull_request:

jobs:
  e2e:
    # Fork PRs get no secrets. Skip instead of failing.
    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    timeout-minutes: 20
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npm install -g devhelm

      - name: Create webhook inbox
        env:
          DEVHELM_API_TOKEN: ${{ secrets.DEVHELM_API_TOKEN }}
          DEVHELM_WORKSPACE_ID: ${{ secrets.DEVHELM_WORKSPACE_ID }}
        run: |
          devhelm inboxes create \
            --name "ci-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.shard }}" \
            -o json > inbox.json
          url=$(jq -r .httpUrl inbox.json)
          echo "::add-mask::$url"
          echo "WEBHOOK_URL=$url" >> "$GITHUB_ENV"
          echo "DEVHELM_INBOX_ID=$(jq -r .id inbox.json)" >> "$GITHUB_ENV"
          rm inbox.json

      - name: Test
        env:
          DEVHELM_API_TOKEN: ${{ secrets.DEVHELM_API_TOKEN }}
          DEVHELM_WORKSPACE_ID: ${{ secrets.DEVHELM_WORKSPACE_ID }}
        run: npx vitest run --shard=${{ matrix.shard }}/2

      - name: Delete webhook inbox
        if: always() && env.DEVHELM_INBOX_ID != ''
        env:
          DEVHELM_API_TOKEN: ${{ secrets.DEVHELM_API_TOKEN }}
          DEVHELM_WORKSPACE_ID: ${{ secrets.DEVHELM_WORKSPACE_ID }}
        run: devhelm inboxes delete "$DEVHELM_INBOX_ID" --yes
```

* The `ci-` name prefix with run ID, attempt, and shard lets a sweeper find leftovers.
* The URL is masked before it reaches `$GITHUB_ENV`: anyone with it can write to the inbox, and logs on public repositories are public.
* The key is set per step, so `npm ci` and its install scripts never see it.
* `if: always()` runs the delete when tests fail or the run is cancelled. Deleting an inbox also deletes its events and raw bodies.

For Playwright, run `npx playwright test --shard=${{ matrix.shard }}/2`; for pytest, `pytest` or `pytest -n auto`. The create and delete steps stay the same.

## Write the tests

Each test builds its own path, registers `<inbox URL><path>` with the system under test, and waits with `pathPrefix`. The path keeps other tests out of the 60 second lookback. `app-client` stands for your own code.

```typescript order-webhooks.test.ts theme={null}
import { Devhelm } from "@devhelm/sdk";
import { expect, test } from "vitest";
import { createSubscription, placeOrder } from "./app-client";

const client = new Devhelm({ token: process.env.DEVHELM_API_TOKEN! });
const inboxId = process.env.DEVHELM_INBOX_ID!;
const webhookUrl = process.env.WEBHOOK_URL!;
const runId = process.env.GITHUB_RUN_ID ?? "local";

test("placing an order sends order.created", async () => {
  const path = `/${runId}/${crypto.randomUUID()}`;
  await createSubscription({ url: webhookUrl + path, events: ["order.created"] });

  const order = await placeOrder({ sku: "sku-123", quantity: 2 });

  const event = await client.inboxes.wait(inboxId, {
    timeoutMs: 30_000,
    http: { method: "POST", pathPrefix: path },
  });
  expect(event.json()).toMatchObject({ type: "order.created", data: { id: order.id } });
}, 45_000);
```

```python test_order_webhooks.py theme={null}
import os
import uuid
from typing import Any, cast

import pytest
from devhelm import Devhelm

from app_client import create_subscription, place_order

client = Devhelm()
INBOX_ID = os.environ["DEVHELM_INBOX_ID"]
WEBHOOK_URL = os.environ["WEBHOOK_URL"]
RUN_ID = os.environ.get("GITHUB_RUN_ID", "local")


@pytest.mark.timeout(45)
def test_placing_an_order_sends_order_created() -> None:
    path = f"/{RUN_ID}/{uuid.uuid4()}"
    create_subscription(url=WEBHOOK_URL + path, events=["order.created"])

    order = place_order(sku="sku-123", quantity=2)

    event = client.inboxes.wait(
        INBOX_ID,
        timeout_ms=30_000,
        http={"method": "POST", "pathPrefix": path},
    )
    body = cast(dict[str, Any], event.json())
    assert body["type"] == "order.created"
    assert body["data"]["id"] == order["id"]
```

Set the runner timeout above `timeoutMs`. A longer runner timeout does not extend the HTTP call, and a shorter one kills the test before DevHelm answers. In pytest the timeout marker needs `pytest-timeout`.

### Email: one address per test

`address()` returns a fresh address on the workspace's assigned domain. Clear it when the test ends. There is no DevHelm Playwright package: a Playwright test calls the SDK the same way a Vitest test does.

```typescript signup.spec.ts theme={null}
import { Devhelm } from "@devhelm/sdk";
import { expect, test } from "@playwright/test";

const client = new Devhelm({ token: process.env.DEVHELM_API_TOKEN! });

test("sign-up sends a 6-digit code", async ({ page }) => {
  test.setTimeout(90_000); // above the 60 s wait
  const address = await client.email.address({ label: "ci-signup" });
  try {
    await page.goto("https://staging.example.com/signup");
    await page.getByLabel("Email").fill(address.email);
    await page.getByRole("button", { name: "Create account" }).click();

    const message = await address.wait({ timeoutMs: 60_000 });
    expect(message.otp?.[0]?.value).toMatch(/^\d{6}$/);
  } finally {
    await address.clear();
  }
});
```

In pytest, make the address a fixture that yields `client.email.address(label=...)` and calls `clear()` after the test. When CI has no mail provider, a test can inject the message with `address.receive()` instead; both paths run the same parsing. The wait response includes `text`, `html`, `otp`, and `links`. The full flow, including links, is in [Test a sign-up email flow](https://docs.devhelm.io/guides/test-signup-email-flow.md).

## Teardown

| Resource | Who removes it | How |
| - | - | - |
| Webhook inbox | The workflow's last step | `devhelm inboxes delete <inbox-id> --yes` |
| Email address | Each test | `address.clear()`, or `devhelm email clear --to <address> --yes` |
| Assigned email domain | Nobody | Keep it |

Never delete the assigned domain in teardown. Every concurrent run uses it, the next `address()` call would create a domain with a new name, and on Free or Developer a second domain fails with `403 INBOUND_EMAIL_DOMAIN_LIMIT_REACHED`.

Retention does not clean up for you. DevHelm does not delete captured data on a schedule today, so events and messages stay until teardown removes them. See [Security and data](https://docs.devhelm.io/testing/security.md#retention-and-cleanup).

### Sweep inboxes from killed runs

A lost or timed-out runner skips the delete step, and its inbox keeps a slot. Run a sweeper on a schedule or at the start of each job, with a cutoff longer than your longest job:

```typescript sweep-inboxes.ts theme={null}
import { Devhelm } from "@devhelm/sdk";

const client = new Devhelm({ token: process.env.DEVHELM_API_TOKEN! });
const cutoff = Date.now() - 60 * 60 * 1000; // 1 hour

for (const inbox of await client.inboxes.list({ search: "ci-" })) {
  // search is a case-insensitive substring match, so check the prefix again
  if (inbox.name.startsWith("ci-") && Date.parse(inbox.createdAt) < cutoff) {
    await client.inboxes.delete(inbox.id);
  }
}
```

## Parallel runs and the wait budget

All wait calls in an organization share 40 per minute: webhook and email waits, every workspace, every CI job, and every developer running tests locally. Over the limit, a wait returns `429 RATE_LIMITED` with `Retry-After`. A suite with 30 waits fits in one minute; two shards of it plus a second PR in flight use 120 and get throttled.

* Wait once per test, then assert everything on the returned event or message.
* Use one long `timeoutMs` (up to 120000), not a loop of short waits. Each call counts.
* Limit concurrent runs with a GitHub `concurrency` group when the inbox cap or the wait budget is tight:

```yaml theme={null}
concurrency:
  group: devhelm-e2e
  cancel-in-progress: false
```

### Back off on 429

Neither SDK retries on its own. Wrap waits in a helper that sleeps for `retryAfter` and tries again. Retrying the same wait is safe, because wait does not consume what it returns.

```typescript theme={null}
import { DevhelmRateLimitError } from "@devhelm/sdk";

export async function withWaitBudget<T>(call: () => Promise<T>, attempts = 3): Promise<T> {
  for (let attempt = 1; ; attempt++) {
    try {
      return await call();
    } catch (err) {
      if (!(err instanceof DevhelmRateLimitError) || attempt >= attempts) throw err;
      await new Promise((resolve) => setTimeout(resolve, (err.retryAfter ?? 5) * 1000));
    }
  }
}

const event = await withWaitBudget(() =>
  client.inboxes.wait(inboxId, { timeoutMs: 30_000, http: { pathPrefix: path } }),
);
```

In Python, catch `DevhelmRateLimitError` and sleep for `err.retry_after` the same way. Size the runner timeout for the retries too: `attempts × (timeoutMs + retryAfter)`.

## Monthly quota

Every captured webhook request and every received or injected email counts toward one monthly event cap per organization: 300 on Free, 3,000 on Developer, 100,000 on Team. A suite that sends 30 events and runs 20 times a day needs 18,000 a month.

At the cap, webhook senders and inject get `403 INBOUND_EVENTS_MONTHLY_LIMIT_REACHED` and nothing is stored. The counter updates asynchronously, so a burst can go slightly over first, and an inject accepted right at the cap can be dropped, which makes its wait return `408`. See [Limits, quotas, and errors](https://docs.devhelm.io/testing/limits.md).

## Troubleshooting

> [accordion] Create step fails with 403 INBOUND\_WEBHOOK\_INBOX\_LIMIT\_REACHED
>
> The organization is at its inbox cap, counting every workspace. Delete leftovers from killed runs (the sweeper does this), reduce matrix shards, or add a `concurrency` group.

> [accordion] Wait returns 408 in CI but passes locally
>
> * `receivedAfter` was set after the trigger, so a fast event is older than the cursor. Omit it when the path is unique to the test.
> * The test's `pathPrefix` does not match the path the system under test called. Paths are case-sensitive.
> * The system under test is slower in CI. Raise `timeoutMs` and the runner timeout with it.
> * The monthly cap was reached. Check the sender's response for `403`.

> [accordion] The job is skipped on pull requests
>
> The PR comes from a fork, and fork runs get no secrets. Run the suite after merge, or have a maintainer push the branch to the main repository.

## Next steps

> [card] [Wait for a webhook](https://docs.devhelm.io/testing/webhooks/assert-in-tests.md)
>
> Wait semantics, the Nth event, and negative assertions.

> [card] [Wait for an email](https://docs.devhelm.io/testing/email/wait.md)
>
> Addresses, `subjectContains`, and repeat sends.

> [card] [Limits, quotas, and errors](https://docs.devhelm.io/testing/limits.md)
>
> Every limit and error code with its fix.

> [card] [Security and data](https://docs.devhelm.io/testing/security.md)
>
> Keys in CI, captured headers, and cleanup.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.