Regression Testing MCP Apps, ChatGPT Apps, and Claude Connectors (August 2026)

TL;DR: Protect the contracts between your MCP server, host, and UI. Test tools/list, tools/call, resources/list, and resources/read; validate structuredContent against a runtime schema; render deterministic states in ChatGPT and Claude host replicas; compare screenshots across the modes you support; and keep a small live-host suite for OAuth, tool selection, and production loading. Run the deterministic suite on every change and review every baseline update.

An MCP App can pass unit tests and still fail for users. A renamed field can leave a blank card, a changed resource URI can stop the host from loading the UI, a CSS update can overflow only in fullscreen, and a host update can expose an assumption your local code never checked.

Regression testing turns behavior that worked yesterday into an executable contract. The goal is not to freeze the app. The goal is to make intentional changes easy to review and accidental changes hard to ship.

What Can Regress in an MCP App?

An MCP App crosses more boundaries than a normal web page, so its regression suite needs more than component tests.

1. Tool discovery changes the tool name, description, input schema, output schema, annotations, or visibility.
2. Tool result changes affect content, structuredContent, _meta, error status, or cancellation behavior.
3. UI resource discovery breaks when the tool no longer points to the right ui:// resource, the resource MIME type changes, or required CSP metadata disappears.
4. Rendering breaks when the resource loads but no longer shows the right data, state, controls, or accessible names.
5. Host behavior varies because ChatGPT, Claude, and other MCP Apps hosts expose different capabilities, shell styles, safe areas, and display modes.
6. Production integration covers OAuth, the public MCP endpoint, resource caching, and a real model’s tool selection.

Treat these as separate test boundaries. A browser screenshot cannot tell you why a resource did not load, and a protocol assertion cannot tell you that a button is clipped.

Start with the MCP Contract

Protocol tests are the fastest place to catch a breaking change because they do not need to render an iframe. Test the public data a host sees, not private handler implementation details.

With the mcp fixture from sunpeak/test, you can inspect tool and resource discovery and call the real server:

import { test, expect } from 'sunpeak/test';

test('weather tool keeps its public MCP contract', async ({ mcp }) => {
  const tools = await mcp.listTools();
  const weather = tools.find((tool) => tool.name === 'get-weather');

  expect(weather).toBeDefined();
  expect(weather?.inputSchema).toMatchObject({
    type: 'object',
    required: ['city'],
    properties: {
      city: { type: 'string' },
    },
  });
  expect(weather?.outputSchema).toMatchObject({
    type: 'object',
    required: ['city', 'temperature', 'forecast'],
  });
  expect(weather?._meta).toMatchObject({
    ui: { resourceUri: 'ui://weather/card.html' },
  });
});

The current MCP Apps metadata shape nests UI fields under _meta.ui. The old flat _meta["ui/resourceUri"] key is deprecated, although compatibility helpers may still emit it for older hosts. Test the current field so a refactor does not silently break new hosts.

Then test the linked resource:

test('weather UI resource stays readable', async ({ mcp }) => {
  const resources = await mcp.listResources();
  const card = resources.find((resource) => resource.uri === 'ui://weather/card.html');

  expect(card).toBeDefined();
  expect(card?.mimeType).toContain('text/html');

  const html = await mcp.readResource('ui://weather/card.html');
  expect(html).toContain('<html');
  expect(html).toContain('weather-card-root');
});

This catches broken tool-to-resource links, renamed resource URIs, incorrect MIME types, empty production bundles, and missing root markup before a browser test produces a vague timeout.

Validate Structured Content at Runtime

TypeScript types disappear at runtime. A handler can claim to return WeatherOutput while an API response or serialization path produces null, a string instead of a number, or a missing array.

Define one runtime schema and use it in the handler, UI types, and tests:

import { z } from 'zod';

export const WeatherOutput = z.object({
  city: z.string(),
  temperature: z.number(),
  unit: z.enum(['celsius', 'fahrenheit']),
  conditions: z.string(),
  forecast: z.array(
    z.object({
      day: z.string(),
      high: z.number(),
      low: z.number(),
    })
  ),
});

export type WeatherOutput = z.infer<typeof WeatherOutput>;
export const outputSchema = WeatherOutput.shape;

Use the same schema in a protocol regression test:

import { test, expect } from 'sunpeak/test';
import { WeatherOutput } from '../../src/types/weather';

test('get-weather returns data the UI can render', async ({ mcp }) => {
  const result = await mcp.callTool('get-weather', { city: 'Portland' });

  expect(result.isError).toBeFalsy();

  const parsed = WeatherOutput.safeParse(result.structuredContent);
  expect(parsed.success).toBe(true);

  if (parsed.success) {
    expect(parsed.data.city).toBe('Portland');
    expect(parsed.data.forecast.length).toBeGreaterThan(0);
  }
});

This approach allows additive fields while rejecting missing or mistyped fields the UI needs. Add exact value assertions only for business rules that must stay stable. Overly broad snapshots of raw API data tend to fail on timestamps, IDs, ordering, and harmless additions.

Also test errors as contracts. A tool that used to return isError: true with useful text should not start throwing an unhandled exception or returning a success-shaped empty object.

Make Test Data Deterministic

Regression comparisons are useful only when the same input produces the same state. Avoid building screenshot tests around live APIs, current time, random IDs, or mutable production accounts.

A sunpeak simulation can pin a tool result and mock app-initiated server tools:

{
  "tool": "get-weather",
  "toolResult": {
    "content": [{ "type": "text", "text": "Weather for Portland" }],
    "structuredContent": {
      "city": "Portland",
      "temperature": 72,
      "unit": "fahrenheit",
      "conditions": "Partly cloudy",
      "forecast": [
        { "day": "Mon", "high": 75, "low": 58 },
        { "day": "Tue", "high": 68, "low": 55 }
      ]
    }
  },
  "serverTools": {
    "save-location": {
      "content": [{ "type": "text", "text": "Saved." }],
      "structuredContent": { "saved": true }
    }
  }
}

Create fixtures for states that have caused bugs or carry risk:

• normal, empty, partial, and long data
• loading, error, retry, and cancellation
• unauthorized and insufficient-scope responses
• app-only tool success and failure
• narrow layouts, long translated text, and missing optional fields

Use real server calls in protocol and integration tests. Use simulations when the test needs a stable UI state. Keeping both prevents a fixture from hiding a server regression and prevents a live dependency from making a visual test flaky.

Run E2E Tests in Each Host

The official MCP Apps project includes a basic-host reference implementation for local testing. A host replica can go further by reproducing host shell, theme, capability, and iframe behavior.

sunpeak’s Playwright config creates ChatGPT and Claude projects, so each E2E test runs once per host:

// playwright.config.ts
import { defineConfig } from 'sunpeak/test/config';

export default defineConfig();

For an existing HTTP or stdio MCP server written in any language, point the same config at the server:

import { defineConfig } from 'sunpeak/test/config';

export default defineConfig({
  server: 'http://localhost:8000/mcp',
});

Render a deterministic fixture by omitting tool input, then assert behavior inside the app iframe:

import { test, expect } from 'sunpeak/test';

test('weather card renders its stable content', async ({ inspector }) => {
  const result = await inspector.renderTool('get-weather', undefined, {
    theme: 'light',
  });
  const app = result.app();

  await expect(app.getByRole('heading', { name: 'Portland' })).toBeVisible();
  await expect(app.getByText('72')).toBeVisible();
  await expect(app.getByRole('button', { name: 'Save location' })).toBeEnabled();
});

Assert user-visible behavior and accessible names. Selectors based on component internals, generated class names, or DOM depth create noisy failures during harmless refactors.

Test Host Differences Deliberately

Cross-host coverage does not mean every host must behave identically. Capability support varies, so encode the expected difference:

test('weather card works in PiP where supported', async ({ inspector }) => {
  test.skip(inspector.host === 'claude', 'Claude does not support PiP');

  const result = await inspector.renderTool('get-weather', undefined, {
    displayMode: 'pip',
  });

  await expect(result.app().getByText('Portland')).toBeVisible();
});

Do not branch on an exact remote client’s name for security or authorization. Client identifiers can vary and are unauthenticated. Use negotiated capabilities for behavior, and use client information only for telemetry or coarse compatibility handling.

Build a Risk-Based Rendering Matrix

Running every data state across every host, theme, display mode, device, and locale creates a suite that is slow to review. Use tiers.

• On every change, run protocol contracts, unit tests, and a small E2E matrix covering both hosts, light and dark themes, the default display mode, and critical user actions.

• On UI changes, add fullscreen, PiP where supported, phone-sized and wide viewports, safe-area insets, long content, empty data, and error states.

• Before release, run production bundles, authentication failures, app-initiated tools, cancellation, live host smoke tests, and any model tool-selection evals tied to changed names or descriptions.

Display mode belongs in the third renderTool argument:

test('weather details fit in fullscreen dark mode', async ({ inspector }) => {
  const result = await inspector.renderTool('get-weather', undefined, {
    theme: 'dark',
    displayMode: 'fullscreen',
  });
  const app = result.app();

  await expect(app.getByRole('heading', { name: 'Hourly forecast' })).toBeVisible();

  const overflow = await app
    .locator('body')
    .evaluate((body) => body.scrollWidth > body.clientWidth);
  expect(overflow).toBe(false);
});

For responsive regressions, assert overflow, clipping, focus order, and tap target behavior in addition to taking screenshots. A pixel comparison can detect movement, but it does not explain whether the page remains usable.

Add Visual Regression Tests

HTML snapshots can catch removed nodes or changed attributes. Screenshot comparisons catch spacing, color, font, wrapping, clipping, and responsive layout changes.

With sunpeak, result.screenshot() captures the app content inside the nested host iframe:

test('weather fullscreen visual baseline', async ({ inspector }) => {
  const result = await inspector.renderTool('get-weather', undefined, {
    theme: 'dark',
    displayMode: 'fullscreen',
  });

  await expect(result.app().getByText('Portland')).toBeVisible();
  await result.screenshot('weather-dark-fullscreen');
});

Normal E2E runs skip screenshot comparison. Run the visual mode explicitly:

pnpm test:visual
pnpm test:visual -- --update

Playwright notes that rendering can vary by operating system, browser version, fonts, hardware, and headless mode. Generate and compare baselines in the same container or CI image. Wait for web fonts and images, disable animation, and hide only fields that are truly volatile.

Use a documented threshold for small rendering differences, but do not set it high enough to hide real movement. A stable environment is a better fix than a broad pixel allowance.

Test State, Actions, and Failure Paths

Static cards are only one MCP App state. Interactive apps also receive partial and final tool input, tool results, host context changes, app state updates, server-tool responses, errors, and cancellation.

Write regression tests around transitions:

1. The initial tool result renders.
2. A user changes local app state.
3. The app calls an app-visible server tool.
4. Success updates the UI and keeps prior state.
5. Failure preserves recoverable state and offers retry.
6. Cancellation stops loading and does not apply a stale result.

For multi-tool apps, test the producer and consumer together. If one tool writes a filter or selection that another tool reads, validate that shared shape at runtime and cover a missing or old version of the state. The multi-tool testing guide goes deeper on those contracts.

Keep a Small Real-Host Suite

Local testing can cover many combinations without deployment, paid host accounts, or AI credits. It cannot prove the complete production path.

Use live smoke tests for:

• the public HTTPS MCP connection
• OAuth discovery, consent, refresh, and insufficient scopes
• real model selection of the expected tool
• production resource fetches and CSP
• host iframe rendering and app-to-host requests
• server logs, latency, retries, and cancellation

Claude has no separate connector staging runtime. Anthropic recommends adding the server as a custom connector because it uses the same runtime as a directory connector. OpenAI’s current ChatGPT workflow lets developers connect an MCP-backed plugin in Developer mode and test its production MCP URL before submission.

Keep the live suite narrow because it is slower and depends on external systems. Run deterministic local tests on every change, then run live tests before release and after host or protocol updates.

Run the Suite in CI

For a sunpeak project, the current template exposes separate unit, E2E, visual, live, and eval commands. The default pnpm test runs the local unit and E2E layers. Visual comparison is opt-in, while live tests and evals require external access or provider credentials.

name: MCP App regression tests

on:
  pull_request:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: pnpm
      - run: pnpm install --frozen-lockfile
      - run: pnpm exec playwright install --with-deps chromium
      - run: pnpm test
      - run: pnpm test:visual

      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: mcp-app-regression-artifacts
          path: |
            playwright-report/
            test-results/
            tests/e2e/__screenshots__/

Pin the lockfile, browser version, fonts, locale, and timezone used for screenshots. Upload the actual, expected, diff, trace, and console output on failure so a reviewer can distinguish a product regression from test infrastructure trouble.

Review Baseline Changes

An intentional UI change should update its baseline. Review the actual, expected, and diff images first, then run:

pnpm test:visual -- --update

Commit the changed baseline with the product change. Avoid bulk baseline updates because they make unrelated movement difficult to spot. When a dependency or host replica update changes many images, put that update in its own pull request and explain the expected differences.

The same rule applies to protocol snapshots and test fixtures. A changed file is evidence to review, not an error to erase.

MCP App Regression Testing Checklist

Before shipping an update, verify:

• tools/list covers names, schemas, annotations, visibility, and resource links
• tools/call results pass runtime schemas and expected error contracts
• resources/list and resources/read cover MIME type, URI, HTML, and security metadata
• deterministic fixtures cover normal, empty, long, error, unauthorized, and cancelled states
• critical interactions run against ChatGPT and Claude host replicas
• host-specific capability differences are explicit
• themes, supported display modes, and representative device sizes are covered
• visual baselines run in a pinned environment and every change is reviewed
• app state and app-initiated server tools have success and failure tests
• production bundles are tested, not only development resources
• a small live-host suite covers connection, OAuth, model selection, and resource loading
• CI saves traces, logs, screenshots, and diffs when a test fails

The sunpeak testing framework can scaffold this stack for any MCP server, including servers not built with sunpeak:

npx sunpeak test init --server http://localhost:8000/mcp
npx sunpeak test

That gives you protocol fixtures, Playwright host projects, simulations, visual baselines, live test scaffolding, and eval scaffolding in one test setup.