MCP App Conformance Testing for ChatGPT Apps and Codex Connectors (August 2026)

An MCP App can pass its browser tests and still fail before the browser loads. The server may negotiate the wrong core protocol revision, publish a stale ui:// link, return the wrong MIME type, expose an app-only tool to the model, or send a host message before the View finishes its handshake.

The word “conformance” now covers several of these failures because the MCP project has an official conformance test framework, while MCP Apps have a separate extension contract and each host still owns part of the runtime.

TL;DR: Use three gates. Run the official MCP conformance suite against each core protocol revision you claim to support. Add fast MCP App contract tests for tool metadata, resources, result channels, security metadata, capability negotiation, and the View lifecycle. Then run one rendered smoke test per View in every supported host runtime. A green result in one layer says nothing about the other two.

Conformance Has Three Layers

Treat these as separate test targets:

Layer	Contract under test	Good tools	What a pass proves
Core MCP	Transport, lifecycle, JSON-RPC messages, tools, resources, prompts, authorization	Official MCP conformance runner	The client or server follows the selected core revision for the scenarios tested
MCP App	Tool-to-View linkage, HTML resource, result data, security metadata, View-to-host protocol	Protocol tests, schema validation, reference host, sunpeak	A compliant host has enough correct data to discover, load, and operate the View
Host runtime	Iframe policy, capabilities, theme, sizing, display modes, approvals, model behavior	Host replicas and live-host tests	The app works in the host implementation you plan to support

This split keeps failures readable. A wire-schema-valid failure belongs to the core wire. A missing _meta.ui.resourceUri belongs to the app server contract. A button clipped only in a narrow Claude iframe belongs to host compatibility.

Run the Official Core MCP Suite First

The official runner can test an MCP client or server. In server mode, it connects to your endpoint, sends scenario requests, records the exchange, and checks behavior against the selected specification version. It also validates instrumented JSON-RPC messages against that revision’s wire schema.

Pin the runner version in CI because its scenarios change over time. As of August 2026, 0.1.16 is the stable release:

npx -y @modelcontextprotocol/conformance@0.1.16 server \
  --url http://127.0.0.1:8000/mcp \
  --suite active \
  --spec-version 2025-11-25 \
  --output-dir .conformance/2025-11-25

The MCP 2026-07-28 revision changed the core lifecycle from the stateful initialize handshake to stateless requests with protocol, client, and capability data in _meta. The current 0.2 alpha line adds frozen requirement sets for that revision:

npx -y @modelcontextprotocol/conformance@0.2.0-alpha.11 server \
  --url http://127.0.0.1:8000/mcp \
  --requirements 2026-07-28 \
  --output-dir .conformance/2026-07-28

Keep the alpha gate nonblocking until you have reviewed its output and upgrade it deliberately. Do not point both commands at one endpoint unless that endpoint actually serves both protocol eras. Some SDKs use a different process, configuration, or URL for the modern stateless path.

The official runner’s frozen requirement sets do not score MCP Apps as required core behavior. Extensions are optional, so a perfect core score does not prove that a View can load. That is why the next layer belongs in your own repository.

Define the MCP App Contract You Claim

The stable MCP Apps specification defines a separate View-to-host protocol version. Do not confuse that with the core MCP revision or the SDK package version.

An app server should expose these linked pieces:

Surface	Required assertion
Tool definition	Stable name, useful description, valid inputSchema, honest annotations
Tool UI metadata	Modern _meta.ui.resourceUri points at the intended View
Tool visibility	Model tools include "model"; app-only helpers use ["app"]
Resource definition	URI starts with ui:// and MIME type is text/html;profile=mcp-app
Resource contents	Complete HTML is returned through text or blob
Resource security	CSP origins, permissions, domain, and border preference match actual use
Tool result	Readable content, model-safe structuredContent, component-only _meta
Output contract	structuredContent matches outputSchema when one is declared
View lifecycle	ui/initialize, ui/notifications/initialized, data delivery, and teardown happen in order
Fallback	A client without MCP Apps support still gets a useful core MCP result

New server code should use nested _meta.ui.resourceUri. The flat _meta["ui/resourceUri"] key remains a compatibility format for older hosts, but the current SDK marks it as deprecated. If your server emits both, test that both values resolve to the same resource.

Write a Fast Discovery and Resource Test

Start with data that a host reads before the View mounts. This test uses the current sunpeak mcp fixture:

import { expect, test } from 'sunpeak/test';

type UiToolMeta = {
  resourceUri?: string;
  visibility?: Array<'model' | 'app'>;
};

function getUiMeta(meta: Record<string, unknown> | undefined): UiToolMeta | undefined {
  const ui = meta?.ui;
  return typeof ui === 'object' && ui !== null ? (ui as UiToolMeta) : undefined;
}

test('every UI tool points at readable MCP App HTML', async ({ mcp }) => {
  const tools = await mcp.listTools();
  const resources = await mcp.listResources();
  const uiTools = tools.filter((tool) => getUiMeta(tool._meta)?.resourceUri);

  expect(uiTools.length).toBeGreaterThan(0);

  for (const tool of uiTools) {
    const ui = getUiMeta(tool._meta);
    const uri = ui?.resourceUri;

    expect(uri).toMatch(/^ui:\/\//);
    expect(tool.description?.length ?? 0).toBeGreaterThan(20);
    expect(tool.inputSchema).toMatchObject({ type: 'object' });

    const resource = resources.find((candidate) => candidate.uri === uri);
    expect(resource, `${tool.name} resource`).toBeDefined();
    expect(resource?.mimeType).toBe('text/html;profile=mcp-app');

    const html = await mcp.readResource(uri!);
    expect(html).toMatch(/<!doctype html|<html/i);
    expect(html).toMatch(/<body[\s>]/i);
  }
});

In sunpeak 0.20.x, mcp.readResource(uri) returns the HTML string. Use mcp.listResources() for the listed MIME type and resource metadata. If you need to inspect per-response content-item _meta, use the official MCP SDK client directly or unit-test the resource handler because content-item metadata can override listing metadata.

Validate Result Channels With Your Schema

Call every UI-launching tool with small deterministic fixtures. Validate your own output schema rather than checking that structuredContent merely exists:

import { z } from 'zod';
import { expect, test } from 'sunpeak/test';

const ReportOutput = z.object({
  title: z.string(),
  rows: z.array(
    z.object({
      label: z.string(),
      value: z.number(),
    })
  ),
});

test('show-report returns a portable result', async ({ mcp }) => {
  const result = await mcp.callTool('show-report', {
    reportId: 'demo-report',
  });

  expect(result.isError).toBeFalsy();
  expect(result.content?.some((block) => block.type === 'text' && block.text)).toBe(true);

  const output = ReportOutput.parse(result.structuredContent);
  expect(output.rows.length).toBeGreaterThan(0);

  const modelVisible = JSON.stringify({
    content: result.content,
    structuredContent: result.structuredContent,
  });
  expect(modelVisible).not.toContain('sk_live_');
});

content and structuredContent are both model-visible. Put only host or component-only data in result _meta, and never treat _meta as a secret store. The MCP App data flow guide covers that boundary in more detail.

Add separate cases for valid input, empty results, invalid input, upstream errors, and writes that need confirmation. Protocol errors and tool execution errors use different channels, so test the status and result shape your host will receive.

Test Security Metadata as an Allowlist

MCP App hosts build iframe policy from resource _meta.ui. The stable specification uses:

• connectDomains for fetch, XHR, EventSource, and WebSocket origins.
• resourceDomains for scripts, styles, images, fonts, audio, and video.
• frameDomains for nested iframe origins.
• baseUriDomains for allowed document base URIs.
• permissions for camera, microphone, geolocation, and clipboard write requests.

Assert exact origins instead of checking only that an array exists:

expect(resourceUi.csp).toEqual({
  connectDomains: ['https://api.example.com'],
  resourceDomains: ['https://cdn.example.com'],
});
expect(resourceUi.permissions ?? {}).toEqual({});

This makes a new network origin or browser permission visible in code review. Also test the browser boundary: an allowed request should succeed, an undeclared origin should fail, and the View should handle denied permissions without hanging.

Test the View Lifecycle in a Host Runtime

The View sends ui/initialize, receives host capabilities and context, then sends ui/notifications/initialized. The host must not send normal View requests or notifications before that handshake completes.

The lifecycle can then include:

• Partial and complete tool input.
• A successful tool result or cancellation.
• Host context changes for theme, locale, display mode, dimensions, and safe areas.
• View requests such as server tool calls, links, messages, or model-context updates.
• Host-requested teardown.

You do not need a long workflow to smoke-test this boundary. Mount each View, assert one stable root, and fail on console or page errors:

import { expect, test } from 'sunpeak/test';

test('report View initializes in each configured host', async ({ inspector }) => {
  const errors: string[] = [];
  inspector.page.on('pageerror', (error) => errors.push(error.message));
  inspector.page.on('console', (message) => {
    if (message.type() === 'error') errors.push(message.text());
  });

  const result = await inspector.renderTool(
    'show-report',
    { reportId: 'demo-report' },
    { theme: 'dark', displayMode: 'inline' }
  );

  await expect(result.app().getByTestId('report-root')).toBeVisible();
  await expect(result.app().getByRole('heading', { name: /report/i })).toBeVisible();
  expect(errors).toEqual([]);
});

Keep interaction-heavy paths in E2E tests, pixel checks in visual tests, and model tool selection in evals. The conformance smoke should tell you that the View can connect and receive its first useful state.

Verify Capability Negotiation and Fallbacks

MCP Apps use the extension identifier io.modelcontextprotocol/ui. In 2025-era core MCP, the client advertises extension support during initialize. In the 2026-07-28 core revision, client identity and capabilities travel in each request’s _meta, and servers can expose capabilities through server/discover.

Test both outcomes your server claims:

1. A client advertises MCP Apps and receives UI-linked tools plus readable ui:// resources.
2. A client omits MCP Apps and still receives a useful text tool or a clear unsupported result.
3. A View calls a tool whose visibility includes "app".
4. The model catalog excludes tools with visibility: ["app"].
5. A missing optional host capability triggers the documented fallback.

Capability negotiation does not replace runtime detection. A host can support the extension while omitting a specific action, display mode, permission, or modality. Check the capability close to the call and test the unsupported branch.

Keep Core Revisions Separate

Use an explicit matrix because the current ecosystem has two active core eras plus a separate Apps version:

Contract	Current version to test	Main lifecycle
Core MCP used by many deployed hosts	2025-11-25	Stateful initialize handshake, optional transport session
Modern core MCP	2026-07-28	Stateless requests with per-request identity and capabilities
Stable MCP Apps extension	2026-01-26	View-to-host ui/initialize over postMessage
sunpeak 0.20.79 server fixture	2025-era core wire	MCP App contract and current ChatGPT/Claude host-replica tests

Passing a 2025 core scenario does not prove its 2026 form because the request envelope and lifecycle differ. Passing the Apps handshake does not prove either HTTP transport. Record the core revision, Apps version, SDK version, host mode, and runner version with every conformance report.

Add Conformance to CI in Stages

Run the cheapest deterministic checks first:

1. Start the built MCP server and wait for its readiness endpoint.
2. Run the pinned official core conformance suite for each supported core revision.
3. Run discovery, resource, metadata, and result contract tests.
4. Render one smoke state per View in each supported host replica.
5. Run the full E2E and visual matrix after those gates pass.
6. Run live-host checks on release or on a schedule.

A repository script can keep the pull-request gate readable:

{
  "scripts": {
    "test:conformance:core": "npx -y @modelcontextprotocol/conformance@0.1.16 server --url http://127.0.0.1:8000/mcp --suite active --spec-version 2025-11-25",
    "test:conformance:app": "playwright test tests/e2e/conformance.spec.ts",
    "test:conformance": "pnpm test:conformance:core && pnpm test:conformance:app"
  }
}

If you use an expected-failures file, assign every entry and set a removal condition. The official runner treats an unexpected pass as a failure, so fixed items cannot remain hidden. Baseline one check when possible instead of suppressing an entire scenario.

Diagnose Failures by Boundary

Symptom	Start here
Protocol version or JSON-RPC schema error	Official core conformance output and wire trace
Tool appears but no View opens	_meta.ui.resourceUri, visibility, and capability negotiation
Host cannot read the View	Resource URI, MIME type, HTML body, and server authorization
View is blank	ui/initialize, CSP, bundled assets, and browser console
View works in one host only	Host capabilities, iframe policy, CSS variables, dimensions, and safe areas
Model calls an app-only helper	Visibility metadata and the host’s model tool catalog
Result renders but model context is wrong	content, structuredContent, _meta, and outputSchema
Modern client fails while current hosts pass	Separate the 2025 and 2026-07-28 core wire tests

This ordering saves time because each failure has one owner. The official runner owns core protocol evidence. Your contract tests own server and View metadata. Host tests own the runtime behavior users actually see.

Where sunpeak Fits

You can implement every layer with the official SDKs, the official conformance runner, the MCP Apps reference host, and Playwright. sunpeak combines the app-specific layers:

• The mcp fixture lists tools and resources, reads HTML, and calls tools through MCP.
• The Inspector renders the View in replicated ChatGPT and Claude runtimes.
• Simulation files pin tool input, tool results, app context, and app-initiated server tool responses.
• Playwright projects repeat the same smoke test by host without paid accounts or host credits.
• Live tests cover the smaller set of behaviors that only the production host can prove.

For an existing TypeScript, Python, Go, or Rust server, scaffold the test harness and keep the official core runner beside it:

npx sunpeak test init --server http://127.0.0.1:8000/mcp
npx sunpeak test

Conformance is a release gate, not a single score. Keep the core revision, MCP App contract, and host runtime visible as separate results so one green check cannot hide a failure in another layer.