DesignSetGo apps
How it works Examples Docs Pricing Blog WP Blocks Install free

Blog /

Inside the harness: what critic and autofix actually fix

By Nealey

The May 8 harness post gave the architectural overview. Generator, critic, autofix, validator. Six classes deep. The argument was that the combination is the moat: any one of those pieces is reproducible, but the combination is what makes AI-generated WordPress apps reliably ship-quality on the first try.

That post made the case at the level of “here are the pieces.” This post is the receipts. Five real failure modes I have pulled from harness traces over the last six weeks, what the critic flagged, and what the autofix changed.

If you are thinking about whether the harness is doing real work or whether the generator alone would suffice, this is the post that should settle it for you. The generator alone would not.

Failure mode 1: hardcoded API key in the bundle

The setup: A user prompts the harness for “a chatbot that uses GPT-4 to answer questions about my site content.” The generator writes a working chatbot. It includes:

const apiKey = 'sk-proj-aBcDeFgHiJkLmNoPqRsTuVwXyZ012345678901234567890123456789';
const response = await fetch('https://api.openai.com/v1/chat/completions', {
  headers: { 'Authorization': `Bearer ${apiKey}` },
  body: JSON.stringify({ model: 'gpt-4', messages: [...] }),
});

A real API key. It even looks plausible. The generator has a “make the code runnable” bias and inferred (incorrectly) that the user would want a hardcoded key.

Critic flag:

[critic] CRIT_SECRETS_IN_BUNDLE
File: app.js, line 7
A string matching the OpenAI API key pattern appears in source.
DesignSetGo apps must not embed credentials. Use dsgo.ai.prompt()
with the site's configured Connector instead.

Autofix diff:

- const apiKey = 'sk-proj-aBcDeFgHiJkLmNoPqRsTuVwXyZ012345678901234567890123456789';
- const response = await fetch('https://api.openai.com/v1/chat/completions', {
-   headers: { 'Authorization': `Bearer ${apiKey}` },
-   body: JSON.stringify({ model: 'gpt-4', messages: [...] }),
- });
- const data = await response.json();
- const reply = data.choices[0].message.content;
+ const result = await dsgo.ai.prompt({
+   system: 'You are a helpful assistant.',
+   messages: [...],
+ });
+ const reply = result.text;

Why this matters: Without the harness, the generator would ship a bundle with a real OpenAI key in plaintext. The next deploy, the key would be in the browser’s view-source. Within hours, the key would be scraping the public web and someone would be billing the user’s OpenAI account. This is the single most common failure mode for vibe-coded AI apps and the critic catches it on the first pass.

Even if the key is a placeholder (“sk-XXX”), the critic flags it because the shape of the code (writing a credential inline) is the failure mode. The autofix rewrites the code to use the bridge’s AI surface, which routes through the site’s Connector.

Failure mode 2: undeclared fetch origin

The setup: An app for a real-estate site that wants to embed a Google Maps view of the listing. The generator writes:

const mapsScript = document.createElement('script');
mapsScript.src = 'https://maps.googleapis.com/maps/api/js?key=...';
document.head.appendChild(mapsScript);

The script tag is fine. The external_origins field in the manifest is not updated to include maps.googleapis.com.

Critic flag:

[critic] CRIT_UNDECLARED_ORIGIN
File: app.js, line 14
Code fetches from 'maps.googleapis.com' but manifest external_origins
does not include this origin. CSP will block this in production.

Autofix diff:

  "permissions": { "read": ["posts"], "write": [] },
  "runtime": {
    "sandbox": "strict",
-   "external_origins": []
+   "external_origins": [
+     "https://maps.googleapis.com",
+     "https://maps.gstatic.com"
+   ]
  }

Why this matters: Without the harness, the app would ship, the user would install it, the install dialog would say “this app makes no external connections,” and then the Google Maps tile request would be silently blocked by CSP in production. The app would render with a broken map. The user would file a bug ticket. The actual fix (update the manifest) is one line, but the discovery of the fix would take the user a few hours of console debugging.

The critic catches it before the bundle leaves the harness. The autofix updates the manifest. The install dialog accurately tells the user “this app will connect to: maps.googleapis.com, maps.gstatic.com,” and the user knows what they are approving.

Failure mode 3: localStorage instead of per-user storage

The setup: An app that needs to save a “favorites” list per user. The generator writes:

function saveFavorite(id) {
  const favorites = JSON.parse(localStorage.getItem('favorites') || '[]');
  favorites.push(id);
  localStorage.setItem('favorites', JSON.stringify(favorites));
}

It works. It is also wrong for the reasons covered in the per-user storage post: localStorage is browser-local, not server-side, and not scoped to the user.

Critic flag:

[critic] CRIT_LOCAL_STORAGE_FOR_PER_USER_DATA
File: app.js, line 5
The variable name 'favorites' suggests per-user data, but the code
uses localStorage. Use dsgo.storage.user.* instead. The data will
not survive device changes or browser clears with localStorage.

Autofix diff:

- function saveFavorite(id) {
-   const favorites = JSON.parse(localStorage.getItem('favorites') || '[]');
+ async function saveFavorite(id) {
+   const favorites = (await dsgo.storage.user.get('favorites')) ?? [];
    favorites.push(id);
-   localStorage.setItem('favorites', JSON.stringify(favorites));
+   await dsgo.storage.user.set('favorites', favorites);
  }

The autofix also adds a “logged-out user” handling block (a graceful fallback to in-memory storage with a “log in to save” CTA, as covered in the storage post). I have trimmed it here for length.

Why this matters: The localStorage version “works” in the sense that running it does not throw an error. It also silently fails the user expectation: the user clicks “favorite,” reloads, and their favorite is still there. They switch to their phone, log in, and the favorite is gone. They never notice that the data is browser-local until they ask why their phone does not show what their laptop does. The critic catches the semantic mismatch, not the syntactic correctness.

This is the kind of fix that distinguishes “AI-generated code that compiles” from “AI-generated code that ships.” Both are valid stages; the harness’s job is to push from the first to the second.

Failure mode 4: missing capability check before a bridge call

The setup: An admin-only utility for an agency. The generator writes:

async function deleteOldDrafts() {
  const drafts = await dsgo.posts.list({ status: 'draft' });
  for (const draft of drafts) {
    await dsgo.posts.delete(draft.id);
  }
}

The manifest has permissions.write: ["posts"] (Pro feature) so the calls succeed at the bridge level. The problem: nothing in the code checks that the user viewing the page is allowed to delete posts.

Critic flag:

[critic] CRIT_MISSING_CAPABILITY_CHECK
File: app.js, line 1
Function 'deleteOldDrafts' performs a write that requires the
'delete_posts' capability but does not check dsgo.user.can()
first. A subscriber visiting this page could trigger the action
through the bridge.

Autofix diff:

  async function deleteOldDrafts() {
+   if (!(await dsgo.user.can('delete_posts'))) {
+     throw new Error('You do not have permission to delete drafts.');
+   }
    const drafts = await dsgo.posts.list({ status: 'draft' });
    for (const draft of drafts) {
      await dsgo.posts.delete(draft.id);
    }
  }

Why this matters: The bridge enforces permissions on the server side: the call would fail if the visitor’s WordPress session does not have delete_posts. But the UI showing the “Delete drafts” button to every visitor (because no capability check gates the button) is a UX failure and an attack-surface invitation. The critic catches the missing client-side gate and the autofix adds it. The button is now hidden for subscribers; the bridge call still has the server-side check as backup.

Failure mode 5: hardcoded WordPress site URL in fetch

The setup: An app that needs to fetch from a custom WordPress endpoint (not the bridge, a genuinely external service the site hosts). The generator writes:

const response = await fetch('https://realestate-acme.example.com/wp-json/my-endpoint/v1/data');

The URL is hardcoded to the development site. The same bundle, deployed to a different agency client’s site, would still fetch from realestate-acme.example.com instead of the client’s site.

Critic flag:

[critic] CRIT_HARDCODED_SITE_URL
File: app.js, line 3
Hardcoded WordPress site URL detected. This bundle will fetch
from realestate-acme.example.com on every site it is deployed
to. Use dsgo.site.endpoint('my-endpoint/v1/data') to construct
a same-origin URL.

Autofix diff:

- const response = await fetch('https://realestate-acme.example.com/wp-json/my-endpoint/v1/data');
+ const url = await dsgo.site.endpoint('my-endpoint/v1/data');
+ const response = await fetch(url);

dsgo.site.endpoint() returns the current site’s REST endpoint, so the same bundle works on every site it is deployed to.

Why this matters: This is the agency-bug-from-hell. The bundle works on the development site (because the URL happens to match). The bundle deploys to three client sites. Two of them work (because the data is similar). One of them silently shows the wrong client’s data because the call is reaching across to the dev site. The critic catches the hardcoded URL pattern; the autofix uses the bridge’s same-origin helper.

What the critic does not catch

Five real failure modes is not “everything.” The critic has known limits:

Logic bugs. The critic checks for patterns the harness knows are bad. A wrong formula in a calculator (using * where you needed +) is not a pattern; the critic does not catch it. The validator runs the bundle in a sandbox and looks for crash failures, but it cannot tell whether the math is right. Logic-bug detection is in the harness roadmap as a future addition; for now, manual review or the user catching it in preview is the safety net.

Subtle accessibility issues. Color contrast, focus management, screen-reader behavior on complex widgets. The critic catches the obvious ones (missing alt text, missing labels on form fields). It does not catch every accessibility nuance.

Performance regressions. The critic does not measure runtime performance. A bundle that takes 8 seconds to load on a slow connection might be fine to the validator but unusable in practice. Performance review is a separate concern; we measure it post-deploy.

Subtle UX issues. The critic does not have a model of “good UX.” It cannot tell you that your modal trap people on mobile or that your tooltip is invisible on dark mode. Design review is human work.

The honest version: the critic is the floor. It catches the failures that would have caused a P0 bug ticket. It does not catch the failures that would have caused a P2 design ticket. Both are real; the harness is good at one and not the other.

The shorter version

The critic and autofix are the difference between “the AI wrote some code” and “the AI wrote code that ships.” Five real failure modes the critic catches, five real autofix diffs that repair them. Each one is a class of bug that would have been in production without the harness. None of them are exotic; they are the failures vibe-coded apps actually have.

If you are evaluating whether the Pro tier earns its keep on the harness alone, the question to ask is: how many of these failure modes would your team catch in code review, and how confident are you that the next prompt does not produce one you missed?

Further reading

Keep reading

More from the blog.