---
title: Check Evaluation
published: true
index: 10
tier: free
---

# Check Selection and Evaluation

Use this workflow whenever versioned audit checks are in scope.

## 1. Resolve scope

- Use focused scope only when the user explicitly names checks or categories and the host has not supplied a persisted catalog.
- Select every active Check for an unqualified audit.
- When the host supplies a persisted catalog, treat it as authoritative and evaluate every catalog ID even when surrounding context sounds narrower.
- Do not narrow the selected comprehensive audit by page, viewport, store archetype, available tool, or preferred audit lens. Those facts control execution and applicability, not catalog membership.

Before selection, record the storefront URL, available repo/revision, verified store/property, missing optional sources, theme/market, safe browser states, non-secret capabilities, comparable evidence and read-only boundary. Retain no secrets or customer/order personal data.

## 2. Read leading frontmatter

The Store Skills MCP connection serves the Check catalog to signed-in customers. At the start of an audit, generate one UUID and save it as `requestId` in the audit’s local run record before asking for checks. Use `skill_run` with `run: "comprehensive-shopify-audit"` and that same `requestId` for every lookup, page, batch read, retry and resume. The first successful protected result uses one audit. A separately requested new audit gets a new UUID.

If Glue reports an exhausted allowance or an expired run, stop and relay the returned reason and reset details. Never start another run automatically. If the tool is missing or the active catalog cannot be read, stop before evaluation. Ask the customer to sign in with `npx glue-fast login --site https://glue.fast`, install or update with `npx glue-fast install-plugin brickspacelab/storeskills`, and reopen the agent. Use the published catalog for the whole audit.

A catalog Check must begin with frontmatter containing:

- a unique stable `id`;
- a positive integer `version`;
- `status: active` or `status: deprecated`;
- a canonical `category`; and
- a first-level heading after frontmatter for the title.

Ignore deprecated checks. Treat missing or invalid required metadata and duplicate IDs as catalog errors. Read optional `subcategory`, `secondary_lenses`, `surfaces`, `viewports`, `store_archetypes`, `applies_to`, `execution_gate`, `execution_mode`, `checkout_phase`, `required_capabilities`, `credentialed_surfaces`, `required_context`, and `supporting_context` fields to plan safe execution and evidence collection.

### Select and read

- **Select with filters.** Call `skill_run` with `status: active` plus any explicitly requested `category`, `subcategory` or `surfaces`. Pass each `nextCursor` back as `cursor` until none is returned, and confirm the collected count equals `total`. Each result gives the Check's `page` address, repository `path` and leading metadata in `fields`; apply the catalog rules above to that metadata.
- **Use `query` only to find named Checks.** When the user names Checks by stable ID, title or behavior rather than by category or surface, call `skill_run` with `status: active` and `query` set to that ID or phrase, then keep only results whose `fields.id` or title is the one named. A query also matches body text, so a result can merely mention the term. Never use `query` for a full or category run, to narrow an authoritative host catalog, or to decide applicability; filters select the catalog.
- **Read bodies in batches.** Read each evaluation batch with `skill_run` and `pages`, up to 20 different `page` addresses per call, in selected-ID order. Each entry answers one address:
  - A page with Markdown is the Check body to evaluate.
  - `deferred: true` means the call reached its size limit; request those addresses again in a later call. Deferral is not a blocker.
  - `unavailable: true`, or `restricted: true` with empty Markdown, means the Check is not included in the customer's access. Record it as Blocked with a shared Limitation that names it as outside the customer's Glue access.

Maintainers testing an unreleased catalog in the Store Skills source repository may instead discover Markdown files beneath its root `comprehensive-checks/` folder, excluding `README.md`, `AGENTS.md`, `authoring.md` and `check-template.md`, with searches such as `rg -l '^status: active$' comprehensive-checks -g '*.md'`. Such a run uses only those files and records their revision.

Extract metadata before reading check bodies. Sort the selected catalog lexicographically by stable ID. Preserve ID, title, category, version, file path, selectors, execution constraints, required context, and supporting context.

Execution gates never widen permissions. When an authoritative catalog includes an account-only check and the declared account report-secret capabilities are unavailable, continue public and repository evidence first. Use Blocked only when the required signed-in page state remains necessary to decide the check.

## 3. Create bounded batches

Batch the ID-sorted catalog within host context/output limits, retaining one immutable selected-ID ledger. Batching changes no scope, categories, evidence requirements or completion rules.

Persist selected IDs/versions and source identity before collection: host-pinned revision, or the `skill_run` paths, page addresses and versions (the source revision for a maintainer run). Resume that identity and reconcile source changes before continuing; never mix versions silently.

Read every check in the current batch completely before evaluation, from the run's one catalog source. Do not summarize away applicability, exclusions, evidence requirements, evaluation states, or fixtures.

Plan each batch's assertions against exact routes, products/variants, controls, states and evidence. Reuse an applicable latest same-target browse plan, collecting fresh evidence and deciding independently. Otherwise follow visible primary-category order; add different existing products/offers only when a selected Check needs them. Explain added/replaced/absent states. Reuse verified viewports, or use supported 1280×800 desktop and 320×800 mobile; record actual dimensions/substitutions, market/currency, session/cookie limits and known revision.

For each assertion, record expected behavior/basis, exact input/action, actual state/value/evidence and comparison or verified exclusion. Make broad wording concrete before execution; “clear and usable” alone is no test. Invent no feature requirement, design preference or cutoff. Name undecidable catalog ambiguity instead of inventing Pass or a defect.

Match evidence to the assertion: before/after semantics and feedback; commerce identities, quantities and arithmetic; raw comparable measurement samples; rendered output and relevant source dependency for implementation claims. Images cannot prove computed semantics, valid source cannot prove deployment and metrics cannot prove causes. Ownership remains optional unless required by the Check. Generic procedures cannot expand permissions or integrations.

Test distinct assertions independently; another control, price, page type or state cannot stand in. Record verified absence or exact missing evidence. Use permitted representative samples without whole-site claims. Plans/assertion notes are external evidence, not new Checks/schema fields.

Record each decision during evaluation and persist its batch before loading the next. Scripts may validate/assemble existing records, never manufacture checkpoints afterward. Record:

- check ID and version;
- applicability as applicable, not applicable, or unresolved, with target evidence;
- every required and supporting context with availability and source;
- exact route, state, viewport, or repository location used;
- expected behavior, actual assertion observations, terminal outcome and a check-specific reason;
- canonical Issue reference for Fail or shared Limitation reference for Blocked.

Host-side files or standard commands may store, sort, merge, and validate these records. They must not author model-judgment outcomes or fill an unevaluated remainder.

## 4. Build context ledgers

For each selected check, record every named required and supporting context source. Mark each source as available, partial, blocked, or not applicable, with the target-bound reason.

Use safe required sources and material supporting sources; explain unused supporting evidence. Missing required evidence blocks only an otherwise undecidable Check; a reproduced defect still establishes Fail. Supporting gaps affect confidence or remain visible unless the Check requires otherwise.

Resolve context by the decision it must support:

- `browser` may use a verified desktop or mobile viewport plus DOM, accessibility tree, interaction, console, network, or screenshot evidence as the check requires.
- `repository` and `github` may establish ownership, configured features, routes, responsive behavior, variants, history, checks, and absence of optional implementations.
- `shopify-admin` may establish safely scoped aggregate orders or abandonments plus catalog, market, discount, availability, and configuration truth for the verified store.
- `ga4` may establish aggregate reach, comparable funnel shape, segments, and measurement quality for the verified property; it never proves a UX cause by itself.
- `catalog-taxonomy` may be supplied by rendered collections and products, public storefront data, repository configuration, or safely scoped Shopify catalog evidence.
- If an active check still requires an excluded source and the permitted evidence cannot independently decide it, treat that as a catalog defect to deprecate rather than a target blocker.
- `prior-audit` supports Changes only when target, state, method, environment, and relevant revision are comparable.
- `partial` required context blocks only when the missing portion is necessary for the decision. Representative evidence is sufficient when the check permits it and no material contradiction remains.

## 5. Collect and challenge evidence

Use host-provided browser, workbench, and connected-data capabilities plus the target repository's existing non-mutating commands. Evidence may include rendered states, DOM and accessibility data, console and network output, repository source and history, safely scoped Shopify and GA4 aggregates, existing tests and validation, current platform documentation, prior compatible reports, and comparable public competitor evidence.

Reproduce automated leads and test exclusions or contradictory states. Generic best practices and third-party examples are expectations to verify, not proof of a target defect.

- Use representative live routes and repository ownership for ordinary storefront checks unless a check requires broader coverage.
- Use Not applicable when an optional feature, component, business model, catalog condition, or customer job is verified absent.
- Use Fail when a required implemented path is directly observed to be broken. Block only distinct downstream checks whose required state cannot be reached.
- Treat failed screenshot capture as a presentation-evidence gap when other evidence completes the check.
- Treat verified browser emulation as mobile or desktop evidence unless the check explicitly requires a named device, engine, keyboard, or assistive technology.

## 6. Assign one terminal outcome

Every selected check receives exactly one:

- **Pass**: the complete required procedure found no actionable problem.
- **Fail**: direct evidence established an actionable problem and one canonical Issue.
- **Blocked**: missing required evidence or an unresolved contradiction prevents a responsible conclusion.
- **Not applicable**: the relevant feature, condition, state, or customer job was verified absent.

Apply outcomes in this order:

1. Decide applicability from the check's prerequisites and target-bound browser, repository, and catalog evidence.
2. If verified absent, record Not applicable with the absence evidence.
3. If applicable, complete every required step that can affect the result and challenge contradictions.
4. Record Pass or Fail when the exercised evidence decides the check.
5. Record Blocked only when an exact required source or state remains unavailable or contradictory after safe attempts, and name the smallest unblock action. “Not tested”, “matrix incomplete” and time spent are not capability failures: continue safe work when it can complete the procedure. Never silently omit a selected blocked Check.

Never infer outcomes from check ID prefixes, category, surface, viewport, context names, regular expressions, batch position, or a catch-all Limitation. Do not use cart cleanup, comparison incompatibility, or report-artifact failure as check evidence.

## 7. Reconcile batches

After all batches, compare the immutable selected-ID ledger with persisted outcomes. Reject missing, duplicate or unknown IDs and any outcome not individually recorded during its batch. Never fill a remainder from metadata, generic rationales or a catch-all blocker.

Apply the [reporting contract's final reconciliation](../core/reporting-contract.md#final-reconciliation) for Issue/Check category ownership, reciprocal references, deduplication and comparable Changes. Host-requested candidate grouping must preserve checkpointed decisions and explicit evidence-backed amendments; grouping itself cannot revise an outcome.

Before Pass, reconcile every required assertion and contradiction from the run. Later success cannot erase an earlier defect. Append evidence-backed corrections to checkpointed decisions with reason/superseded record; preserve the original.

Retest a prior same-target failure's exact element/claim/state when safe and applicable. Explain changed outcomes with current defects/corrections, changed applicability/version/route/offer or incomparable environment. Prior outcomes are leads, not automatic failures. Block unavailable required states unless another defect proves Fail; document verified inapplicability. Another control/price cannot pass an untested prior assertion. Missing history does not block fresh audits; Resolved Changes still need compatible comparison.

Do not call the audit complete while any selected Check lacks a terminal outcome. Completion covers the selected Comprehensive catalog, not storefront certification or a separate performance/funnel report.
