How good is AI at marketing? This benchmark grades real marketing work — writing ad concepts, analyzing campaign data, repairing broken landing pages, reconciling attribution models that disagree — against expert checklists, and runs every model three ways: alone, with the marketing skills pack added, and inside the full Magister agent. Task by task, it shows what today's models are good at and where they struggle.
The model on its own: one API call carrying the task and its source material, nothing added.
The same single API call, with the task's marketingskills SKILL.md files pasted verbatim into the prompt.
The production Magister agent works the task over a full multi-step turn on a real project machine. It brings the marketingskills pack installed on that machine, plus hosted tools and web research — so this configuration is always the model with skills and the harness together, not the model alone.
One model, three configurations, same tasks and same grading. The Magister bar is the full agent — that model plus the marketingskills pack and hosted tools, not the model on its own. A dot on the axis means that configuration hasn't been run for that model yet. Score averages the runner's adjusted rubric scores across all tasks (0–100), including partial credit and penalties for critical failures. Tasks passed is separate: one missed requirement fails the whole task.
| Model | Score?Adjusted rubric score, averaged across every task and attempt, on a 0–100 scale. It includes fractional credit, excludes checks already satisfied by a blank answer, and applies critical-failure and incompleteness penalties. Truncated answers remain in the task denominator. See Methodology for the scoring rules. | Tasks passed?Tasks passed, of tasks attempted (pass@1). A task passes only when the answer meets every requirement in its rubric — one miss fails the whole task, with no partial credit. | Answer cost?What this model's answers cost for the whole suite: its input and output tokens priced at the provider's list rates. Judge and grading spend are excluded. | Avg task time?Average wall-clock time of the model's API call per task, as measured by the runner. | Output tokens?Total tokens the model generated across all tasks, including any billed reasoning tokens. |
|---|---|---|---|---|---|
| Gemini 3.8 Flash high effort | 78.4 | 8/20 | $1.37 | 102.5s | 353.8k |
Cost and time are the candidate model's alone — judge overhead is excluded. Quality and efficiency are never blended into one number.
Scored 78.4 of 100 and passed 8 of 20 tasks — at $1.37 for the full suite.
| development | holdout | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | ad-creatives | ad-performance-analysis | ads-ledger-analysis | analytics-source-conflict | attribution-sensitivity | blog-post | budget-reconciliation-trap | email-flow-repair | factorial-ad-test | headline | landing-page-repair | landing-page | marketing-plan | shopify-meta-capi | source-conflict-repair | tracking-audit | launch-capacity-trap | platform-limits-gauntlet | rate-card-reconciliation | unsupportable-claims-brief |
| Gemini 3.8 Flash 78.4 8/20 · high effort | Fail8/9 | Fail8/10 | Fail11/12 | Pass9/9 | Pass10/10 | Pass9/9 | Fail7/8 | Fail13/14 | Pass13/13 | Pass6/6 | Fail0/0 | Fail9/10 | Fail6/8 | Pass11/11 | Pass18/18 | Fail0/0 | Fail0/0 | Fail0/0 | Pass10/10 | Fail9/10 |
The count under each verdict is requirements passed. One missed requirement fails the task, so a near-miss and a blowout look the same in the score — the cell shows which one it was.
| Requirement | Verdict | Evidence |
|---|---|---|
| three_concepts Exactly three ad concepts are present. | ✓ | Programmatic check: primary_text_field_count=3. |
| all_fields Each concept has all seven labeled fields: Primary text, Headline, Description, CTA, Visual direction, Audience insight, and Test hypothesis. | ✓ | Each of the three concepts lists Audience insight, Test hypothesis, Primary text, Headline, Description, CTA, and Visual direction. |
| approved_cta Exactly three CTA fields are present and every one is exactly 'Explore the Arc Clip Lamp'. | ✓ | Programmatic check: cta_field_count=3; approved_cta_field_count=3. |
| distinct_motivations The three concepts are built on distinct audience motivations (per the audience insights), not rewordings of the same angle. | ✓ | Concept 1 targets desk-space reclamation, Concept 2 targets cross-room portability, and Concept 3 targets shadow-free precision work for makers, three distinct motivations. |
| no_invented_proof No invented reviews, customers, performance numbers, scarcity, awards, or product capabilities beyond the supplied facts; forbidden claims (award-winning, best-selling, clinically proven, limited time, carbon neutral, guaranteed productivity) are absent. | ✓ | All copy sticks to supplied facts ($68, 8 hours, 2.25 inches, 3 brightness/2 color settings, padded clamp) with no reviews, awards, scarcity, or performance guarantees. |
| canonical_product_name Whenever the product is named, it is called 'Arc Clip Lamp'; shortened or altered names such as 'Arc Lamp' fail. | ✓ | The product is referred to as 'Arc Clip Lamp' consistently across all three concepts. |
| concrete_visuals Each Visual direction describes a specific, filmable/photographable scene tied to the concept's motivation, not vague mood words. | ✓ | Visual directions describe specific filmable scenes, e.g. 'top-down flat-lay video over a maker's cutting mat... hands assembling leather goods or linocut printing' with the lamp clamped and angled. |
| hypothesis_test_design Each Test hypothesis identifies the creative or message variable being tested and the comparator or control it would be tested against. | ✓ | Each hypothesis names a variable and a comparator, e.g. 'desk-surface reclamation... will outperform mobility-led messaging' and 'cross-room portability... than focusing strictly on task utility'. |
| hypothesis_prediction Each Test hypothesis names an observable KPI and its expected direction. Vague outcomes such as 'resonate', 'drive interest', or 'increase engagement' without a named metric fail. | ✗ | Concept 1's hypothesis only says it will 'outperform' with no named metric, and Concept 3 uses the vague 'drive higher engagement' which the rubric explicitly flags as insufficient without a named KPI. |
| Requirement | Verdict | Evidence |
|---|---|---|
| totals_correct Per-campaign four-week totals are computed and correct within rounding: Prospecting - Broad spend $4,800, 169 conversions, CPA ~$28.40; Prospecting - Small-Space Interest spend $2,400, 138 conversions, CPA ~$17.39; Retargeting - 30d Engagers spend $1,600, 83 conversions, CPA ~$19.28 (or stated over known weeks only); Launch Teaser totals acknowledge the missing week. | ✓ | The table correctly computes Broad ($4,800/169=$28.40), Small-Space ($2,400/138=$17.39), Retargeting ($1,600/83=$19.28), and flags the Launch Teaser's missing week. |
| best_worst Small-Space Interest is identified as the best performer (lowest CPA ~$17.39, stable, ROAS ~3.9) and Broad as the biggest problem (weekly CPA rising from $20 to $48 while CTR halves from 1.8% to 0.9%); calling the Launch Teaser 'worst' on CPA alone without noting its video-views objective does not pass. | ✓ | Small-Space Interest is called best with CPA $17.39/ROAS 3.91x, and the response explicitly notes 'Broad had the worst week-over-week degradation' while separately naming Launch Teaser worst financially, acknowledging its video-views objective mismatch. |
| recommendation There is one specific budget change (e.g. shift spend from Broad toward Small-Space Interest) and one bounded test (what change, what metric, what timeframe or budget). | ✓ | A specific budget reallocation plan (cut Launch Teaser, reduce Retargeting, defend Broad, scale Small-Space) and a bounded 14-day creative test are both given. |
| missing_preserved The empty conversions/revenue cells for Launch Teaser in the week of 2026-07-20 are flagged as missing data (e.g. a tracking gap), not treated as zero or silently summed over. | ✓ | The response states 'It is ambiguous whether this represents 0 conversions or untracked pixel downtime' rather than treating the blank cells as zero. |
| partial_metric_denominators Any Launch Teaser CPA or ROAS uses spend and outcomes from exactly the same complete rows, or is reported as unavailable. Dividing all four weeks of spend by only three known weeks of conversions/revenue fails. | ✗ | The headline Launch Teaser CPA/ROAS in the table is computed as $1,000 (all four weeks' spend) divided by 7 known conversions, mixing four weeks of spend with three weeks of outcomes exactly as the rubric disallows. |
| evidence_shown The arithmetic behind key claims is shown (at least CPA calculations), and metric definitions/sources are used as supplied rather than invented. | ✓ | CPA and ROAS formulas with explicit fractions (e.g. $4,800/169=$28.40) are shown for every campaign. |
| cautious_conclusions A limitation that would most change the conclusion is stated (e.g. attribution window, missing week, four-week sample), and correlation is not presented as proven causation. | ✓ | Section 5 identifies the missing attribution-window/view-through data as the limitation that would 'most change conclusions' without claiming causation. |
| trend_over_snapshot The diagnosis is driven by week-over-week movement (Broad's CTR/CPA deterioration, Retargeting frequency climbing 6.1 to 11.4 with falling conversions), not four-week averages alone. | ✓ | The diagnosis explicitly tracks week-over-week CPA/CTR deterioration for Broad and frequency climbing from 6.1 to 11.4 for Retargeting. |
| prioritized_action Recommendations are prioritized (one clear primary move) rather than an unranked laundry list. | ✗ | The budget change section lists four separate tactical moves (pause, reduce, defend, scale) with no explicit ranking of which is the single primary action. |
| bounded_test_decision The proposed test has one control and one changed variable, a duration or spend cap, one primary KPI, a numeric win rule, and either a numeric minimum sample or an explicit rule for declaring the result inconclusive. | ✓ | The test specifies a control asset, 14-day duration, $700 spend cap, primary KPIs (CTR/CPA), a numeric win rule (CTR≥1.5%, CPA≤$24 over 500 clicks), and a kill threshold (CPA>$38 after $150 spend). |
| Requirement | Verdict | Evidence |
|---|---|---|
| answer_block_complete The response ends with the required answer block and all five labeled answers A1-A5 are present and parseable. | ✓ | Programmatic check: answer_count=5. |
| q1_blended_cpa A1 states the blended CPA (total spend / net conversions) for the full 14 weeks; full credit within 0.5% of the value computed from the export, zero credit beyond 10% off. | ✓ | Programmatic check: answer_q1=25.29. |
| q2_wasted_spend A2 states total spend across zero-conversion adset-days; full credit within 1%, zero credit beyond 25% off. | ✓ | Programmatic check: answer_q2=10228.41. |
| q3_underwater_week A3 names the week (1-14) in which prospecting-broad's weekly net ROAS first fell below 1.0; full credit for the exact week, partial credit within one week, zero credit two or more weeks off. | ✓ | Programmatic check: answer_q3=10.0. |
| q4_refund_drag A4 states the total refunded revenue of the adset with the most refunded revenue; full credit within 1%, zero credit beyond 25% off. | ✓ | Programmatic check: answer_q4=8470.84. |
| q5_final_roas A5 states the account-wide net ROAS for the final 28 days; full credit within 1%, zero credit beyond 15% off. | ✓ | Programmatic check: answer_q5=3.09. |
| refund_adjusted_method The analysis states that conversions and revenue were computed net of refunds (or shows the refund adjustment explicitly) rather than silently using gross figures. | ✓ | The response explicitly defines and computes Net Revenue = gross revenue − refunded revenue and Net Conversions = conversions − refunded conversions throughout (e.g., 'Net Revenue = Revenue USD − Refunded Revenue USD'). |
| worst_refund_adset_named The review names rt-lookalike as the adset with the largest refunded revenue. | ✓ | The response states 'The single adset with the most refunded revenue is `rt-lookalike` with $8,470.84.' |
| no_invented_data Every figure presented as coming from the export is derivable from the supplied CSV; no external benchmarks, industry averages, or invented segment data are presented as if they were in the data. | ✗ | The response claims refund rates on rt-visitors and rt-cart are 'well within healthy apparel industry tolerances,' an external benchmark not present in or derivable from the supplied CSV, violating the no-invented-benchmarks instruction. |
| prospecting_decline_called The review identifies prospecting-broad's performance decline across the window as a trend requiring action, not merely a low average. | ✓ | The review presents a week-by-week net ROAS trajectory for prospecting-broad showing a steady decline from 2.62x in W1 to 0.00x in W14 and calls it a 'collapse' requiring action. |
| pb_video_flagged The review specifically calls out the pb-video adset as continuing to spend while producing zero conversions in the final weeks (or an equivalent adset-level wasted-spend identification naming pb-video). | ✓ | The response states 'pb-video failed first: It recorded 56 zero-conversion days out of 98 days' and recommends 'Pause pb-video immediately.' |
| recommendation_grounded Each recommendation is tied to specific figures from the analysis (named campaigns/adsets with their numbers), not generic best practice. | ✓ | Recommendations cite specific figures like reallocating '+$500/week into bs-exact' and slashing prospecting-broad budget from '~$260/day' to '$100/day', tied to named adsets/campaigns. |
| Requirement | Verdict | Evidence |
|---|---|---|
| platform_oracle Meta platform totals are correct: $2,400 spend, 60 reported conversions, $5,400 reported revenue, blended CPA of $40.00, and ROAS of 2.25. | ✓ | Response computes $2,400 spend, 60 conversions, $5,400 revenue, $40.00 CPA, 2.25x ROAS exactly as required. |
| ga4_oracle GA4 figures for Meta are correct: 36 purchases, $3,240 revenue, CPA of about $66.67, and ROAS of 1.35 against the same $2,400 spend. | ✓ | Response states 36 purchases, $3,240 revenue, CPA ~$66.67, ROAS 1.35x against $2,400 spend. |
| no_single_truth The response never declares one reconciled 'true' Meta conversion count and never averages or splits the difference between 36 and 60 to produce one. | ✓ | The response presents a range (36-60) and explicitly separates 'robust' vs 'source-dependent' conclusions rather than averaging or declaring one true number. |
| defensible_range Meta-attributed purchases are bounded between 36 and 60 (optionally noting the 8 lost-UTM sessions push GA4's floor upward), presented as a range rather than a point estimate. | ✓ | It states 'Defensible Range of Attributed Purchases: 36 to 60 purchases' and notes the 8 lost-UTM sessions could push the floor to 36-44. |
| documented_causes_only The disagreement is explained using only the documented causes — view-through counting, the differing attribution windows/models, the Oct 12-13 lost-UTM sessions classified as Direct, and cross-platform over-claiming (149 claimed versus 118 store orders) — with no invented causes such as privacy signal loss, bot traffic, or tracking-pixel bugs. | ✓ | The four causes listed (view-through, attribution windows, Oct 12-13 UTM outage, 149 vs 118 over-claiming) are all documented in the notes; no invented causes like bots or pixel bugs appear. |
| store_anchor The store platform's 118 completed purchases is treated as the order-count source of truth, and the 149 summed platform claims are identified as over-claiming against it. | ✓ | The response states 'summing all platform reports... totals 149 claimed conversions, even though the store platform recorded only 118 real orders' and later calls this 'non-deduplicated' over-claiming exceeding orders 'by 31 orders.' |
| no_causal_overclaim Neither source's attribution is presented as proof that Meta ads caused the purchases; incrementality is acknowledged as unmeasured. | ✓ | The response explicitly states 'Neither number represents *incrementality*... purchases that occurred strictly because of the ads' and recommends a causal test since 'neither observational attribution model measures true cause-and-effect.' |
| robust_vs_dependent The recommendation explicitly separates at least one conclusion that holds under both sources from at least one that depends on the source chosen, and any spend decision is framed conditionally rather than as settled. | ✓ | The response has explicit 'Robust Conclusions' and 'Source-Dependent Conclusions' sections, e.g. 'Whether the ads printed money' is framed conditionally on which source you believe. |
| grounded_next_step At least one concrete next measurement step comes from the notes (repairing auto-tagging/UTMs, reconciling platform claims against store orders, or aligning attribution settings) rather than from invented tooling. | ✓ | It recommends 'Reconcile the discrepancy between the store's 118 orders and GA4's 112 orders, and audit auto-tagging/UTM pipelines,' both grounded in the notes. |
| Requirement | Verdict | Evidence |
|---|---|---|
| deduped_total The exact duplicate O-009 row is counted once: 13 unique orders have total net revenue of $1,430. | ✓ | Response states 'leaving 13 unique orders' and sums net revenue to $1,430. |
| reconciliation Both models reconcile to $1,330 assigned non-Direct net revenue plus $100 unattributed Direct-only revenue. First-touch splits the $1,330 into $1,120 paid and $210 Organic; last-touch splits it into $1,330 paid and $0 Organic. | ✓ | Reconciliation table shows FT: $1,120 paid + $210 Organic + $100 Direct = $1,430; LT: $1,330 paid + $0 Organic + $100 Direct = $1,430. |
| first_touch_oracle First-touch paid results are correct: Meta $450 / 0.75 ROAS, Paid Search $380 / about 0.63 ROAS, Email $290 / about 0.48 ROAS. | ✓ | FT table lists Meta $450/0.75x, Paid Search $380/0.63x, Email $290/0.48x, matching expected values. |
| last_touch_oracle Last-touch paid results are correct: Meta $400 / about 0.67 ROAS, Paid Search $300 / 0.50 ROAS, Email $630 / 1.05 ROAS. | ✓ | LT table lists Meta $400/0.67x, Paid Search $300/0.50x, Email $630/1.05x, matching expected values. |
| winners_flip Meta is named the first-touch winner and Email the last-touch winner; no single paid winner is claimed across both models. | ✓ | Section (3) names 'First-Touch Winner: Meta' and 'Last-Touch Winner: Email' without declaring a single overall winner. |
| refund_preserved O-013 remains -$120 and is attributed to Meta under first-touch and Email under last-touch; it is not dropped, zeroed, or converted to positive revenue. | ✓ | O-013 is listed as -$120 with Meta as First-Touch and Email as Last-Touch attribution, included negatively in both channel sums. |
| model_definitions The analysis applies the supplied first/last non-Direct definitions consistently and does not invent fractional or platform-reported attribution. | ✓ | The attribution matrix correctly excludes Direct touches (e.g., O-006 Paid Search/Paid Search, O-001 Meta/Email) with no fractional or platform-based credit used. |
| assumptions_and_limit Assumptions are explicit; one consequential data limitation is named with how resolving it could change the recommendation; and attribution is not presented as proof of causation. Valid limitations are not restricted to one prescribed answer. | ✓ | Section (5) lists four explicit assumptions and names lack of incrementality/counterfactual data as the key limitation, explaining how a holdout test could reverse the last-touch recommendation, while explicitly noting attribution 'does not measure incremental lift or establish causation.' |
| conditional_recommendation The recommendation correctly distinguishes model-robust from model-dependent conclusions, does not declare a universal Meta-versus-Email winner, and does not treat either attribution rule as causal grounds for a decisive reallocation. It may either note that Paid Search is not the descriptive winner under these rules or defer all reallocation pending stronger evidence. | ✓ | The response separates 'Robust Conclusions' (e.g., Paid Search underperforms under both models) from 'Model-Dependent Conclusions' (Email's FT vs LT profitability) without declaring a universal Meta-vs-Email winner. |
| auditable_math The response shows enough order grouping, sums, and revenue/spend arithmetic for a reviewer to reproduce both tables. | ✓ | The 'Arithmetic Audit by Channel' section itemizes each order's contribution to Meta, Paid Search, and Email sums under both models, enabling reproduction of the tables. |
| Requirement | Verdict | Evidence |
|---|---|---|
| seo_fields A labeled SEO title and a labeled meta description are both present. | ✓ | The response begins with '**SEO Title:** ...' and '**Meta Description:** ...' clearly labeled. |
| h2_count The body has at least four H2 sections. | ✓ | Programmatic check: h2_count=5. |
| professional_section One descriptive H2 section is specifically about when to call a professional. | ✓ | There is a dedicated H2 'When to Call a Professional for Your North Valley Furnace' covering escalation reasoning. |
| cta The post closes with a call to action for the $95 furnace diagnostic visit. | ✓ | The post ends with 'Contact Brightline Heating today to book your $95 furnace diagnostic visit...' as its closing CTA. |
| length The body is substantive: at least roughly 1,200 words (allow 1,150+). There is no upper limit. | ✓ | Programmatic check: word_count=2159. |
| safety_preserved Nothing instructs the reader to open, bypass, or repair gas or electrical components; dangerous signs (e.g. gas smell) route to shutting off at accessible user controls, leaving, or calling emergency/professional help. | ✓ | The text repeatedly instructs readers to avoid opening panels ('leave the panels closed until a certified professional inspects the unit') and for gas smells tells them to evacuate and call the gas emergency line rather than touching switches or components. |
| no_invented_facts No response-time promises, invented statistics, certifications, review counts, or customer stories; the only price used is the $95 diagnostic. | ✓ | Only the $95 diagnostic price is used, booking language mirrors the source ('Booking availability varies...'), and no certifications, reviews, or customer stories are invented. |
| noise_specificity The post names multiple distinguishable furnace noises and ties each to a safe observation or a clear professional/emergency escalation path rather than generic filler. It is not required to invent a mechanical diagnosis absent from the sources. | ✓ | The post names distinct noises (rattling, squealing, popping, banging at startup, humming/buzzing) and each is tied to either a safe observation or a professional escalation path. |
| actionable_triage A reader can tell which situations are safe to observe versus which require a professional or emergency response. | ✓ | Separate sections clearly separate 'Safe Homeowner Observations' from 'Red Flags... That Require Professional or Emergency Attention,' giving explicit criteria for each category. |
| Requirement | Verdict | Evidence |
|---|---|---|
| minimum_sum_stated The response states that the five committed minimums total exactly $13,500 (3,800 + 4,500 + 2,400 + 1,200 + 1,600), with the arithmetic visible. | ✓ | The response computes '$3,800 + $4,500 + $2,400 + $1,200 + $1,600 = $13,500' explicitly. |
| shortfall_stated The response states the conflict against the $12,000 cap as exactly a $1,500 overage (or equivalently that the cap is $1,500 short of the commitments). | ✓ | It states 'Budget variance / deficit: $12,000 - $13,500 = -$1,500' and 'immediate $1,500 deficit'. |
| infeasibility_declared The response plainly declares that the brief as written is infeasible: no allocation is presented that claims to satisfy every minimum within the $12,000 cap. | ✓ | It states plainly 'No allocation can satisfy every stakeholder commitment as written without violating Finance's rule that not a single dollar over $12,000 will be approved.' |
| no_invented_funds No unsupplied money, discount, negotiated rate, vendor concession, or deferred billing is invented to close the gap. | ✓ | No hidden vendor discounts, concessions, or deferred billing are introduced; Option 1 explicitly asks Finance to raise the cap rather than inventing funds. |
| explicit_constraint_changes Every resolution option names exactly which stated commitment it changes (which minimum is reduced or dropped, or that the cap must rise) rather than silently altering or omitting one. | ✓ | Each option names the exact change: Option 1 raises the cap to $13,500, Option 2 cuts LinkedIn ads by $1,000 and Content refresh by $500, Option 3 cuts webinar by $1,200 and content refresh by $300. |
| auditable_arithmetic Every proposed allocation's line items are numeric and sum exactly to its stated total, and no line in an option that claims to honor a minimum falls below that minimum. | ✓ | All three tables' line items sum correctly to their stated totals (13,500, 12,000, 12,000) and no line claiming to honor a minimum falls below it. |
| multiple_options At least two materially different resolution options are proposed, each with a complete revised line-item allocation. | ✓ | Three materially different resolution options (raise cap, cut sales-facing channels, consolidate webinar/content) are each given a complete line-item allocation. |
| grounded_recommendation One option is recommended with a rationale tied to the brief's or brand's stated goals, without invented performance predictions or unsupported ROI claims. | ✗ | The response never explicitly recommends one option over the others; it ends after Option 3's rationale without a concluding recommendation tied to brand goals. |
| Requirement | Verdict | Evidence |
|---|---|---|
| four_emails_labeled The corrected sequence has exactly four emails, each with all five labeled fields: Timing, Subject, Preheader, Body, and CTA. | ✓ | All four emails include Timing, Subject, Preheader, Body, and CTA fields. |
| sequence_only The deliverable is the corrected sequence itself; accurate draft content (clamp use cases, included accessories, three brightness and two color-temperature settings, audience framing) is preserved rather than discarded. | ✓ | Accurate draft content (clamp use, accessories, brightness/color settings, audience framing) is preserved across the emails. |
| price_corrected The $59 price is gone; if a price appears anywhere it is exactly $68. | ✓ | The $59 price is replaced with '$68' in Email 3 and no other price appears. |
| canonical_name Every product reference uses the canonical name Arc Clip Lamp; the shortened 'Arc Lamp' does not appear. | ✓ | Every reference uses 'Arc Clip Lamp'; the shortened 'Arc Lamp' never appears. |
| battery_corrected The 12-hour battery claim is gone; any battery claim says up to eight hours on the lowest brightness. | ✓ | Email 2 states 'runs up to eight hours on the lowest brightness', matching the allowed battery statement. |
| clamp_corrected The 3-inch clamp claim is gone; any clamp thickness stated is 2.25 inches. | ✓ | Email 1 now reads 'up to 2.25 inches thick' instead of 3 inches. |
| testimonial_removed The invented Maya S. testimonial is absent and no replacement customer quote or review proof is invented. | ✓ | The Maya S. quote is absent from Email 2 with no replacement testimonial. |
| guarantee_removed The 'guaranteed to double your focus' claim is absent and no other guaranteed-outcome claim is added. | ✓ | The 'guaranteed to double your focus' line is removed and no new guarantee is added. |
| urgency_removed The 'Only 50 left' scarcity and the 24-hour deadline are absent and no replacement urgency, scarcity, or countdown tactic is added. | ✓ | 'Only 50 left' and the 24-hour deadline are gone from Email 3 with no replacement urgency tactic. |
| shipping_claim_removed The free 2-day shipping offer is absent and no other unsupplied offer or promotion is added. | ✓ | The free 2-day shipping offer no longer appears anywhere in the sequence. |
| cta_exact Every CTA field is exactly 'Explore the Arc Clip Lamp'; 'Buy Now' does not appear as a CTA. | ✓ | Every CTA reads exactly 'Explore the Arc Clip Lamp'; 'Buy Now' is absent. |
| merge_tag_fixed The malformed '{{first_name}' merge tag is either repaired to a well-formed tag or removed; no malformed personalization token remains. | ✓ | The merge tag is corrected to the well-formed '{{first_name}}' in Email 1. |
| one_job_per_email Each corrected email keeps one job and one primary call to action, consistent with the lifecycle-email guidance. | ✓ | Each email retains a single CTA and a coherent single focus (welcome, education, value/compact-space framing, last touch). |
| no_new_claims No new product fact, capability, audience, proof point, offer, or performance claim appears beyond the supplied sources. | ✗ | Email 3 retains the unsupported economic/performance claim 'it pays for itself in reclaimed desk space alone,' which is not backed by facts.md or product-marketing.md. |
| Requirement | Verdict | Evidence |
|---|---|---|
| full_factorial Exactly four cells A-D are present and cover Renters x Surface-space saving, Renters x Lighting control, Makers x Surface-space saving, and Makers x Lighting control exactly once. | ✓ | Cells A-D cover Renters/Surface, Renters/Lighting, Makers/Surface, Makers/Lighting exactly once. |
| cell_fields Every cell labels Audience, Appeal, Headline, Primary text, Visual direction, CTA, and Creative rationale. | ✓ | Each cell (A-D) lists Audience, Appeal, Headline, Primary text, Visual direction, CTA, and Creative rationale. |
| budget_and_duration The plan reserves a maximum of $4,800 as a $1,200 cap per cell over 14 days; it does not promise that a stopped cell will spend its full cap. | ✓ | Execution rules state '$4,800 reserved' and 'Exactly $1,200 cap per cell' over 14 days, with the stop rule clarifying unused budget from a stopped cell remains unspent rather than guaranteeing full spend. |
| execution_rules_section A labeled Execution rules section is present. | ✓ | A section titled '## Execution rules' is present. |
| hypotheses_section A separate labeled Preregistered hypotheses section is present. | ✓ | A separate section titled '## Preregistered hypotheses' is present. |
| approved_cta All four CTA fields are exactly 'Explore the Arc Clip Lamp'. | ✓ | All four cells list CTA as 'Explore the Arc Clip Lamp' verbatim. |
| grounded_copy All copy and visuals use only supplied Arc Clip Lamp facts and avoid invented customers, results, proof, scarcity, offers, or capabilities. | ✓ | Copy uses only supplied facts (2.25-inch clamp, eight-hour battery, $68 price, included items) with no invented reviews, results, or scarcity. |
| cells_isolate_factors Product identity, offer, copy/visual format, placement, run dates, and measurement rules stay constant; cells vary only the assigned audience language and the assigned appeal's supported message and visual. | ✓ | Execution rules explicitly hold product identity, offer, format/placement, run dates, and measurement rules constant while only audience language and appeal-specific message/visual vary across cells. |
| factorial_hypotheses Exactly three directional landing-page-view-rate hypotheses are preregistered: one audience main effect, one appeal main effect, and one audience-by-appeal interaction; each names its comparison without changing an extra factor. | ✓ | Three directional hypotheses are given: Makers vs Renters (audience main effect), Surface-space saving vs Lighting control (appeal main effect), and a Makers×Lighting-control interaction, each comparing LPVR without altering an extra factor. |
| kpi_and_guardrail Landing-page-view rate is correctly defined as landing-page views / link clicks and cost per AddToCart is the guardrail. | ✓ | LPVR is defined as Landing-Page Views / Link Clicks and Cost per AddToCart is named as the guardrail metric. |
| stop_rule_exact The stop rule is exactly $300 spend with zero AddToCart for an individual cell; unused budget is not reallocated and any triggered stop makes the final result inconclusive. | ✓ | Rule states $300 spend with zero AddToCart triggers a stop, unused budget cannot be reallocated, and any stop makes the result inconclusive. |
| win_rule_exact A winner requires at least 15% relative landing-page-view-rate lift versus the click-weighted pooled rate of the other three cells (their total landing-page views / total link clicks) and at least 200 link clicks in every cell. | ✓ | Win rule requires ≥15% relative lift vs click-weighted pooled rate of the other three cells and ≥200 link clicks in every cell. |
| inconclusive_rule The plan explicitly declares the result inconclusive when the full win rule is not met and does not pick a winner from directional data alone. | ✓ | Inconclusive rule explicitly lists stop, insufficient clicks, or failure to meet the 15% lift as conditions for declaring the test inconclusive rather than picking a winner from directional data. |
| Requirement | Verdict | Evidence |
|---|---|---|
| five_options Exactly five headline options are present and labeled Option 1 through Option 5. | ✓ | Programmatic check: option_labels=[1, 2, 3, 4, 5]. |
| rationales Each option has a one-line rationale explaining the appeal it tests. | ✓ | Each of the 5 options has a rationale line starting with 'Rationale:' explaining the appeal tested. |
| grounded_claims No invented customers, rankings, certifications, guarantees, or outcomes; none of the forbidden claims appear (award-winning, best-selling, clinically proven, limited time, carbon neutral, guaranteed productivity); any price stated is $68. | ✓ | No forbidden claims (awards, guarantees, scarcity, etc.) appear and no price is stated incorrectly. |
| specific_language At least two headlines lead with a concrete supplied product fact (e.g. clamp-anywhere design, up to eight hours battery on lowest brightness, three brightness / two color-temperature settings, $68) rather than generic superlatives. | ✓ | Option 1 leads with the clamp fact (2.25 inches) and Option 2 leads with the lighting settings fact (three brightness, two color temperatures). |
| positioning_fit The set communicates the approved positioning - adaptable light without surrendering permanent surface space, for renters, students, and makers in compact spaces - not generic lamp copy. | ✓ | Option 4 directly states 'Adaptable light for compact spaces, without surrendering permanent surface area,' matching approved positioning, and others reference renters/makers/students and reclaiming surface space. |
| distinct_appeals The five options test meaningfully different appeals or motivations rather than paraphrasing one idea. | ✓ | The five options test different angles—physical fit, lighting adjustability, mobility, brand positioning, and clearing table space—rather than repeating one idea. |
The judge marked this response incomplete — it was cut off at the output-token ceiling — so the task fails regardless of the per-criterion results below.
| Requirement | Verdict | Evidence |
|---|---|---|
| complete_html The response is one complete, parseable HTML document from an HTML5 doctype through the closing html tag, with no Markdown fence. | ✗ | Programmatic check: html_complete=False; html_parse_error_count=2. |
| no_external_assets The HTML uses inline CSS and has no CSS asset references, active content, scripts, images, stylesheets, fonts, media, iframes, embeds, objects, SVG, or canvas assets. | ✓ | Programmatic check: inline_css_count=1; resource_request_count=0; forbidden_asset_tag_count=0; active_content_count=0. |
| duration The repaired page consistently states a 45-minute consultation and removes the flawed 60-minute duration. | ✓ | The page repeatedly states '45-minute' virtual consultation and never mentions the flawed 60-minute duration. |
| price The repaired page consistently states the $160 price and removes the flawed $129 price. | ✓ | The Price section and 'What Is Included' both state $160 with no mention of the flawed $129 price. |
| offering_type The offer is accurately described as one virtual educational routine-planning consultation, not a medical consultation. | ✓ | The intro explicitly calls it a 'virtual routine-planning conversation' for 'non-clinical bedtime habits', not a medical consultation. |
| sections_and_order All requested sections appear in the exact cautious-visitor decision order. | ✗ | The document is truncated mid-sentence in the Price section and never reaches the required Limitations, Objections, or Next Step sections. |
| cta_target The exact CTA 'Book a consultation' appears once or twice and every instance is an anchor whose href is exactly '/consultations/harborline'. | ✓ | Programmatic check: book_cta_count=1; book_cta_count=1; book_cta_all_instances_are_anchors=True; book_cta_all_targeted=True. |
| no_competing_action 'Start sleeping better', 'Call now', and every competing primary action from the flawed draft are absent. | ✓ | Only the 'Book a consultation' CTA appears; 'Start sleeping better' and 'Call now' are absent. |
| educational_scope The page clearly says Harborline is educational coaching, not a medical clinic, and does not diagnose or treat sleep disorders; cure, diagnosis, and treatment claims are absent. | ✓ | The page states 'It is not a medical clinic... does not diagnose or treat medical conditions' with no cure claims. |
| prescription_claim_removed The invented prescription-review service is absent and no replacement clinical service is added. | ✓ | No prescription-review service is mentioned anywhere in the visible text. |
| credential_removed The invented certified-clinician credential is absent and no replacement credential is invented. | ✓ | No certified-clinician credential is mentioned in the response. |
| testimonial_removed The invented Taylor M. testimonial is absent and no replacement patient or customer proof is invented. | ✓ | No testimonial or patient quote appears anywhere in the text. |
| guarantee_removed All cure and guaranteed-improvement claims are absent, and the page explicitly says outcomes are not guaranteed. | ✗ | The document is cut off before ever reaching an explicit 'outcomes are not guaranteed' statement, which is required but absent. |
| insurance_removed The insurance-coverage claim is absent, and the page explicitly says insurance is not represented. | ✗ | The truncated text never includes an explicit statement that insurance is not represented. |
| urgency_removed The three-appointments scarcity claim is absent and no replacement urgency, scarcity, or pressure tactic is added. | ✓ | No scarcity or 'only three appointments' language appears in the visible content. |
| emergency_direction The page says it is not emergency care and directs urgent symptoms or safety concerns to local emergency services or an appropriate licensed clinician. | ✗ | Although an '.emergency-box' CSS class exists, the document is truncated before any emergency-direction text is written. |
| heading_hierarchy The document has exactly one h1 and ordered heading levels with no skipped levels. | ✓ | Programmatic check: h1_count=1; heading_order_valid=True. |
| semantic_landmarks Semantic landmarks and descriptive section headings organize the page; it is not a div-only document. | ✓ | The page uses header, main, and multiple section/h2 elements with ordered lists rather than a div-only structure. |
| conversion_clarity The corrected page makes the educational offer, fit, included value, limitations, price, and next step concrete without replacing the planted defects with vague wellness copy. | ✗ | Because the document cuts off mid-sentence before limitations, objections, and next step, the offer is left incomplete and not fully concrete. |
| Requirement | Verdict | Evidence |
|---|---|---|
| complete_html The response is one complete, parseable HTML document from an HTML5 doctype through the closing html tag, with no Markdown fence. | ✓ | Programmatic check: html_complete=True; html_parse_error_count=0. |
| no_external_assets The HTML uses inline CSS and has no CSS asset references, active content, scripts, images, stylesheets, fonts, media, iframes, embeds, objects, SVG, or canvas assets. | ✓ | Programmatic check: inline_css_count=5; resource_request_count=0; forbidden_asset_tag_count=0; active_content_count=0. |
| sections_complete All required sections exist: relevance/who it is for, how it works, what is included, price ($160 for 45 minutes), limitations/not-for, objections, next step. | ✓ | The page contains distinct sections for overview/who-it's-for, how-it-works, what's included, $160/45-min pricing, limitations/not-for, FAQ objections, and a next-step CTA section. |
| cta_count The exact primary CTA 'Book a consultation' appears once or twice. | ✓ | Programmatic check: book_cta_count=2; book_cta_count=2. |
| no_competing_action No competing primary action is added. | ✓ | The only interactive action elements are two identical 'Book a consultation' anchors styled as the primary CTA; no other buttons or links compete for primary action. |
| decision_order Sections appear in the cautious-visitor order (relevance, mechanism, included value, risk/limitations, objections, next step) with one job and a descriptive heading per section. | ✓ | Sections appear in the order overview → how-it-works → included → pricing → limitations → FAQ → next-step, matching the cautious-visitor decision flow with a descriptive heading in each section. |
| no_dark_patterns No fake urgency, countdown, invented testimonials or reviews, guaranteed outcomes, or hidden terms; disclosures (not medical care, no diagnosis/treatment, not emergency care, outcomes not guaranteed, no insurance representation) are preserved and readable. | ✓ | No countdowns, testimonials, or guaranteed-outcome claims are present, and the footer/limitations section clearly restates 'not a medical clinic,' 'does not diagnose or treat,' 'not emergency care,' 'outcomes are not guaranteed,' and 'insurance coverage is not represented.' |
| heading_hierarchy The document has exactly one h1 and ordered heading levels with no skipped levels. | ✗ | Programmatic check: h1_count=1; heading_order_valid=False. |
| semantic_landmarks Semantic HTML landmarks (header/main/section/footer or equivalent) organize the page; it is not a div-only document. | ✓ | The document uses <header role="banner">, <main role="main">, multiple <section> elements, and <footer role="contentinfo"> rather than a div-only structure. |
| conversion_clarity The primary action is visually and structurally unmistakable, and copy answers visitor questions concretely instead of vague wellness language. | ✓ | The 'Book a consultation' CTA is styled as a prominent button and the surrounding copy gives concrete specifics (45 minutes, $160, intake review, written summary) rather than vague wellness language, even though the second CTA links to a nonexistent '#booking-form' anchor. |
The judge marked this response incomplete — it was cut off at the output-token ceiling — so the task fails regardless of the per-criterion results below.
| Requirement | Verdict | Evidence |
|---|---|---|
| design_shortfall_stated The response states that the committed deliverables require exactly 64 designer hours (16 + 18 + 6 + 24 + 0) against 48 available (12 per week x 4 weeks), a 16-hour shortfall, with the arithmetic visible. | ✓ | The response explicitly states '64 Designer hours' required vs '48 hours (12 hours/week × 4 weeks)' available with a 16-hour deficit, matching the required figures. |
| copy_capacity_confirmed The response states that copywriter hours fit: 92 required (48 + 12 + 12 + 10 + 10) against 96 available (24 x 4). It does not claim the copywriter is over capacity. | ✗ | The response never states the 92-hour copywriter requirement against 96 available hours; the text is cut off before the copywriter total is computed. |
| freeze_conflict_stated The response flags that the required October 28 landing-page go-live falls inside the October 26-30 content freeze, and no proposed plan publishes or launches anything during the freeze. | ✓ | The response explicitly states the Oct 28 go-live 'directly violates' the Oct 26-30 content freeze mandate. |
| infeasibility_declared The response plainly declares that the brief as written cannot be executed: no schedule is presented that claims to deliver every commitment within the stated capacity and calendar. | ✓ | The response states 'the commitments... cannot all be honored simultaneously' and 'no schedule can be produced that delivers the original scope within the stated capacity and calendar constraints.' |
| no_invented_capacity No unapproved freelance, contractor, overtime, or extra-staff hours are counted as available capacity. Proposing approval of the freelance budget as an option is fine only when it is explicitly labeled as requiring a decision leadership has not yet made. | ✓ | The response notes freelance design support 'has not been approved' and does not count it as available capacity in the deficit calculation. |
| explicit_constraint_changes Every resolution option names exactly which stated commitment it changes (which deliverables are cut or reduced, that the go-live date moves, or that new capacity must be approved) rather than silently altering or omitting one. | ✗ | The response is truncated before presenting any resolution options, so no constraint changes are named. |
| auditable_arithmetic Hour totals per person are numeric, follow the brief's per-unit production standards, and sum correctly in the original reconciliation and in every option. | ✗ | The per-deliverable table is cut off mid-row (lifecycle email designer hours, landing page, and video script rows are missing), so the totals are not fully substantiated with visible arithmetic. |
| multiple_options At least two materially different resolution options are proposed, each with a complete revised deliverable list whose copywriter hours fit 96 and designer hours fit 48 (or fit an explicitly approved capacity increase), and a go-live date outside the freeze. | ✗ | The response is truncated before any resolution options are presented, despite promising 'three concrete, fully audited resolution options.' |
| grounded_recommendation One option is recommended with a rationale tied to the launch's or brand's stated goals, without invented performance predictions or unsupported claims about what the content will achieve. | ✗ | No recommendation or rationale is present; the response cuts off during the initial arithmetic audit. |
| Requirement | Verdict | Evidence |
|---|---|---|
| all_sections All seven labeled sections are present: ICP/positioning, channel plan, content calendar, activation metrics, budget allocation, measurement plan, risks. | ✓ | All seven labeled sections (1)-(7) are present and clearly titled. |
| three_channels Exactly three channels are chosen, each with a rationale, and the budget allocation across them sums to the $15,000 total. | ✓ | Exactly three channels (Paid Search, LinkedIn, Lifecycle Email) each have a rationale, and the budget table sums $7,500+$4,500+$3,000=$15,000. |
| activation_anchored Funnel metrics are explicitly tied to the documented activation event (inviting two teammates and completing one shared workflow), not a generic signup metric. | ✓ | The funnel explicitly ties metrics to 'a trial team invites two teammates and completes one shared workflow' with formulas for Teammate Invitation Rate and Workflow Completion Rate. |
| no_invented_proof No invented customers, logos, revenue, market statistics, benchmark conversion rates stated as fact, or testimonials; assumptions are labeled as assumptions. | ✓ | No fabricated customers, revenue, market-share stats, or testimonials appear; figures used are the given $15,000 budget and internal targets, not invented external benchmarks. |
| measurement_defined Each tracked metric names its source/definition (where the number comes from), and the measurement plan is weekly as requested. | ✓ | The weekly measurement table names a specific source (Google Ads Manager, LinkedIn Campaign Manager, Product Telemetry, etc.) for each metric. |
| focused_strategy The plan concentrates on the three chosen channels with reasoning that references the ICP, rather than spreading across every possible tactic. | ✓ | Channel rationales explicitly tie back to the ICP (e.g., LinkedIn targets 'Operations and Delivery titles at 20–250 employee firms'). |
| sequenced_90_days The 90 days are phased (e.g. by month or milestone) with dependencies acknowledged, and the positioning ('faster, visible handoffs without adding another weekly status meeting') carries through the messaging. | ✗ | Only the first four weeks are laid out in detail; there is no explicit month-by-month or milestone phasing for the remaining 60 days with acknowledged dependencies. |
| mature_activation_denominator Activation rate is defined using only trial starts whose full 14-day activation window has closed; newer trials are explicitly pending rather than counted as non-activated. | ✗ | The Trial Activation Rate formula uses 'Total Trial Signups' as the denominator without excluding trials whose 14-day window has not yet closed. |
The judge marked this response incomplete — it was cut off at the output-token ceiling — so the task fails regardless of the per-criterion results below.
| Requirement | Verdict | Evidence |
|---|---|---|
| all_fields_present All eleven labeled fields (RSA-H1, RSA-H2, RSA-H3, RSA-D1, RSA-D2, META-PRIMARY, META-HEADLINE, META-CTA, LI-INTRO, LI-CTA, SMS-BODY) are present; the first line bearing each label is the graded field. | ✗ | Programmatic check: missing_or_duplicate_label_count=11. |
| within_char_limits Every limited field value is at or under its hard character maximum (RSA-H 30, RSA-D 90, META-PRIMARY 125, META-HEADLINE 40, LI-INTRO 150, SMS-BODY 160). | ✓ | Programmatic check: over_limit_field_count=0. |
| approved_cta_exact META-CTA and LI-CTA are each exactly 'Review consultation availability' (a single terminal period is accepted, matching the brand facts' own punctuation). | ✗ | Programmatic check: approved_cta_count=0. |
| disclaimer_everywhere The exact sentence 'Educational only; not medical care.' appears inside META-PRIMARY, LI-INTRO, and SMS-BODY. | ✗ | Programmatic check: disclaimer_field_count=0. |
| sms_optout_present SMS-BODY contains 'Reply STOP to opt out'. | ✗ | Programmatic check: sms_optout_present=False. |
| accurate_price Every stated dollar amount is exactly $160, and the price appears at least once. | ✗ | Programmatic check: non_160_dollar_count=0; price_160_count=0. |
| grounded_claims Every claim uses only the supplied facts (45-minute virtual consultation, $160 paid directly, intake review, routine-planning conversation, written educational summary, adults organizing non-clinical bedtime habits). No outcomes, testimonials, credentials, guarantees, urgency, scarcity, discounts, or clinical framing (diagnosis or treatment) are invented. | ✗ | The response contains no actual ad copy or claims at all, only meta-commentary about CTA punctuation. |
| single_ask Each placement carries one job: getting the reader to review consultation availability. No placement adds a competing ask such as a newsletter signup, follow, download, or second offer. | ✗ | No placements were actually drafted, so there is no ask content to evaluate as single or competing. |
| distinct_rsa_assets The three RSA headlines take meaningfully different angles rather than near-duplicate phrasings, and the two RSA descriptions differ in substance. | ✗ | No RSA headlines or descriptions were produced; the response is entirely a rambling analysis of the word 'exactly' and CTA punctuation. |
| honest_specificity Copy leans on the concrete supplied specifics (45 minutes, virtual, written summary, non-clinical routine help) instead of generic superlatives or vague promises of better sleep. | ✗ | The response never delivers any copy referencing the 45-minute, virtual, or written summary facts. |
| Requirement | Verdict | Evidence |
|---|---|---|
| correct_total The correct October charge is stated as exactly $10,500 (newsletters 6 x $410 = $2,460; sponsored articles 6 x $1,200 = $7,200 minus the 5% volume discount = $6,840; podcast spots 4 x $300 = $1,200), with the arithmetic visible. | ✓ | States 'Correct Audited Amount: $10,500.00' with visible arithmetic for each line (2,460 + 6,840 + 1,200). |
| overbilling_stated The response states the invoice overbills by exactly $1,050 (invoiced $11,550 versus correct $10,500). | ✓ | States 'Total Discrepancy (Overcharge to Dispute): $1,050.00'. |
| newsletter_errors_identified Both newsletter errors are identified: the invoice uses $450 instead of the negotiated $410, and bills 7 placements when the delivery log shows 6 ran and one was canceled. The combined newsletter overcharge is quantified as $690 (or a coherent decomposition that sums to $690). | ✓ | Identifies both the $450 vs $410 rate error and the 7 vs 6 quantity error, summing to a $690 overcharge. |
| discount_missed_identified The missed 5% sponsored-article volume discount is identified as $360, triggered because the $7,200 of sponsored-article charges exceeds the $6,000 monthly threshold. | ✓ | Identifies the missed 5% discount as $360, noting $7,200 exceeds the $6,000 threshold. |
| sow_precedence The negotiated $410 newsletter rate from the agreement is used over the rate card, and the 5% discount is applied only to sponsored-article line items, never to newsletters or podcast spots or the whole invoice. | ✓ | Uses $410 negotiated rate for newsletters and applies the 5% discount only to the sponsored article line, not elsewhere. |
| no_stale_rates Neither the pre-October rates ($400/$1,100/$275) nor the DRAFT 2027 rates ($500/$1,300/$340) are used in any calculation. | ✓ | All calculations use $410, $1,200, $300 (current/negotiated rates); no $400/$1,100/$275 or 2027 draft rates appear. |
| delivery_grounded Billable quantities come from the delivery log under the make-good clause: the canceled October 22 newsletter is not payable, and no delivered placement is dropped. | ✓ | Counts 6 billable newsletters (excluding canceled #207) and confirms all delivered placements are counted, per the delivery log and make-good clause. |
| no_false_disputes The podcast line (4 x $300 = $1,200) is confirmed correct and is not disputed. | ✓ | Line 3 podcast spots are explicitly marked 'CLEAN / ACCURATE. Do not dispute Line 3.' |
| auditable_math Per-line arithmetic (quantities, rates, the discount base and threshold comparison, and every difference) is shown completely enough for a reviewer to reproduce the audit. | ✓ | Each line shows quantity counts, rate comparisons, discount calculation ($7,200 × 0.05 = $360), and difference arithmetic in full detail. |
| actionable_dispute The response ends with a concrete dispute: the itemized discrepancies summing to $1,050, a request for a corrected invoice or credit, and no invented vendor intent, fraud accusation, or threat. | ✓ | Ends with a professional draft email itemizing the $1,050 discrepancy and requesting a corrected invoice or credit memo, with no fraud accusations or threats. |
| Requirement | Verdict | Evidence |
|---|---|---|
| architecture The guide recommends browser Pixel plus server-side Conversions API with deduplication and explains why both are needed under browser privacy limits. | ✓ | The guide explicitly recommends a 'dual-channel hybrid architecture' with browser Pixel and server CAPI, explaining that CAPI is 'immune to ad blockers, DNS filters, and browser runtime errors' while the Pixel captures first-party client context. |
| events_mapped All four standard events (ViewContent, AddToCart, InitiateCheckout, Purchase) are covered with key parameters, including value, currency, and content_ids on Purchase. | ✓ | All four events (ViewContent, AddToCart, InitiateCheckout, Purchase) are listed in a table with parameters including value, currency, and content_ids, with Purchase also including 'contents' and 'num_items'. |
| code_sketch A concrete server-side Purchase event code sketch or request example is included showing event_name, event_time, event_id, action_source, and a user_data block. | ✓ | The Node.js code sketch includes event_name: 'Purchase', event_time computed from order.created_at, event_id derived from order.id, action_source: 'website', and a full user_data block with hashed fields. |
| validation End-to-end validation is described using Meta's Events Manager test events tool and an explicit check that duplicate browser/server events are being deduplicated. | ✓ | Section 6 instructs navigating to 'Meta Events Manager > Data Sources > [Your Pixel] > Test Events' and explicitly checking that 'the event appears with both Browser and Server indicators grouped together under the same row, marked as deduplicated'. |
| sources_defined For each event it is clear where it fires (browser vs server) and what the source of truth is (e.g. Purchase from the Shopify order webhook/server, not only the thank-you page). | ✓ | The Purchase row notes it fires on 'Order confirmation (order_status page load / server order creation)' and the code sketch clarifies the server-side handler listens to Shopify's 'orders/paid' webhook, indicating the webhook/server as the authoritative source. |
| limits_stated Limitations are stated honestly: no promised attribution-recovery percentages, and consent/regional rules are noted as constraints on what may be sent. | ✓ | No attribution-recovery percentage is promised anywhere, and Section 7 frames consent (GDPR/CCPA, Meta LDU) as a hard constraint on what may be sent, e.g. 'ensure the Meta Pixel does not fire before affirmative user consent is registered.' |
| dedup_correct Deduplication is technically correct: the same event_id and event_name sent on both the Pixel and CAPI event so Meta drops the duplicate; not a vague mention. | ✓ | The guide states Meta matches on 'the exact same event_name and exact same event_id' between browser and server payloads and 'discards the redundant event for attribution and delivery reporting.' |
| event_id_retry_stability The event_id is generated or derived once per commerce event, shared by browser and server, and reused unchanged on webhook retries or order updates; a new random suffix per send or retry fails. | ✓ | For Purchase the event_id is derived deterministically as `shopify_order_${order.id}` tied to the stable Shopify Order ID/Token rather than a random per-send value, matching the guidance to 'use the Shopify Order ID, Order Number, or Checkout Token... across both channels.' |
| event_time_stable The server Purchase event uses the purchase/order occurrence timestamp in Unix seconds and preserves it across retries; using the current send or retry time fails. | ✓ | event_time is computed as `Math.floor(new Date(order.created_at).getTime() / 1000)`, using the order's actual creation timestamp rather than the current send time. |
| pii_hashing Customer data handling is correct: personally identifiable fields (e.g. email) are normalized (trimmed, lowercased) and SHA-256 hashed before sending in user_data. | ✓ | The hashData function trims and lowercases the value before SHA-256 hashing, and the table specifies email/name/address fields require SHA-256 hashing after normalization. |
| actionable_path The recommended Shopify path is decisive and executable for a small DTC store (a clear recommendation with concrete setup steps, not an undifferentiated list of options). | ✓ | The guide gives a decisive recommendation: 'the Native Meta Sales Channel App set to \"Maximum\" data sharing is strongly recommended' with concrete reasoning for a single-product DTC store. |
| Requirement | Verdict | Evidence |
|---|---|---|
| two_deliverables There are exactly two labeled deliverables: a Conflict register table and one Corrected concept. | ✓ | Response contains exactly two labeled sections: '# Conflict register' and '# Corrected concept'. |
| twelve_claim_rows The Conflict register has exactly 12 data rows, one per atomic agency claim; product identity and price are separate rows. | ✓ | The table lists exactly 12 numbered rows, with product name and price as separate rows (#1 and #2). |
| corrected_fields The corrected concept includes all seven labeled fields: Primary text, Headline, Description, CTA, Visual direction, Audience insight, and Test hypothesis. | ✓ | The Corrected concept includes all seven labeled fields: Primary text, Headline, Description, CTA, Visual direction, Audience insight, Test hypothesis. |
| product_identity The register rejects 'Arc Lamp', gives the canonical name Arc Clip Lamp, and the corrected concept uses no altered product name. | ✓ | Row 1 rejects 'Arc Lamp' and gives 'Arc Clip Lamp', and the concept consistently uses 'Arc Clip Lamp' only. |
| price The register rejects $59 and gives the supported price of $68; the corrected concept uses no other price if price is mentioned. | ✓ | Row 2 rejects $59 and states $68, and the concept only mentions $68 in the headline and primary text. |
| battery The register rejects '12 hours at any brightness' and gives the supported limit: up to eight hours on the lowest brightness. | ✓ | Row 3 rejects '12 hours at any brightness' and supplies 'up to eight hours on the lowest brightness', matching the concept's primary text. |
| clamp The register rejects the 3-inch claim and gives the supported maximum of 2.25 inches. | ✓ | Row 4 rejects the 3-inch claim and gives 2.25 inches, consistent with the concept's description. |
| included_items The register rejects the adhesive wall mount; the corrected concept does not add it or any other unsupplied accessory. | ✓ | Row 5 rejects the adhesive wall mount and lists the correct included items, which are echoed exactly in the concept without the wall mount. |
| carbon_certification The register rejects the carbon-neutral certification as unsupported; it is absent from the corrected concept. | ✓ | Row 6 rejects the carbon-neutral claim as unsupported and it does not appear anywhere in the corrected concept. |
| bestseller_proof The register rejects the #1 best-selling claim as unsupported; it is absent from the corrected concept. | ✓ | Row 7 rejects the '#1 best-selling' claim, and it is absent from the concept. |
| productivity_guarantee The register rejects the guaranteed 37% productivity claim as unsupported; it is absent from the corrected concept. | ✓ | Row 8 rejects the 37% productivity guarantee, and it is absent from the concept. |
| customer_quote The register rejects the invented Jamie R. quote as unsupported; it is absent from the corrected concept. | ✓ | Row 9 rejects the Jamie R. quote as an invented testimonial, and it is absent from the concept. |
| performance_claim The register rejects the invented 42% conversion lift because no campaign evidence is supplied; it is absent from the corrected concept. | ✓ | Row 10 rejects the 42% conversion lift claim citing no campaign data exists, and it is absent from the concept. |
| scarcity_claim The register rejects 'only 200 remain' because no inventory evidence is supplied; it is absent from the corrected concept. | ✓ | Row 11 rejects the 'Only 200 remain' urgency claim as unsupported, and it is absent from the concept. |
| approved_cta The register rejects 'Shop Now' and the corrected CTA is exactly 'Explore the Arc Clip Lamp'. | ✓ | Row 12 rejects 'Shop Now' and the concept's CTA field reads exactly 'Explore the Arc Clip Lamp'. |
| supported_direction_preserved The corrected concept preserves the useful supported visual idea of moving the lamp between a narrow workspace/shelf and bedside, tied to adaptable light in compact spaces. | ✓ | The Visual direction preserves the shelf-to-bedside movement idea tied to adaptable light in a compact apartment workspace. |
| operational_hypothesis The Test hypothesis names the changed message or visual variable, its comparator, an observable KPI, and the expected direction. | ✓ | The Test hypothesis names the changed visual variable (clamp versatility imagery) versus a comparator (static single-room imagery), with CTR as the KPI and an expected higher direction. |
| no_new_claims The audit and corrected concept introduce no new product fact, customer, proof point, offer, urgency, or performance claim beyond the authoritative sources. | ✓ | No new product facts, customers, proof points, offers, urgency, or performance claims beyond the supplied facts appear in either section. |
The judge marked this response incomplete — it was cut off at the output-token ceiling — so the task fails regardless of the per-criterion results below.
| Requirement | Verdict | Evidence |
|---|---|---|
| covers_all_sources The audit addresses all supplied artifacts: the campaign links, the GTM/GA4 tag configuration, and the consent setup (a clean-bill mention counts as addressing an artifact). | ✓ | The executive summary explicitly references campaign links, GTM/GA4 tag configuration, and the Consent Mode setup as audit domains. |
| findings_located Each reported defect names its specific location — the exact link/channel, tag, trigger, or setting — precisely enough that an engineer could act without re-deriving the audit. | ✗ | Only Defect 1 (consent) is given a precise location (tag/trigger); all other issues are described only in generic, non-specific terms in the executive summary before the response cuts off. |
| finds_uppercase_medium Flags the uppercase utm_medium=CPC values on the two google-search links as violating the lowercase standard and fragmenting paid-search reporting. | ✗ | The response only vaguely mentions 'uppercase UTMs...splinter Paid Search' without identifying the specific google-search links or utm_medium=CPC values. |
| finds_uppercase_medium_pmax Also flags the google-pmax link's uppercase utm_medium=CPC (the same defect on the Performance Max campaign, not only on the search campaigns). | ✗ | No mention of the google-pmax link or its uppercase utm_medium anywhere in the text. |
| finds_missing_source Flags the klaviyo-welcome link as missing utm_source entirely. | ✗ | The klaviyo-welcome link's missing utm_source is never mentioned. |
| finds_meta_source_inconsistency Flags that some Meta links use utm_source=facebook while the standard (and other links) use utm_source=meta, splitting Meta reporting. | ✗ | There is no mention of the facebook vs meta utm_source split anywhere in the response. |
| finds_campaign_spaces Flags the meta-summer link's utm_campaign value ('2026q3 summer sale') for containing spaces / violating the hyphenated campaign format. | ✗ | The meta-summer campaign's spaced utm_campaign value is never referenced. |
| finds_duplicate_purchase_firing Identifies that purchase fires from two live tags (the confirmation-page tag and the place-order button-click tag), double-counting purchases, and connects this to GA4 reporting far more purchases than Shopify. | ✓ | The summary explicitly states 'a redundant client-side GA4 purchase tag firing on checkout button clicks...duplicates conversions' tied to the +80% GA4 vs Shopify discrepancy. |
| finds_missing_transaction_id Identifies that the GA4 purchase tags omit the transaction_id parameter required by the standards, so duplicate purchase events cannot be deduplicated. | ✓ | The executive summary explicitly cites 'a missing transaction_id' as contributing to duplicate conversion counting. |
| finds_hardcoded_currency Identifies that purchase currency is hardcoded to USD while the ca. storefront transacts in CAD, and connects this to the inflated Canadian revenue. | ✓ | Point 3 explicitly describes hardcoded USD currency on the CAD ca. storefront causing ~35% inflated CAD revenue. |
| finds_consent_default Identifies that analytics_storage defaults to granted for EEA/UK visitors, violating the documented Consent Mode defaults, and connects this to the legal/privacy finding. | ✓ | Defect 1 fully details that analytics_storage defaults to granted for EEA/UK, violating standards and causing the legal flag, citing the exact consent-initialization tag. |
| finds_http_links Flags the two http:// newsletter links as violating the https standard and ties them to newsletter traffic landing in Direct (attribution loss on the redirect). | ✓ | The summary states 'Non-secure HTTP links in Klaviyo emails drop parameters to Direct traffic,' matching the symptom about newsletter sessions appearing under Direct. |
| finds_event_name_casing Flags the 'Add to Cart' event name (spaces, not snake_case) as violating the naming standard and explains the add-to-cart reporting going quiet after the rename. | ✓ | The summary attributes the 'Funnel Metric Blackout' to 'A naming convention change on add_to_cart' breaking CRO reporting, matching the theme-migration rename symptom. |
| finds_duplicate_campaign_param Flags the cart-recovery link carrying utm_campaign twice with different values, and ties it to the cart-recovery campaign appearing under two names. | ✗ | The cart-recovery link's duplicate utm_campaign parameter and its two-name reporting symptom are never specifically mentioned. |
| finds_pending_page_regex Identifies that the purchase trigger RegEx ^/order-confirmation also matches /order-confirmation-pending, so purchases fire for unsettled payments, and connects this to the payment-outage / revenue-runs-ahead symptoms. | ✓ | The summary explicitly cites 'an unanchored URL trigger matching /order-confirmation-pending' as capturing unsettled/failed payments. |
| no_false_positives No conforming item is reported as a defect (for example the google-brand link, the begin_checkout tag, the TikTok/Pinterest/affiliate links, or the ca-storefront links' UTMs). Describing a correct item as clean is fine; flagging it as broken is not. | ✓ | No conforming item (e.g., google-brand, begin_checkout, TikTok/Pinterest links) is misidentified as a defect in the visible text. |
| fix_per_finding Every reported defect comes with a concrete, implementable fix (what to change, not 'review this'). | ✗ | The response is truncated mid-explanation of Defect 1 before any 'Fix' section is reached, so no concrete fixes are actually delivered. |
| impact_prioritized Findings are ordered or explicitly tiered by measurement/compliance impact, with the purchase-integrity and consent issues ranked above cosmetic UTM issues. | ✓ | Findings are explicitly ranked (consent/legal first, then purchase duplication, then currency, then event tracking, then attribution splits) per the 'Prioritized Audit Findings' heading. |
| Requirement | Verdict | Evidence |
|---|---|---|
| required_sections The response contains exactly the three required markdown headings: Landing section, Promo email, and Declined items. | ✓ | Programmatic check: required_section_count=3. |
| supported_offer_used The copy uses the $95 furnace diagnostic offer, and every dollar amount in the copy is exactly $95. | ✓ | Programmatic check: copy_price_95_count=3; copy_non_95_dollar_count=0. |
| safety_guidance_included The copy includes the gas-smell safety guidance consistent with the facts: leave the building, avoid switches or flames, and call the local gas emergency line from a safe location — with no invented additional safety advice. | ✓ | The landing section includes the exact three-step gas-smell guidance: 'Leave the building immediately,' 'Do not use any electrical switches, appliances, or open flames,' and 'Call your local gas emergency line from a safe location outside,' with no extra advice. |
| all_unsupported_declined All four unsupportable requests — the 4.9-star/800+ review claim, the Same-Day Service Guarantee, the 'cut your heating bill by up to 30%' line, and the 'Certified Master Technician' credential — are each named under Declined items, and none of them appears in the landing section or email. | ✓ | All four items (reviews, same-day guarantee, 30% savings claim, and certified technician credential) are listed and explained under Declined items, and the measured copy counts (star_rating_count=0, review_claim_count=0, same_day_count=0, certification_count=0, percent_token_count=0) confirm none appear in the copy. |
| no_fabricated_proof The landing section and email contain no star ratings, review counts, percentage claims, same-day promises, or certification language. | ✓ | Programmatic check: copy_star_rating_count=0; copy_review_claim_count=0; copy_percent_token_count=0; copy_same_day_count=0; copy_certification_count=0. |
| no_softened_claims The copy does not smuggle declined claims back in softened form — no 'highly rated', 'trusted by hundreds', 'usually there within hours', 'up to' savings language, invented credentials, or experience proxies standing in for the declined proof. | ✓ | The copy uses only the approved term 'licensed professional' (already present in the source facts) rather than any softened version of 'certified,' 'highly rated,' or 'up to' savings language. |
| grounded_refusals Each declined item cites the supplied facts or policy as the reason (no reviews, rankings, awards, or certifications are supplied; never promise same-day service, energy savings, a specific diagnosis, or a repair price before inspection) rather than generic legal caution. | ✓ | Each Declined item quotes the specific supplied facts (e.g., 'no reviews, rankings, awards, or certifications are supplied' and 'Never promise same-day service...') rather than generic legal caution. |
| service_area_accurate The copy claims only the North Valley service area and does not claim metro-wide coverage; the owner's metro-area request is corrected or declined, not echoed. | ✓ | Both the landing section and email describe service to 'North Valley' only, and the metro-area request is explicitly declined with a North Valley-focused alternative. |
| compliant_alternatives At least three of the four declined items come with a concrete compliant alternative (for example inviting customers to leave reviews, 'call for today's availability' instead of a same-day guarantee, or comfort-focused benefit language without savings percentages). | ✓ | All four declined items (plus the metro-area item) each include a concrete compliant alternative, exceeding the required minimum of three. |
| evidence_unlock The Declined items section tells the owner what evidence or approval would unlock each claim later (for example a review-platform export, a written and honored guarantee policy, or documented certification), so the pushback is actionable rather than a flat no. | ✗ | The alternatives offered (e.g., 'highlight educational approach,' 'mention booking availability,' 'refer to team as licensed professional') are substitute copy ideas, not statements of what future evidence or approval (like a review export or documented certification) would unlock the original claims. |
| Model | Model only model only, over the API · web search | + Marketing Skills model + marketingskills · web search | + Magister full agent · marketingskills + hosted tools |
|---|---|---|---|
| Gemini 3.8 Flash high effort | 78.4 8/20 | 87.2 12/20 | 94.5 14/20 |
Skills vs. no skills changes only the prompt, so that step isolates what the skill pack alone is worth. The Magister column is that same skill pack running inside the full hosted agent, so it also captures what the harness adds on top. Each score links to that configuration's full run.
Every task is a marketing assignment with supplied source material: write the ad concepts, analyze the campaign data, repair the landing page, reconcile the attribution models. Some tasks plant contradictions and defects the model has to catch; some have exact arithmetic a program can check.
Each task carries a checklist of requirements. Local programmatic checks handle the things a program can verify — sums, counts, required phrases, HTML structure. A blinded LLM judge (with no web tools) scores the rest, returning a verdict and evidence for every requirement.
The quality score averages adjusted rubric scores across every task and attempt on a 0–100 scale. Some numeric checks award partial credit; checks already satisfied by a blank answer are excluded. A critical failure or a judgment of incomplete work caps an attempt at 40. A critical failure proved by an exact reference check scores zero, as does missing or errored grading. Truncated answers stay in the task denominator.
Tasks passed is a separate, all-or-nothing measure: one missed requirement fails the task, even when its quality score is high. With one attempt this is Pass@1. With repeated attempts, a task passes only if every attempt passes. Provider errors and truncations fail this measure. A single repetition is a diagnostic snapshot; it does not establish a reliable advantage.
In the skills configuration, each task's prompt carries the relevant verbatim SKILL.md files from marketingskills, the open marketing skill pack by Corey Haines, co-founder of Magister, which is also used by Magister agents. The runner appends the benchmark checkout's pinned skill files to the prompt as plain text under “Skill guidance you should follow”; the no-skills configuration omits those blocks and changes nothing else, so that comparison is a clean ablation of the skill pack itself.
Each skill links to its folder in the marketingskills repo — the exact files the runner inlines. Each task links to every model's answer and verdict for it.
The Magister configuration runs the same tasks through the production Magister agent on an internal canary project. Canary content can precede the stable customer rollout. The prompt carries only the task and its source material; the agent brings its own machine-installed marketingskills pack, hosted tools, and web research, exactly as it does for customers. The candidate model is pinned per chat session and verified from usage metadata, and nothing external is published or mutated. Every task ran in its own fresh chat session, deleted afterwards, so no task could see another task's transcript — the same starting point the direct configurations get.
Cost and token columns for this configuration count every LLM call the agent made during its working turn, so the number is the agent's full bill rather than the cost of its final answer alone. Grading is identical: same rubric, same blinded judge, same quality score and task-pass rules as every other configuration.
Development tasks help us calibrate the rubric by reviewing answers and clarifying requirements. The four tasks labeled holdout in the September 2026 suite have also been inspected. Their original split labels remain in the reports for traceability, but those scores are not untouched validation. A future untouched test requires new, unseen tasks.
The design borrows deliberately: expert-authored per-task rubrics from Harvey's BigLaw Bench, the strict separation of quality from cost and speed from Artificial Analysis, repeat-attempt reliability (pass^k) from τ-bench, and real-deliverable grading in the spirit of OpenAI's GDPval — applied to marketing, with a partial-credit quality score alongside all-or-nothing task passes.
The tasks, fixtures, rubrics, grading code, and raw run evidence live in the eval-runner repository. Reproducing a run is one line: ./eval.py <model>. Judge verdicts are advisory until human-verified, and reports say which is which.