How good is AI at marketing? This benchmark grades real marketing work — writing ad concepts, analyzing campaign data, repairing broken landing pages, reconciling attribution models that disagree — against expert checklists, and runs every model three ways: alone, with the marketing skills pack added, and inside the full Magister agent. Task by task, it shows what today's models are good at and where they struggle.
The model on its own: one API call carrying the task and its source material, nothing added.
The same single API call, with the task's marketingskills SKILL.md files pasted verbatim into the prompt.
The production Magister agent works the task over a full multi-step turn on a real project machine. It brings the marketingskills pack installed on that machine, plus hosted tools and web research — so this configuration is always the model with skills and the harness together, not the model alone.
One model, three configurations, same tasks and same grading. The Magister bar is the full agent — that model plus the marketingskills pack and hosted tools, not the model on its own. A dot on the axis means that configuration hasn't been run for that model yet. Score averages the runner's adjusted rubric scores across all tasks (0–100), including partial credit and penalties for critical failures. Tasks passed is separate: one missed requirement fails the whole task.
| Model | Score?Adjusted rubric score, averaged across every task and attempt, on a 0–100 scale. It includes fractional credit, excludes checks already satisfied by a blank answer, and applies critical-failure and incompleteness penalties. Truncated answers remain in the task denominator. See Methodology for the scoring rules. | Tasks passed?Tasks passed, of tasks attempted (pass@1). A task passes only when the answer meets every requirement in its rubric — one miss fails the whole task, with no partial credit. | Answer cost?What the whole agent turn cost for the suite: every LLM call the Magister agent made while working each task, priced at the provider's list rates. Judge and grading spend are excluded. | Avg task time?Average wall-clock time of the full agent turn per task — research, tools, and writing included. | Output tokens?Total tokens the model generated across all tasks, including any billed reasoning tokens. |
|---|---|---|---|---|---|
| Magister Gemini 3.8 Flash high effort | 94.5 | 14/20 | $2.76 | 108.7s | 169.3k |
Cost and time cover the agent's entire working turn — every model call and tool round-trip, judge overhead excluded. Quality and efficiency are never blended into one number.
Scored 94.5 of 100 and passed 14 of 20 tasks — at $2.76 for the full suite.
| development | holdout | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | ad-creatives | ad-performance-analysis | ads-ledger-analysis | analytics-source-conflict | attribution-sensitivity | blog-post | budget-reconciliation-trap | email-flow-repair | factorial-ad-test | headline | landing-page-repair | landing-page | marketing-plan | shopify-meta-capi | source-conflict-repair | tracking-audit | launch-capacity-trap | platform-limits-gauntlet | rate-card-reconciliation | unsupportable-claims-brief |
| Magister Gemini 3.8 Flash 94.5 14/20 · high effort | Pass9/9 | Fail8/10 | Pass12/12 | Fail8/9 | Fail9/10 | Pass9/9 | Pass8/8 | Pass14/14 | Pass13/13 | Pass6/6 | Pass19/19 | Pass10/10 | Fail5/8 | Fail10/11 | Pass18/18 | Pass18/18 | Pass9/9 | Pass10/10 | Pass10/10 | Fail8/10 |
The count under each verdict is requirements passed. One missed requirement fails the task, so a near-miss and a blowout look the same in the score — the cell shows which one it was.
| Requirement | Verdict | Evidence |
|---|---|---|
| three_concepts Exactly three ad concepts are present. | ✓ | Programmatic check: primary_text_field_count=3. |
| all_fields Each concept has all seven labeled fields: Primary text, Headline, Description, CTA, Visual direction, Audience insight, and Test hypothesis. | ✓ | Each of the three concepts lists Primary text, Headline, Description, CTA, Visual direction, Audience insight, and Test hypothesis explicitly labeled. |
| approved_cta Exactly three CTA fields are present and every one is exactly 'Explore the Arc Clip Lamp'. | ✓ | Programmatic check: cta_field_count=3; approved_cta_field_count=3. |
| distinct_motivations The three concepts are built on distinct audience motivations (per the audience insights), not rewordings of the same angle. | ✓ | Concept 1 targets desk-space reclamation, Concept 2 targets cordless mobility for makers, and Concept 3 targets evening reading comfort—three clearly different motivations. |
| no_invented_proof No invented reviews, customers, performance numbers, scarcity, awards, or product capabilities beyond the supplied facts; forbidden claims (award-winning, best-selling, clinically proven, limited time, carbon neutral, guaranteed productivity) are absent. | ✓ | No reviews, awards, scarcity, environmental claims, or performance guarantees appear; only facts from the source (clamp size, battery life, brightness/temperature settings) are used. |
| canonical_product_name Whenever the product is named, it is called 'Arc Clip Lamp'; shortened or altered names such as 'Arc Lamp' fail. | ✓ | The product is referred to as 'Arc Clip Lamp' consistently throughout all three concepts with no shortened variants. |
| concrete_visuals Each Visual direction describes a specific, filmable/photographable scene tied to the concept's motivation, not vague mood words. | ✓ | Each Visual direction gives a specific filmable scene, e.g. 'Overhead flat-lay photo of a compact apartment desk setup... clamped securely with its padded clamp'. |
| hypothesis_test_design Each Test hypothesis identifies the creative or message variable being tested and the comparator or control it would be tested against. | ✓ | Each Test hypothesis names the variable (e.g., space-saving vs. cordless mobility vs. mood lighting) and compares it against another concept as the control (e.g., 'Concept 1 will outperform Concept 2'). |
| hypothesis_prediction Each Test hypothesis names an observable KPI and its expected direction. Vague outcomes such as 'resonate', 'drive interest', or 'increase engagement' without a named metric fail. | ✓ | Each hypothesis names a concrete KPI and direction, such as 'higher link click-through rate (CTR)' or 'lower Cost Per Click (CPC) and higher engagement rate'. |
| Requirement | Verdict | Evidence |
|---|---|---|
| totals_correct Per-campaign four-week totals are computed and correct within rounding: Prospecting - Broad spend $4,800, 169 conversions, CPA ~$28.40; Prospecting - Small-Space Interest spend $2,400, 138 conversions, CPA ~$17.39; Retargeting - 30d Engagers spend $1,600, 83 conversions, CPA ~$19.28 (or stated over known weeks only); Launch Teaser totals acknowledge the missing week. | ✓ | Spend/conversions/CPA for all three main campaigns exactly match the rubric’s figures ($4,800/169/$28.40, $2,400/138/$17.39, $1,600/83/$19.28) and Launch Teaser's missing week is explicitly flagged rather than summed as zero. |
| best_worst Small-Space Interest is identified as the best performer (lowest CPA ~$17.39, stable, ROAS ~3.9) and Broad as the biggest problem (weekly CPA rising from $20 to $48 while CTR halves from 1.8% to 0.9%); calling the Launch Teaser 'worst' on CPA alone without noting its video-views objective does not pass. | ✓ | Small-Space Interest is named best with CPA $17.39/ROAS 3.91x and stability evidence, and Section 3 separately diagnoses Broad as the biggest problem with CPA rising $20→$48 and CTR halving 1.80%→0.90%, satisfying both required elements even though Launch Teaser is separately labeled 'worst' with its video-views objective explicitly noted. |
| recommendation There is one specific budget change (e.g. shift spend from Broad toward Small-Space Interest) and one bounded test (what change, what metric, what timeframe or budget). | ✓ | The response gives a cohesive budget-reallocation plan (pause Launch Teaser, trim Broad and Retargeting, scale Small-Space Interest) with a stated net budget impact, plus one bounded 14-day creative test with a budget cap for Broad. |
| missing_preserved The empty conversions/revenue cells for Launch Teaser in the week of 2026-07-20 are flagged as missing data (e.g. a tracking gap), not treated as zero or silently summed over. | ✓ | The response explicitly states 'A blank metric is missing/untracked data rather than verified zero conversions' and separates the matched 3-week CPA from a hypothetical zero-treated version. |
| partial_metric_denominators Any Launch Teaser CPA or ROAS uses spend and outcomes from exactly the same complete rows, or is reported as unavailable. Dividing all four weeks of spend by only three known weeks of conversions/revenue fails. | ✗ | The response computes an alternate CPA/ROAS by dividing the full 4-week spend ($1,000) by only the 7 known conversions ('Full 4-Wk Spend' row and calculation $1,000/7=$142.86), which is exactly the mismatched-denominator calculation the rubric flags as failing. |
| evidence_shown The arithmetic behind key claims is shown (at least CPA calculations), and metric definitions/sources are used as supplied rather than invented. | ✓ | Explicit CPA and ROAS formulas with numeric substitutions are shown for every campaign under 'Exact Arithmetic'. |
| cautious_conclusions A limitation that would most change the conclusion is stated (e.g. attribution window, missing week, four-week sample), and correlation is not presented as proven causation. | ✓ | The response names attribution/incrementality (view-through vs first-party tracking) as the limitation most likely to change conclusions and frames results as correlational rather than proven causal ('we cannot determine the incremental net-new revenue'). |
| trend_over_snapshot The diagnosis is driven by week-over-week movement (Broad's CTR/CPA deterioration, Retargeting frequency climbing 6.1 to 11.4 with falling conversions), not four-week averages alone. | ✓ | Section 3 is built entirely on week-over-week figures (Broad CPA $20→$48, CTR 1.80%→0.90%; Retargeting frequency 6.1→11.4 with conversions 26→14) rather than four-week averages. |
| prioritized_action Recommendations are prioritized (one clear primary move) rather than an unranked laundry list. | ✗ | The budget recommendation lists four separate actions (pause, two reductions, one scale-up) as bullet points without designating a single clear primary move, resembling an unranked list. |
| bounded_test_decision The proposed test has one control and one changed variable, a duration or spend cap, one primary KPI, a numeric win rule, and either a numeric minimum sample or an explicit rule for declaring the result inconclusive. | ✓ | The creative test specifies a control vs. new-variant setup, a $300/week cap, a 14-day window with a Day-7 check, CTR/CPA as KPIs, and numeric win/stop rules (pause if CPA>$40 after 150 clicks; graduate if ROAS≥2.5x). |
| Requirement | Verdict | Evidence |
|---|---|---|
| answer_block_complete The response ends with the required answer block and all five labeled answers A1-A5 are present and parseable. | ✓ | Programmatic check: answer_count=5. |
| q1_blended_cpa A1 states the blended CPA (total spend / net conversions) for the full 14 weeks; full credit within 0.5% of the value computed from the export, zero credit beyond 10% off. | ✓ | Programmatic check: answer_q1=25.37. |
| q2_wasted_spend A2 states total spend across zero-conversion adset-days; full credit within 1%, zero credit beyond 25% off. | ✓ | Programmatic check: answer_q2=10228.41. |
| q3_underwater_week A3 names the week (1-14) in which prospecting-broad's weekly net ROAS first fell below 1.0; full credit for the exact week, partial credit within one week, zero credit two or more weeks off. | ✓ | Programmatic check: answer_q3=10.0. |
| q4_refund_drag A4 states the total refunded revenue of the adset with the most refunded revenue; full credit within 1%, zero credit beyond 25% off. | ✓ | Programmatic check: answer_q4=8470.84. |
| q5_final_roas A5 states the account-wide net ROAS for the final 28 days; full credit within 1%, zero credit beyond 15% off. | ✓ | Programmatic check: answer_q5=3.09. |
| refund_adjusted_method The analysis states that conversions and revenue were computed net of refunds (or shows the refund adjustment explicitly) rather than silently using gross figures. | ✓ | The methodology section explicitly computes 'Net Conversions = 2,697 - 156 = 2,541' and 'Net Revenue = Gross Revenue - Refunded Revenue' for every metric. |
| worst_refund_adset_named The review names rt-lookalike as the adset with the largest refunded revenue. | ✓ | The response lists 'rt-lookalike: $8,470.84 (90 refunded conversions)' as the top adset by refunded revenue, matching A4. |
| no_invented_data Every figure presented as coming from the export is derivable from the supplied CSV; no external benchmarks, industry averages, or invented segment data are presented as if they were in the data. | ✓ | All cited figures (spend, revenue, refunds, CVR/CTR/CPC trends) are framed as derived from the CSV with explicit derivation shown, and no external benchmarks or industry averages are cited. |
| prospecting_decline_called The review identifies prospecting-broad's performance decline across the window as a trend requiring action, not merely a low average. | ✓ | Section 4 Issue A 'Severe Prospecting Fatigue & Funnel Starvation' presents a week-by-week ROAS decline table (2.62x to 0.00x) and calls for immediate budget cuts and creative overhaul. |
| pb_video_flagged The review specifically calls out the pb-video adset as continuing to spend while producing zero conversions in the final weeks (or an equivalent adset-level wasted-spend identification naming pb-video). | ✓ | The response states pb-video 'has not generated a single conversion in the last 28 days while burning ~$250/day' and recommends pausing it immediately. |
| recommendation_grounded Each recommendation is tied to specific figures from the analysis (named campaigns/adsets with their numbers), not generic best practice. | ✓ | Recommendations cite specific figures like 'rt-cart (currently 7.23x Net ROAS, $13.25 CPA)' and 'Reduce prospecting-broad daily budget by 60%... from ~$110/day and ~$90/day down to $40/day each'. |
| Requirement | Verdict | Evidence |
|---|---|---|
| platform_oracle Meta platform totals are correct: $2,400 spend, 60 reported conversions, $5,400 reported revenue, blended CPA of $40.00, and ROAS of 2.25. | ✓ | Table correctly shows $2,400 spend, 60 conversions, $5,400 revenue, $40.00 CPA, and 2.25 ROAS for Meta. |
| ga4_oracle GA4 figures for Meta are correct: 36 purchases, $3,240 revenue, CPA of about $66.67, and ROAS of 1.35 against the same $2,400 spend. | ✓ | Table correctly shows GA4 Meta figures as 36 purchases, $3,240 revenue, $66.67 CPA, and 1.35 ROAS against the same $2,400 spend. |
| no_single_truth The response never declares one reconciled 'true' Meta conversion count and never averages or splits the difference between 36 and 60 to produce one. | ✓ | The response explicitly states 'We must not average these numbers into an arbitrary figure (e.g., 48)' and never produces a single reconciled count. |
| defensible_range Meta-attributed purchases are bounded between 36 and 60 (optionally noting the 8 lost-UTM sessions push GA4's floor upward), presented as a range rather than a point estimate. | ✓ | It presents 'somewhere between 36 and 60 total purchases' and a refined 36–44 sub-range accounting for the 8 lost-UTM sessions, as a range rather than a point estimate. |
| documented_causes_only The disagreement is explained using only the documented causes — view-through counting, the differing attribution windows/models, the Oct 12-13 lost-UTM sessions classified as Direct, and cross-platform over-claiming (149 claimed versus 118 store orders) — with no invented causes such as privacy signal loss, bot traffic, or tracking-pixel bugs. | ✗ | The response invents an undocumented cause for the GA4 112 vs store 118 gap, attributing it to 'client-side ad blockers, cookie opt-outs, or browser privacy barriers,' which the notes never mention. |
| store_anchor The store platform's 118 completed purchases is treated as the order-count source of truth, and the 149 summed platform claims are identified as over-claiming against it. | ✓ | It repeatedly treats the store's 118 completed purchases as ground truth and calls the 149 summed platform claims a '26.3% overcount' / 'severe double-counting.' |
| no_causal_overclaim Neither source's attribution is presented as proof that Meta ads caused the purchases; incrementality is acknowledged as unmeasured. | ✓ | The 'Attribution is not Causation' section states neither 60 nor 36 measures incremental lift and recommends a geo-holdout incrementality test to find true causal impact. |
| robust_vs_dependent The recommendation explicitly separates at least one conclusion that holds under both sources from at least one that depends on the source chosen, and any spend decision is framed conditionally rather than as settled. | ✓ | Section 4 explicitly separates 'Findings Robust to the Disagreement' from 'Conditional Findings (What depends on which source you believe)' and frames scaling decisions conditionally. |
| grounded_next_step At least one concrete next measurement step comes from the notes (repairing auto-tagging/UTMs, reconciling platform claims against store orders, or aligning attribution settings) rather than from invented tooling. | ✓ | Concrete next steps include hardening UTM tracking after the Oct 12-13 outage and managing to a Store Total Orders (118) / Ad Spend ratio, both grounded in the notes. |
| Requirement | Verdict | Evidence |
|---|---|---|
| deduped_total The exact duplicate O-009 row is counted once: 13 unique orders have total net revenue of $1,430. | ✓ | Response states '13 unique customer orders' and reconciles total to $1,430.00, matching the correct dedup total. |
| reconciliation Both models reconcile to $1,330 assigned non-Direct net revenue plus $100 unattributed Direct-only revenue. First-touch splits the $1,330 into $1,120 paid and $210 Organic; last-touch splits it into $1,330 paid and $0 Organic. | ✓ | Response's 'Paid Marketing Subtotal (incl. Email)' rows show $1,120 paid/$210 Organic for first-touch and $1,330 paid/$0 Organic for last-touch, both plus $100 Direct-only, matching the required reconciliation. |
| first_touch_oracle First-touch paid results are correct: Meta $450 / 0.75 ROAS, Paid Search $380 / about 0.63 ROAS, Email $290 / about 0.48 ROAS. | ✓ | Table shows Meta $450.00/0.75x, Paid Search $380.00/0.63x, Email $290.00/0.48x under first-touch, matching expected figures. |
| last_touch_oracle Last-touch paid results are correct: Meta $400 / about 0.67 ROAS, Paid Search $300 / 0.50 ROAS, Email $630 / 1.05 ROAS. | ✓ | Table shows Meta $400.00/0.67x, Paid Search $300.00/0.50x, Email $630.00/1.05x under last-touch, matching expected figures. |
| winners_flip Meta is named the first-touch winner and Email the last-touch winner; no single paid winner is claimed across both models. | ✓ | Section 3 names Meta as first-touch winner and Email as the primary last-touch winner among spent channels, without ever declaring one overall winner across both models. |
| refund_preserved O-013 remains -$120 and is attributed to Meta under first-touch and Email under last-touch; it is not dropped, zeroed, or converted to positive revenue. | ✓ | 'O-013 (-$120.00): Touches: [Meta, Email]. First: Meta (-$120), Last: Email (-$120)' preserves the negative value and correct channel assignment in both models. |
| model_definitions The analysis applies the supplied first/last non-Direct definitions consistently and does not invent fractional or platform-reported attribution. | ✓ | The order-level audit consistently applies single-touch first/last non-Direct boundary assignment with no fractional or platform-reported credit introduced. |
| assumptions_and_limit Assumptions are explicit; one consequential data limitation is named with how resolving it could change the recommendation; and attribution is not presented as proof of causation. Valid limitations are not restricted to one prescribed answer. | ✓ | Section 5 lists explicit assumptions, names timestamp/touch-type as the key limitation with concrete impact on the recommendation, and includes a 'Critical Warning: Attribution ≠ Causation' section. |
| conditional_recommendation The recommendation correctly distinguishes model-robust from model-dependent conclusions, does not declare a universal Meta-versus-Email winner, and does not treat either attribution rule as causal grounds for a decisive reallocation. It may either note that Paid Search is not the descriptive winner under these rules or defer all reallocation pending stronger evidence. | ✗ | Despite the robust/dependent split, Section 4C still issues a concrete decisive reallocation ('trim Paid Search from $600 to $300... shift capital to Meta') based on the attribution-derived ROAS comparison rather than deferring pending stronger evidence. |
| auditable_math The response shows enough order grouping, sums, and revenue/spend arithmetic for a reviewer to reproduce both tables. | ✓ | A full 'Complete Order-Level Arithmetic Audit' lists each order's touches, first/last assignment, and dollar value, enabling reproduction of both tables. |
| Requirement | Verdict | Evidence |
|---|---|---|
| seo_fields A labeled SEO title and a labeled meta description are both present. | ✓ | The response opens with '# SEO Title: ...' and '# Meta Description: ...' clearly labeled lines. |
| h2_count The body has at least four H2 sections. | ✓ | Programmatic check: h2_count=5. |
| professional_section One descriptive H2 section is specifically about when to call a professional. | ✓ | There is an H2 titled 'When to Call a Professional: Critical Safety Boundaries' dedicated to professional escalation. |
| cta The post closes with a call to action for the $95 furnace diagnostic visit. | ✓ | The final section 'Get Peace of Mind with Brightline Heating's $95 Diagnostic Visit' explicitly invites the reader to 'Contact Brightline Heating today to schedule your $95 furnace diagnostic visit.' |
| length The body is substantive: at least roughly 1,200 words (allow 1,150+). There is no upper limit. | ✓ | Programmatic check: word_count=2702. |
| safety_preserved Nothing instructs the reader to open, bypass, or repair gas or electrical components; dangerous signs (e.g. gas smell) route to shutting off at accessible user controls, leaving, or calling emergency/professional help. | ✓ | The post explicitly states 'Never Open the Sealed Blower or Burner Compartments,' 'Never Attempt to Clean, Adjust, or Bypass Gas Valves,' and instructs gas-smell readers to evacuate and call from a safe location, never to search for the leak or touch switches. |
| no_invented_facts No response-time promises, invented statistics, certifications, review counts, or customer stories; the only price used is the $95 diagnostic. | ✓ | No same-day/response-time promise, statistics, certifications, reviews, or customer stories appear, and the only price cited throughout is the $95 diagnostic visit. |
| noise_specificity The post names multiple distinguishable furnace noises and ties each to a safe observation or a clear professional/emergency escalation path rather than generic filler. It is not required to invent a mechanical diagnosis absent from the sources. | ✓ | The post names distinct noises (squealing, banging/booming, rattling, rapid clicking, metal popping, grinding, burning smell, gas smell) and links each to either a safe observation or an escalation path in the table and body text. |
| actionable_triage A reader can tell which situations are safe to observe versus which require a professional or emergency response. | ✓ | A dedicated comparison table maps each symptom to 'Safe Homeowner Action' versus 'Required Professional or Emergency Action,' clearly separating what a homeowner can observe from what needs professional/emergency response. |
| Requirement | Verdict | Evidence |
|---|---|---|
| minimum_sum_stated The response states that the five committed minimums total exactly $13,500 (3,800 + 4,500 + 2,400 + 1,200 + 1,600), with the arithmetic visible. | ✓ | Response includes a table summing the five minimums to '$13,500' with explicit arithmetic shown. |
| shortfall_stated The response states the conflict against the $12,000 cap as exactly a $1,500 overage (or equivalently that the cap is $1,500 short of the commitments). | ✓ | It states '$13,500 - $12,000 = $1,500 Deficit' and calls it a '$1,500 overrun (12.5% over the authorized cap)'. |
| infeasibility_declared The response plainly declares that the brief as written is infeasible: no allocation is presented that claims to satisfy every minimum within the $12,000 cap. | ✓ | States 'The brief's commitments cannot all be honored simultaneously within the $12,000 hard budget cap... mathematically impossible' and no option claims to satisfy all five minimums within the cap. |
| no_invented_funds No unsupplied money, discount, negotiated rate, vendor concession, or deferred billing is invented to close the gap. | ✓ | All three options close the gap via explicit scope reductions/deferrals (fewer pages, one webinar, tighter targeting) rather than invented discounts, concessions, or deferred billing. |
| explicit_constraint_changes Every resolution option names exactly which stated commitment it changes (which minimum is reduced or dropped, or that the cap must rise) rather than silently altering or omitting one. | ✓ | Each option's table shows a 'Variance vs. Brief Minimum' column naming exactly which line is reduced and by how much (e.g., 'Webinar Program -$900'). |
| auditable_arithmetic Every proposed allocation's line items are numeric and sum exactly to its stated total, and no line in an option that claims to honor a minimum falls below that minimum. | ✓ | Each option's line items sum exactly to $12,000 (verified: A=12000, B=12000, C=12000) and lines claimed as 'full' match the original minimum exactly. |
| multiple_options At least two materially different resolution options are proposed, each with a complete revised line-item allocation. | ✓ | Three materially different options (A: paid-focus, B: pro-rata, C: content/webinar-focus with LinkedIn cut) each with a complete revised allocation. |
| grounded_recommendation One option is recommended with a rationale tied to the brief's or brand's stated goals, without invented performance predictions or unsupported ROI claims. | ✓ | Option A is recommended with rationale tied to preserving 'commercial sales commitments' and 'legal renewals' and transparent staging, without invented ROI or performance guarantees in the recommendation itself. |
| Requirement | Verdict | Evidence |
|---|---|---|
| four_emails_labeled The corrected sequence has exactly four emails, each with all five labeled fields: Timing, Subject, Preheader, Body, and CTA. | ✓ | All four emails have Timing, Subject, Preheader, Body, and CTA fields. |
| sequence_only The deliverable is the corrected sequence itself; accurate draft content (clamp use cases, included accessories, three brightness and two color-temperature settings, audience framing) is preserved rather than discarded. | ✓ | Accurate draft content like clamp use cases, accessories, brightness/color settings, and audience framing (renters, students, makers) is preserved across the emails. |
| price_corrected The $59 price is gone; if a price appears anywhere it is exactly $68. | ✓ | The $59 price is removed and $68 appears in Email 3. |
| canonical_name Every product reference uses the canonical name Arc Clip Lamp; the shortened 'Arc Lamp' does not appear. | ✓ | All references use 'Arc Clip Lamp'; no shortened 'Arc Lamp' appears. |
| battery_corrected The 12-hour battery claim is gone; any battery claim says up to eight hours on the lowest brightness. | ✓ | Email 2 states 'up to eight hours on the lowest brightness setting' and the 12-hour claim is gone. |
| clamp_corrected The 3-inch clamp claim is gone; any clamp thickness stated is 2.25 inches. | ✓ | Clamp thickness is stated as 2.25 inches in Email 1 and Email 4, replacing the 3-inch claim. |
| testimonial_removed The invented Maya S. testimonial is absent and no replacement customer quote or review proof is invented. | ✓ | The Maya S. quote is absent and no replacement testimonial appears. |
| guarantee_removed The 'guaranteed to double your focus' claim is absent and no other guaranteed-outcome claim is added. | ✓ | The 'guaranteed to double your focus' claim is absent with no replacement guarantee. |
| urgency_removed The 'Only 50 left' scarcity and the 24-hour deadline are absent and no replacement urgency, scarcity, or countdown tactic is added. | ✓ | 'Only 50 left' and the 24-hour deadline are absent with no replacement urgency tactic. |
| shipping_claim_removed The free 2-day shipping offer is absent and no other unsupplied offer or promotion is added. | ✓ | The free 2-day shipping offer is absent and no other promotion is added. |
| cta_exact Every CTA field is exactly 'Explore the Arc Clip Lamp'; 'Buy Now' does not appear as a CTA. | ✓ | Every CTA reads exactly 'Explore the Arc Clip Lamp'. |
| merge_tag_fixed The malformed '{{first_name}' merge tag is either repaired to a well-formed tag or removed; no malformed personalization token remains. | ✓ | The malformed '{{first_name}' is repaired to '{{first_name}}' in Email 1. |
| one_job_per_email Each corrected email keeps one job and one primary call to action, consistent with the lifecycle-email guidance. | ✓ | Each email focuses on a single theme (welcome, battery/education, positioning, last touch) with one CTA each. |
| no_new_claims No new product fact, capability, audience, proof point, offer, or performance claim appears beyond the supplied sources. | ✓ | Email 3's phrasing about 'focused task lighting' and positioning closely mirrors the marketing doc without introducing new facts or proof points. |
| Requirement | Verdict | Evidence |
|---|---|---|
| full_factorial Exactly four cells A-D are present and cover Renters x Surface-space saving, Renters x Lighting control, Makers x Surface-space saving, and Makers x Lighting control exactly once. | ✓ | Table and cells A-D show Renters/Makers x Surface-space saving/Lighting control exactly once each. |
| cell_fields Every cell labels Audience, Appeal, Headline, Primary text, Visual direction, CTA, and Creative rationale. | ✓ | Each of Cell A-D lists Audience, Appeal, Headline, Primary text, Visual direction, CTA, and Creative rationale. |
| budget_and_duration The plan reserves a maximum of $4,800 as a $1,200 cap per cell over 14 days; it does not promise that a stopped cell will spend its full cap. | ✓ | 'Total Budget: Maximum reserve of $4,800' and 'Exactly $1,200 hard cap per cell allocated equally over the 14-day flight', with unspent budget staying unspent if stopped. |
| execution_rules_section A labeled Execution rules section is present. | ✓ | Section '3. Execution Rules' is clearly labeled. |
| hypotheses_section A separate labeled Preregistered hypotheses section is present. | ✓ | Section '2. Preregistered Hypotheses' is clearly labeled and separate. |
| approved_cta All four CTA fields are exactly 'Explore the Arc Clip Lamp'. | ✓ | All four cells list CTA as 'Explore the Arc Clip Lamp' verbatim. |
| grounded_copy All copy and visuals use only supplied Arc Clip Lamp facts and avoid invented customers, results, proof, scarcity, offers, or capabilities. | ✓ | Copy uses only supplied facts (2.25in clamp, 3 brightness/2 color settings, 8hr battery, USB-C) with no invented reviews, awards, or guarantees per the Grounding & Compliance section. |
| cells_isolate_factors Product identity, offer, copy/visual format, placement, run dates, and measurement rules stay constant; cells vary only the assigned audience language and the assigned appeal's supported message and visual. | ✓ | Execution Rules explicitly hold product, offer, format, placement, run dates, and measurement constant while only audience language and appeal-supported message/visual vary across cells. |
| factorial_hypotheses Exactly three directional landing-page-view-rate hypotheses are preregistered: one audience main effect, one appeal main effect, and one audience-by-appeal interaction; each names its comparison without changing an extra factor. | ✓ | Three hypotheses (H1 audience main effect, H2 appeal main effect, H3 audience×appeal interaction) are each stated with directional comparisons on LPVR. |
| kpi_and_guardrail Landing-page-view rate is correctly defined as landing-page views / link clicks and cost per AddToCart is the guardrail. | ✓ | LPVR is defined as Landing-Page Views/Link Clicks and guardrail is defined as Cost per AddToCart. |
| stop_rule_exact The stop rule is exactly $300 spend with zero AddToCart for an individual cell; unused budget is not reallocated and any triggered stop makes the final result inconclusive. | ✓ | 'Any cell that reaches $300 in spend with zero AddToCart events must be stopped immediately' and 'unused budget remains strictly unspent and is not reallocated', with any stop making the test inconclusive. |
| win_rule_exact A winner requires at least 15% relative landing-page-view-rate lift versus the click-weighted pooled rate of the other three cells (their total landing-page views / total link clicks) and at least 200 link clicks in every cell. | ✓ | Winner rule requires ≥200 link clicks per cell and ≥15% relative lift vs click-weighted pooled rate of the other three cells, matching the exact formula given. |
| inconclusive_rule The plan explicitly declares the result inconclusive when the full win rule is not met and does not pick a winner from directional data alone. | ✓ | Explicit 'Inconclusive Decision Rule' section states inconclusive if any cell stops, fails 200-click threshold, or no cell meets the 15% lift, avoiding a winner from directional data alone. |
| Requirement | Verdict | Evidence |
|---|---|---|
| five_options Exactly five headline options are present and labeled Option 1 through Option 5. | ✓ | Programmatic check: option_labels=[1, 2, 3, 4, 5]. |
| rationales Each option has a one-line rationale explaining the appeal it tests. | ✓ | Each of the 5 options includes a one-line 'Rationale:' explaining the appeal it tests. |
| grounded_claims No invented customers, rankings, certifications, guarantees, or outcomes; none of the forbidden claims appear (award-winning, best-selling, clinically proven, limited time, carbon neutral, guaranteed productivity); any price stated is $68. | ✓ | No invented reviews, awards, guarantees, or forbidden claims appear, and no incorrect price is stated. |
| specific_language At least two headlines lead with a concrete supplied product fact (e.g. clamp-anywhere design, up to eight hours battery on lowest brightness, three brightness / two color-temperature settings, $68) rather than generic superlatives. | ✓ | Options 1-3 lead with concrete facts (2.25-inch clamp, eight-hour battery, three brightness/two color settings). |
| positioning_fit The set communicates the approved positioning - adaptable light without surrendering permanent surface space, for renters, students, and makers in compact spaces - not generic lamp copy. | ✓ | Multiple headlines reference 'no permanent surface' and target renters/students/makers, matching approved positioning. |
| distinct_appeals The five options test meaningfully different appeals or motivations rather than paraphrasing one idea. | ✓ | Each option emphasizes a different angle (clamp utility, battery runtime, lighting versatility, mobility, general positioning) rather than repeating one idea. |
| Requirement | Verdict | Evidence |
|---|---|---|
| complete_html The response is one complete, parseable HTML document from an HTML5 doctype through the closing html tag, with no Markdown fence. | ✓ | Programmatic check: html_complete=True; html_parse_error_count=0. |
| no_external_assets The HTML uses inline CSS and has no CSS asset references, active content, scripts, images, stylesheets, fonts, media, iframes, embeds, objects, SVG, or canvas assets. | ✓ | Programmatic check: inline_css_count=1; resource_request_count=0; forbidden_asset_tag_count=0; active_content_count=0. |
| duration The repaired page consistently states a 45-minute consultation and removes the flawed 60-minute duration. | ✓ | The page repeatedly states '45-minute' virtual consultation and never mentions 60 minutes. |
| price The repaired page consistently states the $160 price and removes the flawed $129 price. | ✓ | Price is stated as $160 in the price box, price paragraph, and objections section, with no $129 anywhere. |
| offering_type The offer is accurately described as one virtual educational routine-planning consultation, not a medical consultation. | ✓ | The lead and 'What This Is' section describe it as educational coaching, non-clinical routine planning, not a medical consultation. |
| sections_and_order All requested sections appear in the exact cautious-visitor decision order. | ✓ | Sections appear in order: What This Is and Who It Is For; How the Consultation Works; What Is Included; Price; Limitations and Who It Is Not For; Common Objections; Next Step. |
| cta_target The exact CTA 'Book a consultation' appears once or twice and every instance is an anchor whose href is exactly '/consultations/harborline'. | ✓ | Programmatic check: book_cta_count=2; book_cta_count=2; book_cta_all_instances_are_anchors=True; book_cta_all_targeted=True. |
| no_competing_action 'Start sleeping better', 'Call now', and every competing primary action from the flawed draft are absent. | ✓ | Only 'Book a consultation' CTAs appear (twice) with no 'Start sleeping better' or 'Call now' text present. |
| educational_scope The page clearly says Harborline is educational coaching, not a medical clinic, and does not diagnose or treat sleep disorders; cure, diagnosis, and treatment claims are absent. | ✓ | The footer and limitations section explicitly say 'Educational coaching only. Not a medical clinic' and 'No Diagnosis or Medical Treatment', with no cure claims. |
| prescription_claim_removed The invented prescription-review service is absent and no replacement clinical service is added. | ✓ | The limitations section states 'No Prescriptions: We do not prescribe, review, dispense...' and no clinical prescription service is offered. |
| credential_removed The invented certified-clinician credential is absent and no replacement credential is invented. | ✓ | No mention of 'certified clinicians' or any invented credential appears anywhere in the page. |
| testimonial_removed The invented Taylor M. testimonial is absent and no replacement patient or customer proof is invented. | ✓ | No Taylor M. quote or any patient testimonial appears in the objections or other sections. |
| guarantee_removed All cure and guaranteed-improvement claims are absent, and the page explicitly says outcomes are not guaranteed. | ✓ | The limitations section explicitly states 'No Guaranteed Outcomes: Individual results vary; outcomes and improvements are not guaranteed.' |
| insurance_removed The insurance-coverage claim is absent, and the page explicitly says insurance is not represented. | ✓ | The price section states 'Harborline does not accept, bill, or represent insurance coverage' and limitations reiterate 'insurance coverage is not represented or available.' |
| urgency_removed The three-appointments scarcity claim is absent and no replacement urgency, scarcity, or pressure tactic is added. | ✓ | No scarcity language like 'three appointments remain' appears; the next-step CTA uses neutral phrasing without pressure tactics. |
| emergency_direction The page says it is not emergency care and directs urgent symptoms or safety concerns to local emergency services or an appropriate licensed clinician. | ✓ | The emergency-notice div states 'People with urgent symptoms or safety concerns should contact local emergency services or an appropriate licensed clinician.' |
| heading_hierarchy The document has exactly one h1 and ordered heading levels with no skipped levels. | ✓ | Programmatic check: h1_count=1; heading_order_valid=True. |
| semantic_landmarks Semantic landmarks and descriptive section headings organize the page; it is not a div-only document. | ✓ | The document uses header, main, section, and footer elements with descriptive h2 headings rather than generic divs for structure. |
| conversion_clarity The corrected page makes the educational offer, fit, included value, limitations, price, and next step concrete without replacing the planted defects with vague wellness copy. | ✓ | Each section gives concrete, specific details (fit, process steps, included items, exact price, limitations, next step) without vague wellness filler replacing the removed defects. |
| Requirement | Verdict | Evidence |
|---|---|---|
| complete_html The response is one complete, parseable HTML document from an HTML5 doctype through the closing html tag, with no Markdown fence. | ✓ | Programmatic check: html_complete=True; html_parse_error_count=0. |
| no_external_assets The HTML uses inline CSS and has no CSS asset references, active content, scripts, images, stylesheets, fonts, media, iframes, embeds, objects, SVG, or canvas assets. | ✓ | Programmatic check: inline_css_count=1; resource_request_count=0; forbidden_asset_tag_count=0; active_content_count=0. |
| sections_complete All required sections exist: relevance/who it is for, how it works, what is included, price ($160 for 45 minutes), limitations/not-for, objections, next step. | ✓ | All required sections are present: overview, how-it-works, included, pricing ($160), limitations, faq (objections), and next-step. |
| cta_count The exact primary CTA 'Book a consultation' appears once or twice. | ✓ | Programmatic check: book_cta_count=2; book_cta_count=2. |
| no_competing_action No competing primary action is added. | ✓ | Only two 'Book a consultation' buttons appear, both linking to #next-step, with no other primary action buttons. |
| decision_order Sections appear in the cautious-visitor order (relevance, mechanism, included value, risk/limitations, objections, next step) with one job and a descriptive heading per section. | ✓ | Sections are ordered overview → how-it-works → included → pricing → limitations → faq → next-step, matching the required cautious-visitor order. |
| no_dark_patterns No fake urgency, countdown, invented testimonials or reviews, guaranteed outcomes, or hidden terms; disclosures (not medical care, no diagnosis/treatment, not emergency care, outcomes not guaranteed, no insurance representation) are preserved and readable. | ✓ | The notice-box and FAQ sections clearly state 'not a medical clinic', 'outcomes are not guaranteed', and 'insurance is not accepted or represented' with no fake urgency or testimonials. |
| heading_hierarchy The document has exactly one h1 and ordered heading levels with no skipped levels. | ✓ | Programmatic check: h1_count=1; heading_order_valid=True. |
| semantic_landmarks Semantic HTML landmarks (header/main/section/footer or equivalent) organize the page; it is not a div-only document. | ✓ | The page uses <header>, <main>, multiple <section> elements, and <footer> rather than div-only structure. |
| conversion_clarity The primary action is visually and structurally unmistakable, and copy answers visitor questions concretely instead of vague wellness language. | ✓ | The 'Book a consultation' button is styled distinctly (btn-primary) and the copy concretely explains price, process, and limitations rather than vague wellness language. |
| Requirement | Verdict | Evidence |
|---|---|---|
| design_shortfall_stated The response states that the committed deliverables require exactly 64 designer hours (16 + 18 + 6 + 24 + 0) against 48 available (12 per week x 4 weeks), a 16-hour shortfall, with the arithmetic visible. | ✓ | The response states '64.0 designer hours' required vs '48.0 available' hours, a '16.0-hour deficit' with per-deliverable arithmetic (16+18+6+24+0=64) shown in the table. |
| copy_capacity_confirmed The response states that copywriter hours fit: 92 required (48 + 12 + 12 + 10 + 10) against 96 available (24 x 4). It does not claim the copywriter is over capacity. | ✓ | It states copywriter workload is '92.0 hrs' against '96.0 available hours', a '+4.0 hour' buffer, confirming feasibility rather than overcapacity. |
| freeze_conflict_stated The response flags that the required October 28 landing-page go-live falls inside the October 26-30 content freeze, and no proposed plan publishes or launches anything during the freeze. | ✓ | The response explicitly flags 'Launching on October 28 is an operational violation' of the Oct 26-30 freeze, and all three options move the go-live date to Oct 23 or Nov 2, never publishing during the freeze. |
| infeasibility_declared The response plainly declares that the brief as written cannot be executed: no schedule is presented that claims to deliver every commitment within the stated capacity and calendar. | ✓ | It states plainly: 'cannot be executed as written... Presenting a schedule that claims to fulfill every commitment within the stated constraints would be mathematically and operationally false.' |
| no_invented_capacity No unapproved freelance, contractor, overtime, or extra-staff hours are counted as available capacity. Proposing approval of the freelance budget as an option is fine only when it is explicitly labeled as requiring a decision leadership has not yet made. | ✓ | Option 2's freelance hours are explicitly framed as requiring leadership sign-off ('securing leadership sign-off for 16.0 hours of freelance design support'), not silently counted as available capacity. |
| explicit_constraint_changes Every resolution option names exactly which stated commitment it changes (which deliverables are cut or reduced, that the go-live date moves, or that new capacity must be approved) rather than silently altering or omitting one. | ✓ | Each option states exactly what changes (Option 1 cuts blog/social counts, Option 2 requires freelance approval and date shift, Option 3 extends the calendar to 6 weeks and moves go-live to Nov 2). |
| auditable_arithmetic Hour totals per person are numeric, follow the brief's per-unit production standards, and sum correctly in the original reconciliation and in every option. | ✓ | Per-unit hour calculations follow the brief's standards and weekly totals sum correctly (e.g., Option 1 weekly design hours 11.5+12+12+11.5=47.0 matching the stated total). |
| multiple_options At least two materially different resolution options are proposed, each with a complete revised deliverable list whose copywriter hours fit 96 and designer hours fit 48 (or fit an explicitly approved capacity increase), and a go-live date outside the freeze. | ✓ | Three materially different options are given (in-house scope reduction, approved freelance, phased calendar extension), each with full deliverable lists, hours within capacity (or an explicitly approved increase), and go-live dates outside the freeze (Oct 23, Oct 23, Nov 2). |
| grounded_recommendation One option is recommended with a rationale tied to the launch's or brand's stated goals, without invented performance predictions or unsupported claims about what the content will achieve. | ✓ | The recommendation ties each option to concrete tradeoffs (budget availability, timing flexibility, protecting 'core conversion elements') without inventing performance predictions or unsupported outcome claims. |
| Requirement | Verdict | Evidence |
|---|---|---|
| all_sections All seven labeled sections are present: ICP/positioning, channel plan, content calendar, activation metrics, budget allocation, measurement plan, risks. | ✓ | All seven sections (1)-(7) are explicitly labeled and present in the response. |
| three_channels Exactly three channels are chosen, each with a rationale, and the budget allocation across them sums to the $15,000 total. | ✗ | The budget table includes a fourth line item ('Testing, UX & Creative Reserve' $1,500) so the allocation across the three chosen channels only sums to $13,500, not the full $15,000. |
| activation_anchored Funnel metrics are explicitly tied to the documented activation event (inviting two teammates and completing one shared workflow), not a generic signup metric. | ✓ | The funnel explicitly defines the 'Core Activation Rate' as completing 1 shared workflow after 2 teammates are invited, matching the documented activation event. |
| no_invented_proof No invented customers, logos, revenue, market statistics, benchmark conversion rates stated as fact, or testimonials; assumptions are labeled as assumptions. | ✗ | Numeric benchmarks like CPC $4.00–$5.50, ~3,500 qualified visitors, 60% open rate, 8% reply rate, etc. are stated as concrete targets without being labeled as assumptions or estimates. |
| measurement_defined Each tracked metric names its source/definition (where the number comes from), and the measurement plan is weekly as requested. | ✓ | The weekly measurement table names a specific source system (Google Ads Console, GA4, PostHog/Mixpanel, Stripe, etc.) for every metric and states cadence is weekly. |
| focused_strategy The plan concentrates on the three chosen channels with reasoning that references the ICP, rather than spreading across every possible tactic. | ✓ | Channel rationale ties directly to ICP pain points (e.g., paid search targeting 'Heads of Ops' searching for handoff pain terms) rather than spreading across many tactics. |
| sequenced_90_days The 90 days are phased (e.g. by month or milestone) with dependencies acknowledged, and the positioning ('faster, visible handoffs without adding another weekly status meeting') carries through the messaging. | ✗ | The plan only provides a detailed calendar for weeks 1-4 and scattered week references (e.g., spend pacing weeks 1-2 and 3-8) without a full month-by-month or milestone-based phasing of the entire 90 days. |
| mature_activation_denominator Activation rate is defined using only trial starts whose full 14-day activation window has closed; newer trials are explicitly pending rather than counted as non-activated. | ✓ | The plan states 'Calculated over completed 14-day cohorts. Ongoing trials inside days 1–13 are recorded as pending, never failed,' matching the required mature-denominator rule. |
| Requirement | Verdict | Evidence |
|---|---|---|
| all_fields_present All eleven labeled fields (RSA-H1, RSA-H2, RSA-H3, RSA-D1, RSA-D2, META-PRIMARY, META-HEADLINE, META-CTA, LI-INTRO, LI-CTA, SMS-BODY) are present; the first line bearing each label is the graded field. | ✓ | Programmatic check: missing_or_duplicate_label_count=0. |
| within_char_limits Every limited field value is at or under its hard character maximum (RSA-H 30, RSA-D 90, META-PRIMARY 125, META-HEADLINE 40, LI-INTRO 150, SMS-BODY 160). | ✓ | Programmatic check: over_limit_field_count=0. |
| approved_cta_exact META-CTA and LI-CTA are each exactly 'Review consultation availability' (a single terminal period is accepted, matching the brand facts' own punctuation). | ✓ | Programmatic check: approved_cta_count=2. |
| disclaimer_everywhere The exact sentence 'Educational only; not medical care.' appears inside META-PRIMARY, LI-INTRO, and SMS-BODY. | ✓ | Programmatic check: disclaimer_field_count=3. |
| sms_optout_present SMS-BODY contains 'Reply STOP to opt out'. | ✓ | Programmatic check: sms_optout_present=True. |
| accurate_price Every stated dollar amount is exactly $160, and the price appears at least once. | ✓ | Programmatic check: non_160_dollar_count=0; price_160_count=6. |
| grounded_claims Every claim uses only the supplied facts (45-minute virtual consultation, $160 paid directly, intake review, routine-planning conversation, written educational summary, adults organizing non-clinical bedtime habits). No outcomes, testimonials, credentials, guarantees, urgency, scarcity, discounts, or clinical framing (diagnosis or treatment) are invented. | ✓ | All copy sticks to supplied facts (45-minute virtual consultation, $160 paid directly, intake review, routine-planning conversation, written educational summary, non-clinical bedtime habits) with no outcomes, guarantees, or clinical framing invented. |
| single_ask Each placement carries one job: getting the reader to review consultation availability. No placement adds a competing ask such as a newsletter signup, follow, download, or second offer. | ✓ | Every placement's call to action is exactly 'Review consultation availability' with no competing asks like signups or downloads. |
| distinct_rsa_assets The three RSA headlines take meaningfully different angles rather than near-duplicate phrasings, and the two RSA descriptions differ in substance. | ✓ | RSA-H1 uses the brand name, RSA-H2 uses a generic service descriptor, and RSA-H3 uses format+price, and the two RSA descriptions cover different substance (pricing/format vs. included process steps), so they are not near-duplicates. |
| honest_specificity Copy leans on the concrete supplied specifics (45 minutes, virtual, written summary, non-clinical routine help) instead of generic superlatives or vague promises of better sleep. | ✓ | Copy repeatedly cites concrete specifics (45-minute virtual, $160, written educational summary, non-clinical routine help) rather than vague superlatives about sleep improvement. |
| Requirement | Verdict | Evidence |
|---|---|---|
| correct_total The correct October charge is stated as exactly $10,500 (newsletters 6 x $410 = $2,460; sponsored articles 6 x $1,200 = $7,200 minus the 5% volume discount = $6,840; podcast spots 4 x $300 = $1,200), with the arithmetic visible. | ✓ | Section 5 shows the audited correct total as $10,500.00 with visible arithmetic for each component (6×$410, 6×$1,200 minus 5%, 4×$300). |
| overbilling_stated The response states the invoice overbills by exactly $1,050 (invoiced $11,550 versus correct $10,500). | ✓ | Executive summary states 'Total Amount to Dispute (Net Overcharge): $1,050.00' derived from $11,550 invoiced vs $10,500 correct. |
| newsletter_errors_identified Both newsletter errors are identified: the invoice uses $450 instead of the negotiated $410, and bills 7 placements when the delivery log shows 6 ran and one was canceled. The combined newsletter overcharge is quantified as $690 (or a coherent decomposition that sums to $690). | ✓ | The response identifies both the rate error ($450 vs negotiated $410) and quantity error (7 billed vs 6 ran, with #207 canceled), summing to $690.00 overcharge. |
| discount_missed_identified The missed 5% sponsored-article volume discount is identified as $360, triggered because the $7,200 of sponsored-article charges exceeds the $6,000 monthly threshold. | ✓ | Line 2 breakdown states the 5% discount ($360.00) was omitted because gross $7,200.00 exceeds the $6,000.00 threshold. |
| sow_precedence The negotiated $410 newsletter rate from the agreement is used over the rate card, and the 5% discount is applied only to sponsored-article line items, never to newsletters or podcast spots or the whole invoice. | ✓ | The $410 negotiated rate is used for newsletters, and the 5% discount is applied only to sponsored articles (Line 2), not to newsletters or podcasts. |
| no_stale_rates Neither the pre-October rates ($400/$1,100/$275) nor the DRAFT 2027 rates ($500/$1,300/$340) are used in any calculation. | ✓ | All calculations use October rates ($410 negotiated newsletter, $1,200 sponsored article, $300 podcast); neither pre-Oct ($400/$1,100/$275) nor 2027 draft rates ($500/$1,300/$340) appear anywhere in the response. |
| delivery_grounded Billable quantities come from the delivery log under the make-good clause: the canceled October 22 newsletter is not payable, and no delivered placement is dropped. | ✓ | The response explicitly cites the delivery log to justify billable quantities, excluding the canceled Oct 22 newsletter and including all delivered sponsored articles and podcast spots without dropping any. |
| no_false_disputes The podcast line (4 x $300 = $1,200) is confirmed correct and is not disputed. | ✓ | The podcast line is explicitly marked 'Confirmed Correct' with '$0.00' discrepancy and 'Approved for payment in full as billed.' |
| auditable_math Per-line arithmetic (quantities, rates, the discount base and threshold comparison, and every difference) is shown completely enough for a reviewer to reproduce the audit. | ✓ | Each line shows quantities, rates, discount base/threshold comparison, and difference calculations (e.g., '6 × $410.00 = $2,460.00', '$7,200.00 × 0.05 = $360.00') sufficient to reproduce the audit. |
| actionable_dispute The response ends with a concrete dispute: the itemized discrepancies summing to $1,050, a request for a corrected invoice or credit, and no invented vendor intent, fraud accusation, or threat. | ✓ | The response ends with a concrete dispute amount of $1,050.00, a request for a corrected invoice/credit memo, and a professional email draft with no fraud accusations or threats. |
| Requirement | Verdict | Evidence |
|---|---|---|
| architecture The guide recommends browser Pixel plus server-side Conversions API with deduplication and explains why both are needed under browser privacy limits. | ✓ | Section 1 explicitly recommends 'Browser Pixel + Server-Side CAPI' redundant hybrid architecture and explains browser privacy limits (ITP, ETP, ad blockers) as the reason both are needed. |
| events_mapped All four standard events (ViewContent, AddToCart, InitiateCheckout, Purchase) are covered with key parameters, including value, currency, and content_ids on Purchase. | ✓ | Section 2 provides a table and detailed parameter lists for ViewContent, AddToCart, InitiateCheckout, and Purchase, with value, currency, and content_ids explicitly listed for Purchase. |
| code_sketch A concrete server-side Purchase event code sketch or request example is included showing event_name, event_time, event_id, action_source, and a user_data block. | ✓ | The Node.js sendMetaPurchaseEvent function includes event_name: 'Purchase', event_time, event_id, action_source: 'website', and a user_data block with hashed PII. |
| validation End-to-end validation is described using Meta's Events Manager test events tool and an explicit check that duplicate browser/server events are being deduplicated. | ✓ | Section 6 describes using the Meta Events Manager 'Test Events' tool and explicitly checking that Purchase shows 'Browser • Server' and status 'Deduplicated' with count incrementing by 1. |
| sources_defined For each event it is clear where it fires (browser vs server) and what the source of truth is (e.g. Purchase from the Shopify order webhook/server, not only the thank-you page). | ✓ | The event table specifies triggers (PDP view, cart click, checkout entry) and for Purchase notes 'Order status page / webhook confirmation' with the code sketch built around a Shopify orders/paid webhook, indicating server-side source of truth. |
| limits_stated Limitations are stated honestly: no promised attribution-recovery percentages, and consent/regional rules are noted as constraints on what may be sent. | ✓ | No attribution-recovery percentages are promised anywhere, and Section 7 details GDPR/CCPA consent gating that blocks Pixel/CAPI transmission when consent is denied. |
| dedup_correct Deduplication is technically correct: the same event_id and event_name sent on both the Pixel and CAPI event so Meta drops the duplicate; not a vague mention. | ✓ | Section 3 explains that identical event_name and event_id sent via Pixel and CAPI let Meta match and merge the two records within a 48-hour window, avoiding double counting. |
| event_id_retry_stability The event_id is generated or derived once per commerce event, shared by browser and server, and reused unchanged on webhook retries or order updates; a new random suffix per send or retry fails. | ✓ | The code derives event_id as `shopify_order_${shopifyOrder.id}`, a deterministic value tied to the order ID rather than a random per-send value, ensuring stability across retries. |
| event_time_stable The server Purchase event uses the purchase/order occurrence timestamp in Unix seconds and preserves it across retries; using the current send or retry time fails. | ✗ | The code sets `event_time: currentTimestamp` using `Math.floor(Date.now() / 1000)` at send time rather than the order's actual purchase/occurrence timestamp, which would change on retries. |
| pii_hashing Customer data handling is correct: personally identifiable fields (e.g. email) are normalized (trimmed, lowercased) and SHA-256 hashed before sending in user_data. | ✓ | Section 4 and the code's hashUserData function trim and lowercase fields like email before applying SHA-256 hashing per Meta's spec. |
| actionable_path The recommended Shopify path is decisive and executable for a small DTC store (a clear recommendation with concrete setup steps, not an undifferentiated list of options). | ✓ | The guide gives a decisive recommendation to 'Deploy the Native Meta Sales Channel App with Data Sharing set to Maximum' with concrete rationale and setup steps for a small DTC store. |
| Requirement | Verdict | Evidence |
|---|---|---|
| two_deliverables There are exactly two labeled deliverables: a Conflict register table and one Corrected concept. | ✓ | Response contains distinct '### Conflict register' and '### Corrected concept' headings. |
| twelve_claim_rows The Conflict register has exactly 12 data rows, one per atomic agency claim; product identity and price are separate rows. | ✓ | The table has exactly 12 numbered rows, splitting the product-name and price bullet into separate rows 1 and 2. |
| corrected_fields The corrected concept includes all seven labeled fields: Primary text, Headline, Description, CTA, Visual direction, Audience insight, and Test hypothesis. | ✓ | Corrected concept lists Primary text, Headline, Description, CTA, Visual direction, Audience insight, and Test hypothesis, all seven fields. |
| product_identity The register rejects 'Arc Lamp', gives the canonical name Arc Clip Lamp, and the corrected concept uses no altered product name. | ✓ | Row 1 rejects 'Arc Lamp' and gives 'Arc Clip Lamp'; concept uses only 'Arc Clip Lamp' throughout. |
| price The register rejects $59 and gives the supported price of $68; the corrected concept uses no other price if price is mentioned. | ✓ | Row 2 rejects $59 and states 'Price is $68'; concept mentions only $68 in Primary text and Headline. |
| battery The register rejects '12 hours at any brightness' and gives the supported limit: up to eight hours on the lowest brightness. | ✓ | Row 3 rejects '12 hours at any brightness' and gives 'up to eight hours on the lowest brightness'; concept repeats this exact phrasing. |
| clamp The register rejects the 3-inch claim and gives the supported maximum of 2.25 inches. | ✓ | Row 4 rejects 3 inches and gives 2.25 inches; concept's Description says 'surfaces up to 2.25"'. |
| included_items The register rejects the adhesive wall mount; the corrected concept does not add it or any other unsupplied accessory. | ✓ | Row 5 rejects the adhesive wall mount and lists only verified items; Primary text lists 'padded clamp, USB-C charging cable, and storage sleeve' with no wall mount. |
| carbon_certification The register rejects the carbon-neutral certification as unsupported; it is absent from the corrected concept. | ✓ | Row 6 rejects the carbon-neutral claim and it does not appear anywhere in the corrected concept. |
| bestseller_proof The register rejects the #1 best-selling claim as unsupported; it is absent from the corrected concept. | ✓ | Row 7 rejects the #1 best-selling claim and it is absent from the corrected concept. |
| productivity_guarantee The register rejects the guaranteed 37% productivity claim as unsupported; it is absent from the corrected concept. | ✓ | Row 8 rejects the 37% productivity guarantee and it is absent from the corrected concept. |
| customer_quote The register rejects the invented Jamie R. quote as unsupported; it is absent from the corrected concept. | ✓ | Row 9 rejects the Jamie R. quote and it does not appear in the corrected concept. |
| performance_claim The register rejects the invented 42% conversion lift because no campaign evidence is supplied; it is absent from the corrected concept. | ✓ | Row 10 rejects the 42% conversion lift claim citing 'no campaign has launched'; absent from corrected concept. |
| scarcity_claim The register rejects 'only 200 remain' because no inventory evidence is supplied; it is absent from the corrected concept. | ✓ | Row 11 rejects the 'only 200 remain' urgency line and it is absent from the corrected concept. |
| approved_cta The register rejects 'Shop Now' and the corrected CTA is exactly 'Explore the Arc Clip Lamp'. | ✓ | Row 12 rejects 'Shop Now' and the CTA field reads exactly 'Explore the Arc Clip Lamp'. |
| supported_direction_preserved The corrected concept preserves the useful supported visual idea of moving the lamp between a narrow workspace/shelf and bedside, tied to adaptable light in compact spaces. | ✓ | Visual direction keeps the lamp moving from a narrow shelf/desk workspace to a bedside table, tied to adaptable, space-saving light. |
| operational_hypothesis The Test hypothesis names the changed message or visual variable, its comparator, an observable KPI, and the expected direction. | ✓ | Test hypothesis names the 'adaptable space-saving angle focused on clamp versatility' vs a 'general ambient lighting concept' comparator, with KPI 'thumb-stop rate ... and CTR' and expected direction 'higher'. |
| no_new_claims The audit and corrected concept introduce no new product fact, customer, proof point, offer, urgency, or performance claim beyond the authoritative sources. | ✓ | No new product facts, testimonials, proof points, offers, urgency, or performance numbers beyond the supplied facts appear in the register or corrected concept. |
| Requirement | Verdict | Evidence |
|---|---|---|
| covers_all_sources The audit addresses all supplied artifacts: the campaign links, the GTM/GA4 tag configuration, and the consent setup (a clean-bill mention counts as addressing an artifact). | ✓ | The report addresses campaign links, GTM/GA4 tags, and consent setup, including clean-bill mentions for each (e.g., Tag 1, Tag 5, Consent Mode non-EEA defaults, and multiple link groups). |
| findings_located Each reported defect names its specific location — the exact link/channel, tag, trigger, or setting — precisely enough that an engineer could act without re-deriving the audit. | ✓ | Each finding cites an exact link number/channel or GTM tag/trigger (e.g., 'GTM Tag 3 (#place-order-btn)', 'Link #8 (meta-retargeting)') precise enough to act on. |
| finds_uppercase_medium Flags the uppercase utm_medium=CPC values on the two google-search links as violating the lowercase standard and fragmenting paid-search reporting. | ✓ | Finding 7 flags Links #1 and #2 (google-search) for uppercase utm_medium=CPC violating the lowercase standard and splitting paid search. |
| finds_uppercase_medium_pmax Also flags the google-pmax link's uppercase utm_medium=CPC (the same defect on the Performance Max campaign, not only on the search campaigns). | ✓ | Finding 7 also lists Link #3 (google-pmax) with the same uppercase utm_medium=CPC defect. |
| finds_missing_source Flags the klaviyo-welcome link as missing utm_source entirely. | ✓ | Finding 6 flags Link #14 (klaviyo-welcome) as missing utm_source entirely. |
| finds_meta_source_inconsistency Flags that some Meta links use utm_source=facebook while the standard (and other links) use utm_source=meta, splitting Meta reporting. | ✓ | Finding 10 flags Links #5 and #7 using utm_source=facebook instead of the standardized 'meta', splitting Meta reporting. |
| finds_campaign_spaces Flags the meta-summer link's utm_campaign value ('2026q3 summer sale') for containing spaces / violating the hyphenated campaign format. | ✓ | Finding 11 flags Link #9 (meta-summer)'s '2026q3 summer sale' campaign value for containing spaces. |
| finds_duplicate_purchase_firing Identifies that purchase fires from two live tags (the confirmation-page tag and the place-order button-click tag), double-counting purchases, and connects this to GA4 reporting far more purchases than Shopify. | ✓ | Finding 1 identifies Tag 3 (click trigger) firing purchase alongside Tag 2 (confirmation page), causing double counting and linking it to the 512 vs 284 discrepancy. |
| finds_missing_transaction_id Identifies that the GA4 purchase tags omit the transaction_id parameter required by the standards, so duplicate purchase events cannot be deduplicated. | ✓ | Finding 1 states Tag 3 'completely omits transaction_id' and Finding 2's fix adds transaction_id to Tag 2 for deduplication, addressing the missing parameter across purchase tags. |
| finds_hardcoded_currency Identifies that purchase currency is hardcoded to USD while the ca. storefront transacts in CAD, and connects this to the inflated Canadian revenue. | ✓ | Finding 3 identifies hardcoded currency:'USD' on Tags 2, 3, 6 and ties it directly to the ~33% inflated Canadian revenue via FX conversion. |
| finds_consent_default Identifies that analytics_storage defaults to granted for EEA/UK visitors, violating the documented Consent Mode defaults, and connects this to the legal/privacy finding. | ✓ | Finding 4 identifies EEA/UK analytics_storage defaulting to 'granted' instead of 'denied', linking it to the legal/privacy flag. |
| finds_http_links Flags the two http:// newsletter links as violating the https standard and ties them to newsletter traffic landing in Direct (attribution loss on the redirect). | ✓ | Finding 5 flags Links #12 and #13's http:// protocol and ties the redirect-based parameter stripping to newsletter sessions appearing as Direct. |
| finds_event_name_casing Flags the 'Add to Cart' event name (spaces, not snake_case) as violating the naming standard and explains the add-to-cart reporting going quiet after the rename. | ✓ | Finding 8 flags Tag 4's 'Add to Cart' event name violating snake_case and explains the resulting reporting drop after the theme migration. |
| finds_duplicate_campaign_param Flags the cart-recovery link carrying utm_campaign twice with different values, and ties it to the cart-recovery campaign appearing under two names. | ✓ | Finding 9 flags Link #8 carrying utm_campaign twice with different values and ties it to the cart-recovery campaign appearing under two names. |
| finds_pending_page_regex Identifies that the purchase trigger RegEx ^/order-confirmation also matches /order-confirmation-pending, so purchases fire for unsettled payments, and connects this to the payment-outage / revenue-runs-ahead symptoms. | ✓ | Finding 2 identifies the unanchored regex ^/order-confirmation matching /order-confirmation-pending, causing premature purchase firing and linking it to the payment-outage revenue symptom. |
| no_false_positives No conforming item is reported as a defect (for example the google-brand link, the begin_checkout tag, the TikTok/Pinterest/affiliate links, or the ca-storefront links' UTMs). Describing a correct item as clean is fine; flagging it as broken is not. | ✓ | All clean items listed (Tag 1, Tag 5, non-EEA consent defaults, ca-storefront links, TikTok, affiliate, google-brand, Klaviyo winback, Instagram) are genuinely conforming per the standards, and the extra Finding 12 (organic-pinterest campaign format) reflects a real deviation from the three-segment campaign convention rather than a false flag. |
| fix_per_finding Every reported defect comes with a concrete, implementable fix (what to change, not 'review this'). | ✓ | Every one of the 12 findings includes an explicit 'Concrete Fix' section with exact parameter/URL changes. |
| impact_prioritized Findings are ordered or explicitly tiered by measurement/compliance impact, with the purchase-integrity and consent issues ranked above cosmetic UTM issues. | ✓ | Findings are explicitly tiered (Critical/High/Medium/Low) with duplicate purchase firing and consent issues ranked above cosmetic UTM defects like spacing and source casing. |
| Requirement | Verdict | Evidence |
|---|---|---|
| required_sections The response contains exactly the three required markdown headings: Landing section, Promo email, and Declined items. | ✓ | Programmatic check: required_section_count=3. |
| supported_offer_used The copy uses the $95 furnace diagnostic offer, and every dollar amount in the copy is exactly $95. | ✓ | Programmatic check: copy_price_95_count=5; copy_non_95_dollar_count=0. |
| safety_guidance_included The copy includes the gas-smell safety guidance consistent with the facts: leave the building, avoid switches or flames, and call the local gas emergency line from a safe location — with no invented additional safety advice. | ✓ | Both the landing section and email include a numbered gas-smell safety notice: 'leave the building,' 'do not operate any electrical switches...or open flames,' and 'call your local gas emergency line from a safe location.' |
| all_unsupported_declined All four unsupportable requests — the 4.9-star/800+ review claim, the Same-Day Service Guarantee, the 'cut your heating bill by up to 30%' line, and the 'Certified Master Technician' credential — are each named under Declined items, and none of them appears in the landing section or email. | ✓ | The Declined items section lists the 4.9-star/800+ review claim, Same-Day Service Guarantee, 'cut your heating bill by up to 30%' line, and 'Certified Master Technician' credential, and none of these phrases appear in the landing section or email (matching copy_star_rating_count:0, copy_same_day_count:0, copy_percent_token_count:0, copy_certification_count:0). |
| no_fabricated_proof The landing section and email contain no star ratings, review counts, percentage claims, same-day promises, or certification language. | ✓ | Programmatic check: copy_star_rating_count=0; copy_review_claim_count=0; copy_percent_token_count=0; copy_same_day_count=0; copy_certification_count=0. |
| no_softened_claims The copy does not smuggle declined claims back in softened form — no 'highly rated', 'trusted by hundreds', 'usually there within hours', 'up to' savings language, invented credentials, or experience proxies standing in for the declined proof. | ✗ | The copy twice asserts 'Our licensed technicians' as a credential claim not explicitly supported by the supplied facts (only product-marketing.md mentions 'licensed professional' in the context of when customers should call for help, not as a stated attribute of Brightline's own staff), functioning as a softened stand-in for the declined 'Certified Master Technician' claim. |
| grounded_refusals Each declined item cites the supplied facts or policy as the reason (no reviews, rankings, awards, or certifications are supplied; never promise same-day service, energy savings, a specific diagnosis, or a repair price before inspection) rather than generic legal caution. | ✓ | Each declined item quotes specific facts.md or product-marketing.md language (e.g., 'No reviews, rankings, awards, or certifications are supplied' and 'Never promise same-day service, energy savings...') rather than generic caution. |
| service_area_accurate The copy claims only the North Valley service area and does not claim metro-wide coverage; the owner's metro-area request is corrected or declined, not echoed. | ✓ | The copy consistently states 'North Valley' as the service area, and the metro-area request is explicitly declined with the reason 'Service area: North Valley.' |
| compliant_alternatives At least three of the four declined items come with a concrete compliant alternative (for example inviting customers to leave reviews, 'call for today's availability' instead of a same-day guarantee, or comfort-focused benefit language without savings percentages). | ✓ | All five declined items (exceeding the required three) include a concrete compliant alternative, e.g., 'call for today's availability' language and 'licensed technicians' phrasing instead of the guarantee/certification. |
| evidence_unlock The Declined items section tells the owner what evidence or approval would unlock each claim later (for example a review-platform export, a written and honored guarantee policy, or documented certification), so the pushback is actionable rather than a flat no. | ✗ | Only the review-claim item states what would unlock it later (a verified Google Business Profile export); the same-day guarantee, 30% savings, certification, and metro-area items give alternatives but no stated path or evidence that would unlock the original claim. |
| Model | Model only model only, over the API · web search | + Marketing Skills model + marketingskills · web search | + Magister full agent · marketingskills + hosted tools |
|---|---|---|---|
| Gemini 3.8 Flash high effort | 78.4 8/20 | 87.2 12/20 | 94.5 14/20 |
Skills vs. no skills changes only the prompt, so that step isolates what the skill pack alone is worth. The Magister column is that same skill pack running inside the full hosted agent, so it also captures what the harness adds on top. Each score links to that configuration's full run.
Every task is a marketing assignment with supplied source material: write the ad concepts, analyze the campaign data, repair the landing page, reconcile the attribution models. Some tasks plant contradictions and defects the model has to catch; some have exact arithmetic a program can check.
Each task carries a checklist of requirements. Local programmatic checks handle the things a program can verify — sums, counts, required phrases, HTML structure. A blinded LLM judge (with no web tools) scores the rest, returning a verdict and evidence for every requirement.
The quality score averages adjusted rubric scores across every task and attempt on a 0–100 scale. Some numeric checks award partial credit; checks already satisfied by a blank answer are excluded. A critical failure or a judgment of incomplete work caps an attempt at 40. A critical failure proved by an exact reference check scores zero, as does missing or errored grading. Truncated answers stay in the task denominator.
Tasks passed is a separate, all-or-nothing measure: one missed requirement fails the task, even when its quality score is high. With one attempt this is Pass@1. With repeated attempts, a task passes only if every attempt passes. Provider errors and truncations fail this measure. A single repetition is a diagnostic snapshot; it does not establish a reliable advantage.
In the skills configuration, each task's prompt carries the relevant verbatim SKILL.md files from marketingskills, the open marketing skill pack by Corey Haines, co-founder of Magister, which is also used by Magister agents. The runner appends the benchmark checkout's pinned skill files to the prompt as plain text under “Skill guidance you should follow”; the no-skills configuration omits those blocks and changes nothing else, so that comparison is a clean ablation of the skill pack itself.
Each skill links to its folder in the marketingskills repo — the exact files the runner inlines. Each task links to every model's answer and verdict for it.
The Magister configuration runs the same tasks through the production Magister agent on an internal canary project. Canary content can precede the stable customer rollout. The prompt carries only the task and its source material; the agent brings its own machine-installed marketingskills pack, hosted tools, and web research, exactly as it does for customers. The candidate model is pinned per chat session and verified from usage metadata, and nothing external is published or mutated. Every task ran in its own fresh chat session, deleted afterwards, so no task could see another task's transcript — the same starting point the direct configurations get.
Cost and token columns for this configuration count every LLM call the agent made during its working turn, so the number is the agent's full bill rather than the cost of its final answer alone. Grading is identical: same rubric, same blinded judge, same quality score and task-pass rules as every other configuration.
Development tasks help us calibrate the rubric by reviewing answers and clarifying requirements. The four tasks labeled holdout in the September 2026 suite have also been inspected. Their original split labels remain in the reports for traceability, but those scores are not untouched validation. A future untouched test requires new, unseen tasks.
The design borrows deliberately: expert-authored per-task rubrics from Harvey's BigLaw Bench, the strict separation of quality from cost and speed from Artificial Analysis, repeat-attempt reliability (pass^k) from τ-bench, and real-deliverable grading in the spirit of OpenAI's GDPval — applied to marketing, with a partial-credit quality score alongside all-or-nothing task passes.
The tasks, fixtures, rubrics, grading code, and raw run evidence live in the eval-runner repository. Reproducing a run is one line: ./eval.py <model>. Judge verdicts are advisory until human-verified, and reports say which is which.