How good is AI at marketing? This benchmark grades real marketing work — writing ad concepts, analyzing campaign data, repairing broken landing pages, reconciling attribution models that disagree — against expert checklists, and runs every model three ways: alone, with the marketing skills pack added, and inside the full Magister agent. Task by task, it shows what today's models are good at and where they struggle.
The model on its own: one API call carrying the task and its source material, nothing added.
The same single API call, with the task's marketingskills SKILL.md files pasted verbatim into the prompt.
The production Magister agent works the task over a full multi-step turn on a real project machine. It brings the marketingskills pack installed on that machine, plus hosted tools and web research — so this configuration is always the model with skills and the harness together, not the model alone.
One model, three configurations, same tasks and same grading. The Magister bar is the full agent — that model plus the marketingskills pack and hosted tools, not the model on its own. A dot on the axis means that configuration hasn't been run for that model yet. Score averages the runner's adjusted rubric scores across all tasks (0–100), including partial credit and penalties for critical failures. Tasks passed is separate: one missed requirement fails the whole task.
| Model | Score?Adjusted rubric score, averaged across every task and attempt, on a 0–100 scale. It includes fractional credit, excludes checks already satisfied by a blank answer, and applies critical-failure and incompleteness penalties. Truncated answers remain in the task denominator. See Methodology for the scoring rules. | Tasks passed?Tasks passed, of tasks attempted (pass@1). A task passes only when the answer meets every requirement in its rubric — one miss fails the whole task, with no partial credit. | Answer cost?What this model's answers cost for the whole suite: its input and output tokens priced at the provider's list rates. Judge and grading spend are excluded. | Avg task time?Average wall-clock time of the model's API call per task, as measured by the runner. | Output tokens?Total tokens the model generated across all tasks, including any billed reasoning tokens. |
|---|---|---|---|---|---|
| Gemini 3.8 Flash high effort | 87.2 | 12/20 | $1.69 | 151.9s | 425.5k |
Cost and time are the candidate model's alone — judge overhead is excluded. Quality and efficiency are never blended into one number.
Scored 87.2 of 100 and passed 12 of 20 tasks — at $1.69 for the full suite.
| development | holdout | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | ad-creatives | ad-performance-analysis | ads-ledger-analysis | analytics-source-conflict | attribution-sensitivity | blog-post | budget-reconciliation-trap | email-flow-repair | factorial-ad-test | headline | landing-page-repair | landing-page | marketing-plan | shopify-meta-capi | source-conflict-repair | tracking-audit | launch-capacity-trap | platform-limits-gauntlet | rate-card-reconciliation | unsupportable-claims-brief |
| Gemini 3.8 Flash 87.2 12/20 · high effort | Pass9/9 | Fail9/10 | Fail0/0 | Pass9/9 | Pass0/0 | Pass9/9 | Fail7/8 | Fail13/14 | Pass13/13 | Pass6/6 | Pass19/19 | Fail9/10 | Fail7/8 | Pass11/11 | Pass18/18 | Fail0/0 | Fail0/0 | Pass10/10 | Pass10/10 | Pass10/10 |
The count under each verdict is requirements passed. One missed requirement fails the task, so a near-miss and a blowout look the same in the score — the cell shows which one it was.
Skill guidance from ad-creative — folders in the marketingskills pack.
| Requirement | Verdict | Evidence |
|---|---|---|
| three_concepts Exactly three ad concepts are present. | ✓ | Programmatic check: primary_text_field_count=3. |
| all_fields Each concept has all seven labeled fields: Primary text, Headline, Description, CTA, Visual direction, Audience insight, and Test hypothesis. | ✓ | Each of the three concepts includes labeled Primary text, Headline, Description, CTA, Visual direction, Audience insight, and Test hypothesis fields. |
| approved_cta Exactly three CTA fields are present and every one is exactly 'Explore the Arc Clip Lamp'. | ✓ | Programmatic check: cta_field_count=3; approved_cta_field_count=3. |
| distinct_motivations The three concepts are built on distinct audience motivations (per the audience insights), not rewordings of the same angle. | ✓ | Concept 1 targets renters wanting clear surfaces, Concept 2 targets makers needing workbench clearance, and Concept 3 targets students needing cordless mobility across shifting study spaces—three clearly distinct motivations. |
| no_invented_proof No invented reviews, customers, performance numbers, scarcity, awards, or product capabilities beyond the supplied facts; forbidden claims (award-winning, best-selling, clinically proven, limited time, carbon neutral, guaranteed productivity) are absent. | ✓ | No reviews, awards, scarcity, guarantees, or unapproved claims appear; all factual claims (price, clamp size, brightness settings, battery life, USB-C) match the supplied facts.md. |
| canonical_product_name Whenever the product is named, it is called 'Arc Clip Lamp'; shortened or altered names such as 'Arc Lamp' fail. | ✓ | The product is referred to as 'Arc Clip Lamp' consistently in all headlines and body copy, e.g., 'Arc Clip Lamp: Save Your Table Space'. |
| concrete_visuals Each Visual direction describes a specific, filmable/photographable scene tied to the concept's motivation, not vague mood words. | ✓ | Each Visual direction describes a specific filmable scene, e.g., 'clamped to the vertical edge of a narrow wooden shelf... casts a warm downward pool of light' for Concept 1. |
| hypothesis_test_design Each Test hypothesis identifies the creative or message variable being tested and the comparator or control it would be tested against. | ✓ | Each hypothesis names a comparator, e.g., Concept 2 states it 'will generate a lower CPC... compared to Concept 1'. |
| hypothesis_prediction Each Test hypothesis names an observable KPI and its expected direction. Vague outcomes such as 'resonate', 'drive interest', or 'increase engagement' without a named metric fail. | ✓ | Each hypothesis names an observable KPI and direction, e.g., Concept 3 predicts 'a higher Outbound CTR and landing page conversion rate (CVR) than Concept 1'. |
Skill guidance from analytics — folders in the marketingskills pack.
| Requirement | Verdict | Evidence |
|---|---|---|
| totals_correct Per-campaign four-week totals are computed and correct within rounding: Prospecting - Broad spend $4,800, 169 conversions, CPA ~$28.40; Prospecting - Small-Space Interest spend $2,400, 138 conversions, CPA ~$17.39; Retargeting - 30d Engagers spend $1,600, 83 conversions, CPA ~$19.28 (or stated over known weeks only); Launch Teaser totals acknowledge the missing week. | ✓ | The response computes Broad spend $4,800/169 conv/CPA $28.40, Small-Space $2,400/138 conv/CPA $17.39, Retargeting $1,600/83 conv/CPA $19.28, matching expected totals exactly, and explicitly flags Launch Teaser's missing week rather than silently summing it in. |
| best_worst Small-Space Interest is identified as the best performer (lowest CPA ~$17.39, stable, ROAS ~3.9) and Broad as the biggest problem (weekly CPA rising from $20 to $48 while CTR halves from 1.8% to 0.9%); calling the Launch Teaser 'worst' on CPA alone without noting its video-views objective does not pass. | ✓ | Small-Space Interest is named best with CPA $17.39/ROAS 3.91x and stability evidence, and while Launch Teaser is labeled 'worst', the response explicitly notes its video_views objective mismatch as a caveat, and Broad's CPA-rising/CTR-halving deterioration is thoroughly documented with the exact evidence cited in the rubric. |
| recommendation There is one specific budget change (e.g. shift spend from Broad toward Small-Space Interest) and one bounded test (what change, what metric, what timeframe or budget). | ✓ | The response specifies a concrete budget reallocation (pause Launch Teaser, cut Retargeting 50%, shift $450/week into Small-Space Interest) and a bounded 14-day/$600-capped creative A/B test with defined KPIs. |
| missing_preserved The empty conversions/revenue cells for Launch Teaser in the week of 2026-07-20 are flagged as missing data (e.g. a tracking gap), not treated as zero or silently summed over. | ✓ | Section 6 explicitly flags the blank Week 3 Launch Teaser row as ambiguous/missing tracking data rather than treating it as zero, discussing implications for CPA if nulls vs zeros are assumed. |
| partial_metric_denominators Any Launch Teaser CPA or ROAS uses spend and outcomes from exactly the same complete rows, or is reported as unavailable. Dividing all four weeks of spend by only three known weeks of conversions/revenue fails. | ✗ | Despite presenting a 'matched' 3-week CPA, the response also computes and reports a 'Full-Period Spend (4 weeks)' CPA of $142.86 by dividing all four weeks of spend ($1,000) by only 3 weeks' known conversions (7), which the rubric explicitly says fails even if labeled. |
| evidence_shown The arithmetic behind key claims is shown (at least CPA calculations), and metric definitions/sources are used as supplied rather than invented. | ✓ | The response shows explicit CPA and ROAS formulas and step-by-step arithmetic for every campaign, using the CSV's supplied spend/conversions/revenue fields rather than invented figures. |
| cautious_conclusions A limitation that would most change the conclusion is stated (e.g. attribution window, missing week, four-week sample), and correlation is not presented as proven causation. | ✓ | Section 5 identifies missing view-through attribution and holdout data as the limitation that would most change conclusions, and hedges with language like 'highly probable' rather than asserting causation. |
| trend_over_snapshot The diagnosis is driven by week-over-week movement (Broad's CTR/CPA deterioration, Retargeting frequency climbing 6.1 to 11.4 with falling conversions), not four-week averages alone. | ✓ | Section 3 diagnoses the trend via week-over-week CTR halving (1.80%→0.90%) and CPA rise ($20→$48) in Broad, and frequency climbing 6.1→11.4 with falling conversions in Retargeting, not four-week averages. |
| prioritized_action Recommendations are prioritized (one clear primary move) rather than an unranked laundry list. | ✓ | The budget change is presented as one clear headline action (pause Video Views, rebalance into Small-Space) with numbered implementation steps, rather than an unranked list of unrelated options. |
| bounded_test_decision The proposed test has one control and one changed variable, a duration or spend cap, one primary KPI, a numeric win rule, and either a numeric minimum sample or an explicit rule for declaring the result inconclusive. | ✓ | The proposed creative test specifies control vs. variant, a 14-day duration, a $300/week spend cap, a primary KPI (CTR ≥1.6% and CPA ≤$25), and a numeric kill-switch/inconclusive rule (>$150 spend with zero conversions or CPA >$45 over 7 days). |
Skill guidance from analytics — folders in the marketingskills pack.
The judge marked this response incomplete — it was cut off at the output-token ceiling — so the task fails regardless of the per-criterion results below.
| Requirement | Verdict | Evidence |
|---|---|---|
| answer_block_complete The response ends with the required answer block and all five labeled answers A1-A5 are present and parseable. | ✗ | Programmatic check: answer_count=0. |
| q1_blended_cpa A1 states the blended CPA (total spend / net conversions) for the full 14 weeks; full credit within 0.5% of the value computed from the export, zero credit beyond 10% off. | ✗ | Programmatic check: answer_q1=None. |
| q2_wasted_spend A2 states total spend across zero-conversion adset-days; full credit within 1%, zero credit beyond 25% off. | ✗ | Programmatic check: answer_q2=None. |
| q3_underwater_week A3 names the week (1-14) in which prospecting-broad's weekly net ROAS first fell below 1.0; full credit for the exact week, partial credit within one week, zero credit two or more weeks off. | ✗ | Programmatic check: answer_q3=None. |
| q4_refund_drag A4 states the total refunded revenue of the adset with the most refunded revenue; full credit within 1%, zero credit beyond 25% off. | ✗ | Programmatic check: answer_q4=None. |
| q5_final_roas A5 states the account-wide net ROAS for the final 28 days; full credit within 1%, zero credit beyond 15% off. | ✗ | Programmatic check: answer_q5=None. |
| refund_adjusted_method The analysis states that conversions and revenue were computed net of refunds (or shows the refund adjustment explicitly) rather than silently using gross figures. | ✓ | The response explicitly says 'After accounting for $14,405.42 in refunded revenue and 156 refunded orders' to derive net revenue and net conversions. |
| worst_refund_adset_named The review names rt-lookalike as the adset with the largest refunded revenue. | ✗ | The response text provided contains only the executive summary table and never names rt-lookalike or any adset-level refund figures. |
| no_invented_data Every figure presented as coming from the export is derivable from the supplied CSV; no external benchmarks, industry averages, or invented segment data are presented as if they were in the data. | ✓ | All figures shown are aggregate totals presented as derived from the CSV with no external benchmarks or invented data visible in the response. |
| prospecting_decline_called The review identifies prospecting-broad's performance decline across the window as a trend requiring action, not merely a low average. | ✗ | The response does not mention prospecting-broad's performance trend or decline anywhere in the provided text. |
| pb_video_flagged The review specifically calls out the pb-video adset as continuing to spend while producing zero conversions in the final weeks (or an equivalent adset-level wasted-spend identification naming pb-video). | ✗ | There is no mention of pb-video or any adset-level wasted spend callout in the response. |
| recommendation_grounded Each recommendation is tied to specific figures from the analysis (named campaigns/adsets with their numbers), not generic best practice. | ✗ | The response ends after the summary table with no recommendations section at all. |
Skill guidance from analytics — folders in the marketingskills pack.
| Requirement | Verdict | Evidence |
|---|---|---|
| platform_oracle Meta platform totals are correct: $2,400 spend, 60 reported conversions, $5,400 reported revenue, blended CPA of $40.00, and ROAS of 2.25. | ✓ | Response computes spend $2,400.00, conversions 60, revenue $5,400.00, CPA $40.00, and ROAS 2.25× exactly as required. |
| ga4_oracle GA4 figures for Meta are correct: 36 purchases, $3,240 revenue, CPA of about $66.67, and ROAS of 1.35 against the same $2,400 spend. | ✓ | Response computes GA4 Paid Social figures as 36 purchases, $3,240.00 revenue, CPA $66.67, and ROAS 1.35× against $2,400 spend. |
| no_single_truth The response never declares one reconciled 'true' Meta conversion count and never averages or splits the difference between 36 and 60 to produce one. | ✓ | The response explicitly states 'Do Not Average the Numbers' and never proposes a single reconciled conversion count. |
| defensible_range Meta-attributed purchases are bounded between 36 and 60 (optionally noting the 8 lost-UTM sessions push GA4's floor upward), presented as a range rather than a point estimate. | ✓ | It states 'The Defensible Range is 36 to 60 Purchases' and notes the GA4 floor excludes the 8 untagged Meta sessions, pushing the true floor higher. |
| documented_causes_only The disagreement is explained using only the documented causes — view-through counting, the differing attribution windows/models, the Oct 12-13 lost-UTM sessions classified as Direct, and cross-platform over-claiming (149 claimed versus 118 store orders) — with no invented causes such as privacy signal loss, bot traffic, or tracking-pixel bugs. | ✓ | Causes cited are view-through counting, differing attribution logic, the Oct 12-13 UTM outage, and the 149-vs-118 over-claiming, all sourced from the notes with no invented causes like bots or pixel bugs. |
| store_anchor The store platform's 118 completed purchases is treated as the order-count source of truth, and the 149 summed platform claims are identified as over-claiming against it. | ✓ | The response calls the store platform 'the absolute source of truth for completed orders' at 118 and frames the 149 platform claims as over-claiming against it. |
| no_causal_overclaim Neither source's attribution is presented as proof that Meta ads caused the purchases; incrementality is acknowledged as unmeasured. | ✓ | A dedicated subsection states 'Attribution is Not Proof of Causation' and 'Neither source measures incremental lift.' |
| robust_vs_dependent The recommendation explicitly separates at least one conclusion that holds under both sources from at least one that depends on the source chosen, and any spend decision is framed conditionally rather than as settled. | ✓ | Section A lists conclusions 'Robust to the Measurement Disagreement' and Section B lists conclusions 'Dependent on Which Source You Believe,' with conditional framing in Section C. |
| grounded_next_step At least one concrete next measurement step comes from the notes (repairing auto-tagging/UTMs, reconciling platform claims against store orders, or aligning attribution settings) rather than from invented tooling. | ✓ | The response recommends 'Fix the auto-tagging infrastructure to prevent UTM stripping outages,' a step grounded directly in the tracking notes. |
Skill guidance from analytics — folders in the marketingskills pack.
| Requirement | Verdict | Evidence |
|---|---|---|
| deduped_total The exact duplicate O-009 row is counted once: 13 unique orders have total net revenue of $1,430. | ✓ | Response states '13 unique orders' with total reconciled net revenue of $1,430 after removing the duplicate O-009 row. |
| reconciliation Both models reconcile to $1,330 assigned non-Direct net revenue plus $100 unattributed Direct-only revenue. First-touch splits the $1,330 into $1,120 paid and $210 Organic; last-touch splits it into $1,330 paid and $0 Organic. | ✓ | FT reconciliation shows $1,120 paid + $210 Organic + $100 Direct = $1,430, and LT shows $1,330 paid + $0 Organic + $100 Direct = $1,430, matching the expected split. |
| first_touch_oracle First-touch paid results are correct: Meta $450 / 0.75 ROAS, Paid Search $380 / about 0.63 ROAS, Email $290 / about 0.48 ROAS. | ✓ | Table lists Meta $450/0.75x, Paid Search $380/0.63x, Email $290/0.48x under First-Touch, matching the oracle values. |
| last_touch_oracle Last-touch paid results are correct: Meta $400 / about 0.67 ROAS, Paid Search $300 / 0.50 ROAS, Email $630 / 1.05 ROAS. | ✓ | Table lists Meta $400/0.67x, Paid Search $300/0.50x, Email $630/1.05x under Last-Touch, matching the oracle values. |
| winners_flip Meta is named the first-touch winner and Email the last-touch winner; no single paid winner is claimed across both models. | ✓ | Deliverable 3 names Meta as the First-Touch winner and Email as the Last-Touch winner without claiming one channel wins under both models. |
| refund_preserved O-013 remains -$120 and is attributed to Meta under first-touch and Email under last-touch; it is not dropped, zeroed, or converted to positive revenue. | ✓ | The journey table lists O-013 as -$120 attributed to Meta (first-touch) and Email (last-touch), and the reconciliation arithmetic explicitly subtracts $120 from Meta (FT) and Email (LT). |
| model_definitions The analysis applies the supplied first/last non-Direct definitions consistently and does not invent fractional or platform-reported attribution. | ✓ | The response consistently applies single first/last non-Direct touch rules (stripping trailing Direct touches) without introducing fractional or platform-reported credit. |
| assumptions_and_limit Assumptions are explicit; one consequential data limitation is named with how resolving it could change the recommendation; and attribution is not presented as proof of causation. Valid limitations are not restricted to one prescribed answer. | ✓ | Five explicit assumptions are listed and a data limitation (absence of incrementality/holdout testing) is named with concrete examples of how resolving it (e.g., Email iROAS collapsing to 0.21x) could change the recommendation, alongside an explicit statement that attribution models are 'purely descriptive accounting rules' not causal proof. |
| conditional_recommendation The recommendation correctly distinguishes model-robust from model-dependent conclusions, does not declare a universal Meta-versus-Email winner, and does not treat either attribution rule as causal grounds for a decisive reallocation. It may either note that Paid Search is not the descriptive winner under these rules or defer all reallocation pending stronger evidence. | ✓ | The response separates 'Robust Conclusions' (e.g., Paid Search underperforms in both models) from 'Model-Dependent Conclusions' (Email's viability flips between models) and avoids naming a single overall Meta-vs-Email winner. |
| auditable_math The response shows enough order grouping, sums, and revenue/spend arithmetic for a reviewer to reproduce both tables. | ✓ | Detailed per-order arithmetic blocks (e.g., 'Meta: $120 (O-001) + $80 (O-002) + ... = $450') are shown for both First-Touch and Last-Touch reconciliations, enabling reproduction of the tables. |
Skill guidance from copywriting — folders in the marketingskills pack.
| Requirement | Verdict | Evidence |
|---|---|---|
| seo_fields A labeled SEO title and a labeled meta description are both present. | ✓ | The response begins with explicitly labeled '**SEO Title:**' and '**Meta Description:**' lines. |
| h2_count The body has at least four H2 sections. | ✓ | Programmatic check: h2_count=6. |
| professional_section One descriptive H2 section is specifically about when to call a professional. | ✓ | There is a dedicated H2 section titled '## When to Call a Professional for Your North Valley Heating System'. |
| cta The post closes with a call to action for the $95 furnace diagnostic visit. | ✓ | The post closes with 'Call Brightline Heating today or schedule your $95 furnace diagnostic visit online to restore quiet, dependable heat to your North Valley home.' |
| length The body is substantive: at least roughly 1,200 words (allow 1,150+). There is no upper limit. | ✓ | Programmatic check: word_count=2802. |
| safety_preserved Nothing instructs the reader to open, bypass, or repair gas or electrical components; dangerous signs (e.g. gas smell) route to shutting off at accessible user controls, leaving, or calling emergency/professional help. | ✓ | The text explicitly states 'Homeowners must never open electrical service panels, reach into mechanical cabinets, bypass internal safety switches, or attempt repairs on gas train components' and routes gas smell to evacuation and calling the gas utility, never instructing gas/electrical tampering. |
| no_invented_facts No response-time promises, invented statistics, certifications, review counts, or customer stories; the only price used is the $95 diagnostic. | ✓ | The post only mentions the $95 diagnostic price, notes booking availability 'varies' without promising a response time, and presents an itemized estimate only after inspection, with no fabricated reviews, certifications, or customer stories. |
| noise_specificity The post names multiple distinguishable furnace noises and ties each to a safe observation or a clear professional/emergency escalation path rather than generic filler. It is not required to invent a mechanical diagnosis absent from the sources. | ✓ | The post names distinct noises—squealing, rattling, popping, humming, grinding, repeated ignition clicking, burning smell with hum, and short cycling—each tied to a safe observation or an escalation path. |
| actionable_triage A reader can tell which situations are safe to observe versus which require a professional or emergency response. | ✓ | A clear table and section structure separate safe homeowner observations (filter check, timing, airflow) from emergency/professional situations (gas smell, grinding, ignition issues, burning smell, short cycling). |
Skill guidance from analytics — folders in the marketingskills pack.
| Requirement | Verdict | Evidence |
|---|---|---|
| minimum_sum_stated The response states that the five committed minimums total exactly $13,500 (3,800 + 4,500 + 2,400 + 1,200 + 1,600), with the arithmetic visible. | ✓ | The response includes a table summing $3,800+$4,500+$2,400+$1,200+$1,600 = $13,500 with explicit arithmetic. |
| shortfall_stated The response states the conflict against the $12,000 cap as exactly a $1,500 overage (or equivalently that the cap is $1,500 short of the commitments). | ✓ | It explicitly states 'an unavoidable deficit of $1,500 (a 12.50% budget overage)' against the $12,000 cap. |
| infeasibility_declared The response plainly declares that the brief as written is infeasible: no allocation is presented that claims to satisfy every minimum within the $12,000 cap. | ✓ | The response states the minimums 'cannot all be honored simultaneously' and labels the baseline as exceeding the cap rather than presenting it as compliant. |
| no_invented_funds No unsupplied money, discount, negotiated rate, vendor concession, or deferred billing is invented to close the gap. | ✓ | No discounts, vendor concessions, or deferred billing are invented; options either ask Finance for more budget or explicitly cut scope, not manufacture funds. |
| explicit_constraint_changes Every resolution option names exactly which stated commitment it changes (which minimum is reduced or dropped, or that the cap must rise) rather than silently altering or omitting one. | ✓ | Each option names exactly which commitment changes, e.g., Option B says 'Sales VP must concede a $1,500 reduction in LinkedIn advertising' and Option A says Finance must grant a cap exception. |
| auditable_arithmetic Every proposed allocation's line items are numeric and sum exactly to its stated total, and no line in an option that claims to honor a minimum falls below that minimum. | ✓ | Each option's line items are shown to sum exactly to $12,000 or $13,500 with explicit reconciliation checks, and no untouched line falls below its stated minimum. |
| multiple_options At least two materially different resolution options are proposed, each with a complete revised line-item allocation. | ✓ | Four distinct options (A: raise cap, B: trim LinkedIn, C: trim webinars/content, D: proportional cuts) are each given complete revised allocations. |
| grounded_recommendation One option is recommended with a rationale tied to the brief's or brand's stated goals, without invented performance predictions or unsupported ROI claims. | ✗ | The recommendation for Option B asserts LinkedIn 'has a higher cost per trial signup and does not aid trial activation' as an unsupported comparative performance claim with no underlying data, which is an invented performance/ROI-style claim. |
Skill guidance from emails, copywriting — folders in the marketingskills pack.
| Requirement | Verdict | Evidence |
|---|---|---|
| four_emails_labeled The corrected sequence has exactly four emails, each with all five labeled fields: Timing, Subject, Preheader, Body, and CTA. | ✓ | All four emails (Day 0, 2, 4, 7) each contain Timing, Subject, Preheader, Body, and CTA fields. |
| sequence_only The deliverable is the corrected sequence itself; accurate draft content (clamp use cases, included accessories, three brightness and two color-temperature settings, audience framing) is preserved rather than discarded. | ✓ | Accurate draft content like clip use cases, brightness/color-temp settings, USB-C cable/storage sleeve, and audience framing (renters, students, makers) are retained across emails. |
| price_corrected The $59 price is gone; if a price appears anywhere it is exactly $68. | ✓ | Email 3 now reads 'At $68, it pays for itself' and the $59 figure is gone. |
| canonical_name Every product reference uses the canonical name Arc Clip Lamp; the shortened 'Arc Lamp' does not appear. | ✓ | Every mention uses 'Arc Clip Lamp'; the shortened 'Arc Lamp' does not appear anywhere in the response. |
| battery_corrected The 12-hour battery claim is gone; any battery claim says up to eight hours on the lowest brightness. | ✓ | Email 2 states 'runs up to eight hours on the lowest brightness,' replacing the 12-hour claim. |
| clamp_corrected The 3-inch clamp claim is gone; any clamp thickness stated is 2.25 inches. | ✓ | Email 1 and Email 4 both state '2.25 inches thick' instead of the original 3-inch claim. |
| testimonial_removed The invented Maya S. testimonial is absent and no replacement customer quote or review proof is invented. | ✓ | The Maya S. quote is absent from Email 2 and no replacement testimonial appears anywhere. |
| guarantee_removed The 'guaranteed to double your focus' claim is absent and no other guaranteed-outcome claim is added. | ✓ | The 'guaranteed to double your focus' line from the original Email 3 is gone and no substitute guarantee appears. |
| urgency_removed The 'Only 50 left' scarcity and the 24-hour deadline are absent and no replacement urgency, scarcity, or countdown tactic is added. | ✓ | Email 3's subject/body no longer mention 'Only 50 left' or a 24-hour deadline, and no new urgency tactic is introduced. |
| shipping_claim_removed The free 2-day shipping offer is absent and no other unsupplied offer or promotion is added. | ✓ | The 'free 2-day shipping' line is absent from the corrected Email 3 with no replacement offer added. |
| cta_exact Every CTA field is exactly 'Explore the Arc Clip Lamp'; 'Buy Now' does not appear as a CTA. | ✓ | Every email's CTA field reads exactly 'Explore the Arc Clip Lamp' with no 'Buy Now' remaining. |
| merge_tag_fixed The malformed '{{first_name}' merge tag is either repaired to a well-formed tag or removed; no malformed personalization token remains. | ✓ | Email 1 body now reads 'Hi {{first_name}},' with a properly closed double-brace merge tag. |
| one_job_per_email Each corrected email keeps one job and one primary call to action, consistent with the lifecycle-email guidance. | ✓ | Each email retains a single focused message and exactly one CTA button ('Explore the Arc Clip Lamp'). |
| no_new_claims No new product fact, capability, audience, proof point, offer, or performance claim appears beyond the supplied sources. | ✗ | Email 3 retains the unsupported financial claim 'it pays for itself in reclaimed desk space alone,' which is not grounded in facts.md or product-marketing.md. |
Skill guidance from ad-creative, analytics — folders in the marketingskills pack.
| Requirement | Verdict | Evidence |
|---|---|---|
| full_factorial Exactly four cells A-D are present and cover Renters x Surface-space saving, Renters x Lighting control, Makers x Surface-space saving, and Makers x Lighting control exactly once. | ✓ | Cells A-D each cover a distinct Audience x Appeal pairing (Renters/Surface, Renters/Lighting, Makers/Surface, Makers/Lighting) exactly once. |
| cell_fields Every cell labels Audience, Appeal, Headline, Primary text, Visual direction, CTA, and Creative rationale. | ✓ | Each cell lists Audience, Appeal, Headline, Primary text, Visual direction, CTA, and Creative rationale. |
| budget_and_duration The plan reserves a maximum of $4,800 as a $1,200 cap per cell over 14 days; it does not promise that a stopped cell will spend its full cap. | ✓ | Execution rules state a $4,800 max reserve, $1,200 hard cap per cell over 14 days, and specify unused budget from a stopped cell remains unspent, not promising full spend. |
| execution_rules_section A labeled Execution rules section is present. | ✓ | The response includes a clearly labeled '## Execution rules' section. |
| hypotheses_section A separate labeled Preregistered hypotheses section is present. | ✓ | A separate '## Preregistered hypotheses' section is present after the execution rules. |
| approved_cta All four CTA fields are exactly 'Explore the Arc Clip Lamp'. | ✓ | All four cells list CTA as 'Explore the Arc Clip Lamp' verbatim. |
| grounded_copy All copy and visuals use only supplied Arc Clip Lamp facts and avoid invented customers, results, proof, scarcity, offers, or capabilities. | ✓ | Copy uses only supplied facts (price $68, package contents, three brightness/two color-temperature settings, eight-hour battery, 2.25-inch clamp) with no invented reviews, awards, scarcity, or performance guarantees. |
| cells_isolate_factors Product identity, offer, copy/visual format, placement, run dates, and measurement rules stay constant; cells vary only the assigned audience language and the assigned appeal's supported message and visual. | ✓ | All cells keep the $68 offer, package contents, 1:1 static Meta feed format, and identical execution rules constant while varying only audience framing and appeal-specific message/visual. |
| factorial_hypotheses Exactly three directional landing-page-view-rate hypotheses are preregistered: one audience main effect, one appeal main effect, and one audience-by-appeal interaction; each names its comparison without changing an extra factor. | ✓ | Exactly three hypotheses are given—an audience main effect (A+B vs C+D), an appeal main effect (A+C vs B+D), and an audience-by-appeal interaction—each with a clear LPVR comparison. |
| kpi_and_guardrail Landing-page-view rate is correctly defined as landing-page views / link clicks and cost per AddToCart is the guardrail. | ✓ | The plan defines LPVR as landing-page views divided by link clicks and names cost per AddToCart as the guardrail metric. |
| stop_rule_exact The stop rule is exactly $300 spend with zero AddToCart for an individual cell; unused budget is not reallocated and any triggered stop makes the final result inconclusive. | ✓ | Execution rules specify stopping a cell at $300 spend with zero AddToCart, no budget reallocation, and that any stop makes the result inconclusive. |
| win_rule_exact A winner requires at least 15% relative landing-page-view-rate lift versus the click-weighted pooled rate of the other three cells (their total landing-page views / total link clicks) and at least 200 link clicks in every cell. | ✓ | The win rule requires ≥15% relative lift vs. the click-weighted pooled rate of the other three cells and at least 200 link clicks per cell, matching the required formula. |
| inconclusive_rule The plan explicitly declares the result inconclusive when the full win rule is not met and does not pick a winner from directional data alone. | ✓ | The Inconclusive Rule explicitly states that failing any part of the win criteria (lift, sample size, or stop) results in a formally declared inconclusive outcome. |
Skill guidance from copywriting — folders in the marketingskills pack.
| Requirement | Verdict | Evidence |
|---|---|---|
| five_options Exactly five headline options are present and labeled Option 1 through Option 5. | ✓ | Programmatic check: option_labels=[1, 2, 3, 4, 5]. |
| rationales Each option has a one-line rationale explaining the appeal it tests. | ✓ | Each of the five options includes an italicized 'Rationale:' line explaining the appeal it tests. |
| grounded_claims No invented customers, rankings, certifications, guarantees, or outcomes; none of the forbidden claims appear (award-winning, best-selling, clinically proven, limited time, carbon neutral, guaranteed productivity); any price stated is $68. | ✓ | No fabricated reviews, awards, certifications, guarantees, scarcity, or environmental claims appear, and no price is misstated. |
| specific_language At least two headlines lead with a concrete supplied product fact (e.g. clamp-anywhere design, up to eight hours battery on lowest brightness, three brightness / two color-temperature settings, $68) rather than generic superlatives. | ✓ | Option 1 leads with the clamp-thickness fact and Option 2 leads with the brightness/color-temperature settings fact, both directly from facts.md. |
| positioning_fit The set communicates the approved positioning - adaptable light without surrendering permanent surface space, for renters, students, and makers in compact spaces - not generic lamp copy. | ✓ | Multiple headlines (e.g., Option 4 'Task light where you need it, without surrendering your desk') directly echo the approved 'adaptable light without surrendering surface space' positioning. |
| distinct_appeals The five options test meaningfully different appeals or motivations rather than paraphrasing one idea. | ✓ | The rationales explicitly label differing appeals: spec-first, adjustability-first, daily routine, core positioning, and problem-aware frustration. |
Skill guidance from cro, site-architecture — folders in the marketingskills pack.
| Requirement | Verdict | Evidence |
|---|---|---|
| complete_html The response is one complete, parseable HTML document from an HTML5 doctype through the closing html tag, with no Markdown fence. | ✓ | Programmatic check: html_complete=True; html_parse_error_count=0. |
| no_external_assets The HTML uses inline CSS and has no CSS asset references, active content, scripts, images, stylesheets, fonts, media, iframes, embeds, objects, SVG, or canvas assets. | ✓ | Programmatic check: inline_css_count=1; resource_request_count=0; forbidden_asset_tag_count=0; active_content_count=0. |
| duration The repaired page consistently states a 45-minute consultation and removes the flawed 60-minute duration. | ✓ | The page consistently states '45-minute' consultation in multiple sections and never mentions 60 minutes. |
| price The repaired page consistently states the $160 price and removes the flawed $129 price. | ✓ | The price section and objections consistently state '$160' with no mention of $129. |
| offering_type The offer is accurately described as one virtual educational routine-planning consultation, not a medical consultation. | ✓ | The page describes 'an educational coaching service... It is an educational service, not a medical clinic' and calls it a 'virtual educational routine-planning consultation'. |
| sections_and_order All requested sections appear in the exact cautious-visitor decision order. | ✓ | Sections appear in exact order: what this is/who for, how it works, what's included, price, limitations/who not for, common objections, next step. |
| cta_target The exact CTA 'Book a consultation' appears once or twice and every instance is an anchor whose href is exactly '/consultations/harborline'. | ✓ | Programmatic check: book_cta_count=2; book_cta_count=2; book_cta_all_instances_are_anchors=True; book_cta_all_targeted=True. |
| no_competing_action 'Start sleeping better', 'Call now', and every competing primary action from the flawed draft are absent. | ✓ | No 'Start sleeping better' or 'Call now' text appears; only 'Book a consultation' CTAs are present. |
| educational_scope The page clearly says Harborline is educational coaching, not a medical clinic, and does not diagnose or treat sleep disorders; cure, diagnosis, and treatment claims are absent. | ✓ | States 'It is an educational service, not a medical clinic, and does not provide clinical care or diagnose or treat medical conditions' with no cure/diagnosis claims. |
| prescription_claim_removed The invented prescription-review service is absent and no replacement clinical service is added. | ✓ | Limitations list explicitly states 'No prescriptions: No medical prescriptions, prescription reviews, or medication adjustments are provided.' |
| credential_removed The invented certified-clinician credential is absent and no replacement credential is invented. | ✓ | The page uses generic 'educational coach' language and never claims 'certified clinicians' or any invented credential. |
| testimonial_removed The invented Taylor M. testimonial is absent and no replacement patient or customer proof is invented. | ✓ | No blockquote or testimonial like 'Taylor M.' appears anywhere in the document. |
| guarantee_removed All cure and guaranteed-improvement claims are absent, and the page explicitly says outcomes are not guaranteed. | ✓ | States 'No guaranteed outcomes: Outcomes are not guaranteed, and results will vary based on individual implementation.' |
| insurance_removed The insurance-coverage claim is absent, and the page explicitly says insurance is not represented. | ✓ | States 'No insurance billing: Insurance billing is not provided, and insurance coverage is not represented.' |
| urgency_removed The three-appointments scarcity claim is absent and no replacement urgency, scarcity, or pressure tactic is added. | ✓ | No scarcity phrase like 'only three appointments' or any replacement urgency tactic appears. |
| emergency_direction The page says it is not emergency care and directs urgent symptoms or safety concerns to local emergency services or an appropriate licensed clinician. | ✓ | A notice box states exactly 'People with urgent symptoms or safety concerns should contact local emergency services or an appropriate licensed clinician.' |
| heading_hierarchy The document has exactly one h1 and ordered heading levels with no skipped levels. | ✓ | Programmatic check: h1_count=1; heading_order_valid=True. |
| semantic_landmarks Semantic landmarks and descriptive section headings organize the page; it is not a div-only document. | ✓ | The document uses header, main, footer, section, h2/h3, and ul/li elements rather than a div-only structure. |
| conversion_clarity The corrected page makes the educational offer, fit, included value, limitations, price, and next step concrete without replacing the planted defects with vague wellness copy. | ✓ | The page provides concrete, specific detail on fit, process, inclusions, price, limitations, objections, and next step without resorting to vague wellness language. |
Skill guidance from cro, site-architecture — folders in the marketingskills pack.
| Requirement | Verdict | Evidence |
|---|---|---|
| complete_html The response is one complete, parseable HTML document from an HTML5 doctype through the closing html tag, with no Markdown fence. | ✓ | Programmatic check: html_complete=True; html_parse_error_count=0. |
| no_external_assets The HTML uses inline CSS and has no CSS asset references, active content, scripts, images, stylesheets, fonts, media, iframes, embeds, objects, SVG, or canvas assets. | ✗ | Programmatic check: inline_css_count=2; resource_request_count=0; forbidden_asset_tag_count=11; active_content_count=0. |
| sections_complete All required sections exist: relevance/who it is for, how it works, what is included, price ($160 for 45 minutes), limitations/not-for, objections, next step. | ✓ | The page includes hero/audience section, 'How the consultation works', 'What is included', 'Consultation fee' with $160/45-minute language, a detailed 'Service limitations & who this is NOT for' section, a 'Common questions & objections' FAQ, and a 'The next step' CTA section. |
| cta_count The exact primary CTA 'Book a consultation' appears once or twice. | ✓ | Programmatic check: book_cta_count=2; book_cta_count=2. |
| no_competing_action No competing primary action is added. | ✓ | Only two instances of the 'Book a consultation' anchor button appear (hero and next-step section), with no other primary action buttons present. |
| decision_order Sections appear in the cautious-visitor order (relevance, mechanism, included value, risk/limitations, objections, next step) with one job and a descriptive heading per section. | ✓ | Sections run hero(relevance/who)->how it works->what's included->price->limitations/not-for->objections->next step, matching the required cautious-visitor order exactly. |
| no_dark_patterns No fake urgency, countdown, invented testimonials or reviews, guaranteed outcomes, or hidden terms; disclosures (not medical care, no diagnosis/treatment, not emergency care, outcomes not guaranteed, no insurance representation) are preserved and readable. | ✓ | No countdowns, testimonials, or guarantees are present; the limitations section and footer explicitly restate 'not a medical clinic', 'no diagnosis or treatment', 'not emergency care', 'outcomes are not guaranteed', and 'insurance is not represented.' |
| heading_hierarchy The document has exactly one h1 and ordered heading levels with no skipped levels. | ✓ | Programmatic check: h1_count=1; heading_order_valid=True. |
| semantic_landmarks Semantic HTML landmarks (header/main/section/footer or equivalent) organize the page; it is not a div-only document. | ✓ | The document uses <header>, <main>, multiple <section> elements with aria-labelledby, <article> cards, and a <footer>, avoiding a div-only structure. |
| conversion_clarity The primary action is visually and structurally unmistakable, and copy answers visitor questions concretely instead of vague wellness language. | ✓ | The 'Book a consultation' button is styled prominently (bold color, shadow, hover/focus states) and appears at top and bottom, while copy gives concrete specifics like the $160 flat fee, exact inclusions, and explicit disorder examples rather than vague wellness language. |
Skill guidance from analytics — folders in the marketingskills pack.
The judge marked this response incomplete — it was cut off at the output-token ceiling — so the task fails regardless of the per-criterion results below.
| Requirement | Verdict | Evidence |
|---|---|---|
| design_shortfall_stated The response states that the committed deliverables require exactly 64 designer hours (16 + 18 + 6 + 24 + 0) against 48 available (12 per week x 4 weeks), a 16-hour shortfall, with the arithmetic visible. | ✓ | The response states 'A 16-hour designer capacity deficit... require 64 design hours, but the dedicated designer has only 48 available hours' with a table showing 16+18+6+24=64. |
| copy_capacity_confirmed The response states that copywriter hours fit: 92 required (48 + 12 + 12 + 10 + 10) against 96 available (24 x 4). It does not claim the copywriter is over capacity. | ✗ | The response is cut off before showing the video-script row or any copywriter-hour total, so it never states 92 hours required against 96 available. |
| freeze_conflict_stated The response flags that the required October 28 landing-page go-live falls inside the October 26-30 content freeze, and no proposed plan publishes or launches anything during the freeze. | ✓ | It explicitly states 'October 28 falls squarely inside this freeze window' referencing the Oct 26-30 freeze. |
| infeasibility_declared The response plainly declares that the brief as written cannot be executed: no schedule is presented that claims to deliver every commitment within the stated capacity and calendar. | ✓ | The response's heading states 'The Brief Cannot Be Honored as Written' and explicitly says no schedule pretending to satisfy the conflicts is presented. |
| no_invented_capacity No unapproved freelance, contractor, overtime, or extra-staff hours are counted as available capacity. Proposing approval of the freelance budget as an option is fine only when it is explicitly labeled as requiring a decision leadership has not yet made. | ✓ | No freelance, contractor, or extra-staff hours are counted as available capacity anywhere in the visible text, satisfying the forbidding criterion by absence. |
| explicit_constraint_changes Every resolution option names exactly which stated commitment it changes (which deliverables are cut or reduced, that the go-live date moves, or that new capacity must be approved) rather than silently altering or omitting one. | ✗ | The response is truncated before presenting any of the promised 'three concrete... resolution options', so no constraint changes are named. |
| auditable_arithmetic Hour totals per person are numeric, follow the brief's per-unit production standards, and sum correctly in the original reconciliation and in every option. | ✗ | Per-person (copywriter) totals are never computed and no option-level arithmetic is shown because the response cuts off mid-table before any options or final sums. |
| multiple_options At least two materially different resolution options are proposed, each with a complete revised deliverable list whose copywriter hours fit 96 and designer hours fit 48 (or fit an explicitly approved capacity increase), and a go-live date outside the freeze. | ✗ | The response promises 'three concrete, fully scheduled resolution options' but the text ends before any option is actually presented. |
| grounded_recommendation One option is recommended with a rationale tied to the launch's or brand's stated goals, without invented performance predictions or unsupported claims about what the content will achieve. | ✗ | No recommendation is given anywhere in the truncated response. |
Skill guidance from copywriting, analytics — folders in the marketingskills pack.
| Requirement | Verdict | Evidence |
|---|---|---|
| all_sections All seven labeled sections are present: ICP/positioning, channel plan, content calendar, activation metrics, budget allocation, measurement plan, risks. | ✓ | The response contains all seven labeled sections (1) ICP and Positioning through (7) Top Three Risks and Mitigations. |
| three_channels Exactly three channels are chosen, each with a rationale, and the budget allocation across them sums to the $15,000 total. | ✓ | Exactly three channels (Google Paid Search, LinkedIn Paid Social, Lifecycle & Onboarding Assets) are chosen with rationale, and the budget table sums $6,500+$5,500+$3,000=$15,000. |
| activation_anchored Funnel metrics are explicitly tied to the documented activation event (inviting two teammates and completing one shared workflow), not a generic signup metric. | ✓ | The funnel explicitly marks 'Shared Workflow Completed' as the '★ CORE ACTIVATION EVENT (trial_activated)' tied to inviting two teammates and completing one shared workflow. |
| no_invented_proof No invented customers, logos, revenue, market statistics, benchmark conversion rates stated as fact, or testimonials; assumptions are labeled as assumptions. | ✗ | The funnel table states specific numeric 'Target Conversion Benchmark' figures (e.g., '4.0% – 6.0%', '35.0% of signups', '20.0% – 25.0%') as fact without labeling them as assumptions or citing any source. |
| measurement_defined Each tracked metric names its source/definition (where the number comes from), and the measurement plan is weekly as requested. | ✓ | Each metric in the Measurement Framework Table names a source (e.g., 'GA4', 'Mixpanel', 'Google Ads Manager / LinkedIn Campaign Manager') and the section is explicitly weekly cadence. |
| focused_strategy The plan concentrates on the three chosen channels with reasoning that references the ICP, rather than spreading across every possible tactic. | ✓ | The plan concentrates all budget and content on the three chosen channels, with rationale tied to ICP roles like 'Operations Leads' and 'Project/Program Managers'. |
| sequenced_90_days The 90 days are phased (e.g. by month or milestone) with dependencies acknowledged, and the positioning ('faster, visible handoffs without adding another weekly status meeting') carries through the messaging. | ✓ | The plan phases work into a 4-week content calendar and monthly budget blocks (~$5,000/month), while the headline positioning 'faster, visible handoffs without adding another weekly status meeting' recurs throughout ad copy and emails. |
| mature_activation_denominator Activation rate is defined using only trial starts whose full 14-day activation window has closed; newer trials are explicitly pending rather than counted as non-activated. | ✓ | The response states 'A trial signup inside its 14-day trial window is pending, not failed' and defines the Cohort Activation Rate using only 'Total workspace signups whose 14-day trial window has expired'. |
Skill guidance from ad-creative, copywriting — folders in the marketingskills pack.
| Requirement | Verdict | Evidence |
|---|---|---|
| all_fields_present All eleven labeled fields (RSA-H1, RSA-H2, RSA-H3, RSA-D1, RSA-D2, META-PRIMARY, META-HEADLINE, META-CTA, LI-INTRO, LI-CTA, SMS-BODY) are present; the first line bearing each label is the graded field. | ✓ | Programmatic check: missing_or_duplicate_label_count=0. |
| within_char_limits Every limited field value is at or under its hard character maximum (RSA-H 30, RSA-D 90, META-PRIMARY 125, META-HEADLINE 40, LI-INTRO 150, SMS-BODY 160). | ✓ | Programmatic check: over_limit_field_count=0. |
| approved_cta_exact META-CTA and LI-CTA are each exactly 'Review consultation availability' (a single terminal period is accepted, matching the brand facts' own punctuation). | ✓ | Programmatic check: approved_cta_count=2. |
| disclaimer_everywhere The exact sentence 'Educational only; not medical care.' appears inside META-PRIMARY, LI-INTRO, and SMS-BODY. | ✓ | Programmatic check: disclaimer_field_count=3. |
| sms_optout_present SMS-BODY contains 'Reply STOP to opt out'. | ✓ | Programmatic check: sms_optout_present=True. |
| accurate_price Every stated dollar amount is exactly $160, and the price appears at least once. | ✓ | Programmatic check: non_160_dollar_count=0; price_160_count=5. |
| grounded_claims Every claim uses only the supplied facts (45-minute virtual consultation, $160 paid directly, intake review, routine-planning conversation, written educational summary, adults organizing non-clinical bedtime habits). No outcomes, testimonials, credentials, guarantees, urgency, scarcity, discounts, or clinical framing (diagnosis or treatment) are invented. | ✓ | All copy references only the supplied facts (45-minute virtual consultation, $160, intake review, routine-planning conversation, written summary, non-clinical bedtime habits) with no outcomes, testimonials, guarantees, urgency, or clinical framing invented. |
| single_ask Each placement carries one job: getting the reader to review consultation availability. No placement adds a competing ask such as a newsletter signup, follow, download, or second offer. | ✓ | Every placement's CTA is 'Review consultation availability' with no competing asks like signup, download, or secondary offer. |
| distinct_rsa_assets The three RSA headlines take meaningfully different angles rather than near-duplicate phrasings, and the two RSA descriptions differ in substance. | ✓ | RSA-H1 (brand name), RSA-H2 (price/format), and RSA-H3 (benefit: consistent bedtime routines) take distinct angles, and RSA-D1 (what's included) differs substantively from RSA-D2 (price/format/audience summary). |
| honest_specificity Copy leans on the concrete supplied specifics (45 minutes, virtual, written summary, non-clinical routine help) instead of generic superlatives or vague promises of better sleep. | ✓ | Copy consistently uses concrete specifics like '45-minute virtual consultation', '$160', 'written educational summary', and 'non-clinical bedtime habits' rather than vague superlatives. |
Skill guidance from analytics — folders in the marketingskills pack.
| Requirement | Verdict | Evidence |
|---|---|---|
| correct_total The correct October charge is stated as exactly $10,500 (newsletters 6 x $410 = $2,460; sponsored articles 6 x $1,200 = $7,200 minus the 5% volume discount = $6,840; podcast spots 4 x $300 = $1,200), with the arithmetic visible. | ✓ | Response states 'Correct / Audited Amount $10,500.00' with visible arithmetic (6×$410=$2,460; $7,200−$360=$6,840; 4×$300=$1,200). |
| overbilling_stated The response states the invoice overbills by exactly $1,050 (invoiced $11,550 versus correct $10,500). | ✓ | The table lists 'Total Amount to Dispute $1,050.00' derived from $11,550 invoiced vs $10,500 correct. |
| newsletter_errors_identified Both newsletter errors are identified: the invoice uses $450 instead of the negotiated $410, and bills 7 placements when the delivery log shows 6 ran and one was canceled. The combined newsletter overcharge is quantified as $690 (or a coherent decomposition that sums to $690). | ✓ | Response identifies both the $450 vs negotiated $410 rate error and the 7-vs-6 billable quantity error, decomposing the $690 overcharge into $450 (canceled unit) + $240 (rate variance) = $690. |
| discount_missed_identified The missed 5% sponsored-article volume discount is identified as $360, triggered because the $7,200 of sponsored-article charges exceeds the $6,000 monthly threshold. | ✓ | Response states the 5% discount was omitted, quantifies it as $360.00, and notes $7,200 pre-discount charges exceed the $6,000 threshold per Clause 3. |
| sow_precedence The negotiated $410 newsletter rate from the agreement is used over the rate card, and the 5% discount is applied only to sponsored-article line items, never to newsletters or podcast spots or the whole invoice. | ✓ | The audit applies the negotiated $410 newsletter rate (Clause 2) over the rate card and applies the 5% discount only to the sponsored-article line ($1,200×0.95=$1,140), not to newsletters or podcasts. |
| no_stale_rates Neither the pre-October rates ($400/$1,100/$275) nor the DRAFT 2027 rates ($500/$1,300/$340) are used in any calculation. | ✓ | All calculations use the Oct 1 current rates ($410 negotiated, $1,200, $300); no pre-October ($400/$1,100/$275) or DRAFT 2027 rates appear anywhere. |
| delivery_grounded Billable quantities come from the delivery log under the make-good clause: the canceled October 22 newsletter is not payable, and no delivered placement is dropped. | ✓ | The reconciliation table marks the Oct 22 canceled newsletter as not billable under Clause 4 while confirming all other 16 delivered placements as billable. |
| no_false_disputes The podcast line (4 x $300 = $1,200) is confirmed correct and is not disputed. | ✓ | Line 3 podcast spots is explicitly marked 'Clean / Confirmed' with '$0.00' variance and 'Do not dispute' instruction. |
| auditable_math Per-line arithmetic (quantities, rates, the discount base and threshold comparison, and every difference) is shown completely enough for a reviewer to reproduce the audit. | ✓ | The response shows per-line quantities, rates, discount base/threshold comparison, and independent component checks (e.g., $450+$240=$690) reproducible by a reviewer. |
| actionable_dispute The response ends with a concrete dispute: the itemized discrepancies summing to $1,050, a request for a corrected invoice or credit, and no invented vendor intent, fraud accusation, or threat. | ✓ | The response ends with a concrete $1,050 dispute, a draft dispute notice requesting a corrected invoice or credit memo, and no fabricated vendor intent, fraud claim, or threat. |
Skill guidance from analytics — folders in the marketingskills pack.
| Requirement | Verdict | Evidence |
|---|---|---|
| architecture The guide recommends browser Pixel plus server-side Conversions API with deduplication and explains why both are needed under browser privacy limits. | ✓ | Section 1 explicitly recommends the hybrid Pixel + CAPI model with deduplication and explains browser vulnerabilities (ITP, ad blockers) vs server strengths, arguing neither is sufficient alone. |
| events_mapped All four standard events (ViewContent, AddToCart, InitiateCheckout, Purchase) are covered with key parameters, including value, currency, and content_ids on Purchase. | ✓ | Section 2 covers ViewContent, AddToCart, InitiateCheckout, and Purchase each with value, currency, content_ids, content_type, and Purchase additionally includes order_id and contents. |
| code_sketch A concrete server-side Purchase event code sketch or request example is included showing event_name, event_time, event_id, action_source, and a user_data block. | ✓ | The Node.js sketch includes event_name: 'Purchase', event_time, event_id, action_source: 'website', and a full user_data block with hashed fields. |
| validation End-to-end validation is described using Meta's Events Manager test events tool and an explicit check that duplicate browser/server events are being deduplicated. | ✓ | Section 6 describes using Meta Events Manager's Test Events tool with a test_event_code and explicitly instructs confirming a 'Deduplicated' badge appears for the matched browser/server Purchase pair. |
| sources_defined For each event it is clear where it fires (browser vs server) and what the source of truth is (e.g. Purchase from the Shopify order webhook/server, not only the thank-you page). | ✓ | The architecture diagram separates 'Client-side JavaScript' from 'Server / Webhook (Shopify Backend)' capturing 'Validated orders', and Purchase's detail notes it fires 'on the order confirmation page or via backend order creation', clarifying browser vs server source of truth. |
| limits_stated Limitations are stated honestly: no promised attribution-recovery percentages, and consent/regional rules are noted as constraints on what may be sent. | ✓ | No specific attribution-recovery percentage is promised anywhere, and Section 7 states CAPI 'must never be used to bypass a user's opt-out or lack of consent' under GDPR/CCPA. |
| dedup_correct Deduplication is technically correct: the same event_id and event_name sent on both the Pixel and CAPI event so Meta drops the duplicate; not a vague mention. | ✓ | Section 3 states Meta matches events using the same event_name and 'the exact same event_id' sent from both Pixel and CAPI, not a vague mention. |
| event_id_retry_stability The event_id is generated or derived once per commerce event, shared by browser and server, and reused unchanged on webhook retries or order updates; a new random suffix per send or retry fails. | ✓ | The code derives event_id deterministically as `shopify_order_${order.id}` rather than a random string per send, which remains stable across retries since it is tied to the order ID, and the doc notes the Purchase key should be the deterministic 'Shopify Order ID or Checkout Token'. |
| event_time_stable The server Purchase event uses the purchase/order occurrence timestamp in Unix seconds and preserves it across retries; using the current send or retry time fails. | ✓ | The code sets `event_time: Math.floor(new Date(order.createdAt).getTime() / 1000)`, using the order's creation timestamp in Unix seconds rather than the current send time. |
| pii_hashing Customer data handling is correct: personally identifiable fields (e.g. email) are normalized (trimmed, lowercased) and SHA-256 hashed before sending in user_data. | ✓ | Section 4's table and the hashParameter() function both specify trimming, lowercasing, and SHA-256 hashing of email, phone, name, and address fields before sending in user_data. |
| actionable_path The recommended Shopify path is decisive and executable for a small DTC store (a clear recommendation with concrete setup steps, not an undifferentiated list of options). | ✓ | Section 5 gives a decisive recommendation ('Implement the native Meta sales channel app on the \"Maximum\" data-sharing tier') with concrete rationale tailored to a small DTC store, rather than an undifferentiated list. |
Skill guidance from copywriting, ad-creative — folders in the marketingskills pack.
| Requirement | Verdict | Evidence |
|---|---|---|
| two_deliverables There are exactly two labeled deliverables: a Conflict register table and one Corrected concept. | ✓ | The response has exactly two labeled sections: '## 1. Conflict Register' and '## 2. Corrected Concept'. |
| twelve_claim_rows The Conflict register has exactly 12 data rows, one per atomic agency claim; product identity and price are separate rows. | ✓ | The Conflict Register table contains exactly 12 numbered rows, with product name (#1) and price (#2) treated as separate claims. |
| corrected_fields The corrected concept includes all seven labeled fields: Primary text, Headline, Description, CTA, Visual direction, Audience insight, and Test hypothesis. | ✓ | The Corrected Concept lists all seven fields: Primary text, Headline, Description, CTA, Visual direction, Audience insight, and Test hypothesis. |
| product_identity The register rejects 'Arc Lamp', gives the canonical name Arc Clip Lamp, and the corrected concept uses no altered product name. | ✓ | Row 1 rejects 'Arc Lamp' and corrects to 'Arc Clip Lamp', and the concept uses 'Arc Clip Lamp' consistently with no shortened name. |
| price The register rejects $59 and gives the supported price of $68; the corrected concept uses no other price if price is mentioned. | ✓ | Row 2 rejects $59 and gives $68; the Primary text and Description both state '$68'. |
| battery The register rejects '12 hours at any brightness' and gives the supported limit: up to eight hours on the lowest brightness. | ✓ | Row 3 rejects '12 hours at any brightness' and gives 'up to eight hours on the lowest brightness', matching the Primary text's battery claim. |
| clamp The register rejects the 3-inch claim and gives the supported maximum of 2.25 inches. | ✓ | Row 4 rejects the 3-inch claim and gives 'up to 2.25 inches thick', matching the Primary text and Visual direction. |
| included_items The register rejects the adhesive wall mount; the corrected concept does not add it or any other unsupplied accessory. | ✓ | Row 5 rejects the adhesive wall mount and lists the correct accessories; the concept's Primary text lists only lamp, clamp, USB-C cable, and storage sleeve. |
| carbon_certification The register rejects the carbon-neutral certification as unsupported; it is absent from the corrected concept. | ✓ | Row 6 rejects the carbon-neutral claim outright and it does not appear anywhere in the corrected concept. |
| bestseller_proof The register rejects the #1 best-selling claim as unsupported; it is absent from the corrected concept. | ✓ | Row 7 rejects the '#1 best-selling' claim and it is absent from the corrected concept. |
| productivity_guarantee The register rejects the guaranteed 37% productivity claim as unsupported; it is absent from the corrected concept. | ✓ | Row 8 rejects the 37% productivity guarantee and it is absent from the corrected concept. |
| customer_quote The register rejects the invented Jamie R. quote as unsupported; it is absent from the corrected concept. | ✓ | Row 9 rejects the fabricated Jamie R. quote and it is absent from the corrected concept. |
| performance_claim The register rejects the invented 42% conversion lift because no campaign evidence is supplied; it is absent from the corrected concept. | ✓ | Row 10 rejects the invented 42% conversion lift, citing that no campaign has launched, and it is absent from the corrected concept. |
| scarcity_claim The register rejects 'only 200 remain' because no inventory evidence is supplied; it is absent from the corrected concept. | ✓ | Row 11 rejects the 'Only 200 remain' scarcity claim and it does not appear in the corrected concept. |
| approved_cta The register rejects 'Shop Now' and the corrected CTA is exactly 'Explore the Arc Clip Lamp'. | ✓ | Row 12 rejects 'Shop Now' and the CTA field of the corrected concept reads exactly 'Explore the Arc Clip Lamp'. |
| supported_direction_preserved The corrected concept preserves the useful supported visual idea of moving the lamp between a narrow workspace/shelf and bedside, tied to adaptable light in compact spaces. | ✓ | The Visual direction preserves the agency's shelf-to-bedside movement, showing the lamp clamped to a narrow desk/shelf then moving to a bedside table, tied to adaptable light in compact spaces. |
| operational_hypothesis The Test hypothesis names the changed message or visual variable, its comparator, an observable KPI, and the expected direction. | ✓ | The Test hypothesis names the changed variable (surface-saving utility angle), the comparator (an aesthetic/decor-focused sibling concept), the metric (CTR), the audience, and the expected direction (higher CTR). |
| no_new_claims The audit and corrected concept introduce no new product fact, customer, proof point, offer, urgency, or performance claim beyond the authoritative sources. | ✓ | All facts used (name, price, battery, clamp size, accessories, CTA) trace directly to facts.md and product-marketing.md with no fabricated stats, testimonials, or urgency introduced. |
Skill guidance from analytics — folders in the marketingskills pack.
The judge marked this response incomplete — it was cut off at the output-token ceiling — so the task fails regardless of the per-criterion results below.
| Requirement | Verdict | Evidence |
|---|---|---|
| covers_all_sources The audit addresses all supplied artifacts: the campaign links, the GTM/GA4 tag configuration, and the consent setup (a clean-bill mention counts as addressing an artifact). | ✓ | The executive summary touches on campaign-link UTM issues, GTM/GA4 tag issues (purchase, currency), and consent-mode configuration before the report is cut off. |
| findings_located Each reported defect names its specific location — the exact link/channel, tag, trigger, or setting — precisely enough that an engineer could act without re-deriving the audit. | ✓ | Finding 1 pinpoints the 'GTM Consent Initialization Tag / Consent Mode v2 settings' and quotes the exact granted/denied values, giving an actionable location. |
| finds_uppercase_medium Flags the uppercase utm_medium=CPC values on the two google-search links as violating the lowercase standard and fragmenting paid-search reporting. | ✗ | Only a generic mention of 'uppercase UTM medium values splitting Paid Search' appears in the summary with no reference to the specific google-search links. |
| finds_uppercase_medium_pmax Also flags the google-pmax link's uppercase utm_medium=CPC (the same defect on the Performance Max campaign, not only on the search campaigns). | ✗ | No mention of the pmax link's uppercase utm_medium anywhere in the visible text. |
| finds_missing_source Flags the klaviyo-welcome link as missing utm_source entirely. | ✗ | The klaviyo-welcome missing utm_source defect is never mentioned. |
| finds_meta_source_inconsistency Flags that some Meta links use utm_source=facebook while the standard (and other links) use utm_source=meta, splitting Meta reporting. | ✗ | There is no mention of the facebook vs meta utm_source inconsistency anywhere in the response. |
| finds_campaign_spaces Flags the meta-summer link's utm_campaign value ('2026q3 summer sale') for containing spaces / violating the hyphenated campaign format. | ✗ | The meta-summer campaign's spaced utm_campaign value is never referenced. |
| finds_duplicate_purchase_firing Identifies that purchase fires from two live tags (the confirmation-page tag and the place-order button-click tag), double-counting purchases, and connects this to GA4 reporting far more purchases than Shopify. | ✓ | The summary states inflated purchase counts are 'caused by a rogue button-click tag left running from June, compounded by an overly broad trigger regex,' tying it to the GA4 vs Shopify discrepancy. |
| finds_missing_transaction_id Identifies that the GA4 purchase tags omit the transaction_id parameter required by the standards, so duplicate purchase events cannot be deduplicated. | ✗ | The response never mentions the absence of transaction_id in the purchase tags. |
| finds_hardcoded_currency Identifies that purchase currency is hardcoded to USD while the ca. storefront transacts in CAD, and connects this to the inflated Canadian revenue. | ✓ | The summary explicitly cites 'hardcoded "USD" currency parameters on CAD transactions' as the cause of inflated Canadian revenue. |
| finds_consent_default Identifies that analytics_storage defaults to granted for EEA/UK visitors, violating the documented Consent Mode defaults, and connects this to the legal/privacy finding. | ✓ | Finding 1 details that EEA/UK analytics_storage defaults to granted instead of denied, quoting the standard and tying it to the legal/privacy complaint. |
| finds_http_links Flags the two http:// newsletter links as violating the https standard and ties them to newsletter traffic landing in Direct (attribution loss on the redirect). | ✓ | The summary attributes email traffic showing as Direct to 'http:// redirects stripping email UTMs,' matching the newsletter attribution symptom. |
| finds_event_name_casing Flags the 'Add to Cart' event name (spaces, not snake_case) as violating the naming standard and explains the add-to-cart reporting going quiet after the rename. | ✗ | No mention of the 'Add to Cart' event naming/casing issue appears anywhere in the visible text. |
| finds_duplicate_campaign_param Flags the cart-recovery link carrying utm_campaign twice with different values, and ties it to the cart-recovery campaign appearing under two names. | ✗ | The summary only vaguely references 'duplicate/malformed UTM campaign parameters' without naming the cart-recovery link or its two-name reporting symptom. |
| finds_pending_page_regex Identifies that the purchase trigger RegEx ^/order-confirmation also matches /order-confirmation-pending, so purchases fire for unsettled payments, and connects this to the payment-outage / revenue-runs-ahead symptoms. | ✓ | The summary notes an 'overly broad trigger regex that fires on pending/failed payment confirmation states,' linking it to the payment-outage revenue symptom. |
| no_false_positives No conforming item is reported as a defect (for example the google-brand link, the begin_checkout tag, the TikTok/Pinterest/affiliate links, or the ca-storefront links' UTMs). Describing a correct item as clean is fine; flagging it as broken is not. | ✓ | No conforming item (e.g., google-brand link, begin_checkout tag) is misidentified as a defect in the visible content. |
| fix_per_finding Every reported defect comes with a concrete, implementable fix (what to change, not 'review this'). | ✗ | The response is truncated mid-explanation of Finding 1 before any concrete fix is given, and no other findings are fully written up with fixes. |
| impact_prioritized Findings are ordered or explicitly tiered by measurement/compliance impact, with the purchase-integrity and consent issues ranked above cosmetic UTM issues. | ✓ | A severity matrix explicitly ranks 'Critical' (consent/legal and purchase integrity) above 'High' and 'Medium' cosmetic UTM issues. |
Skill guidance from copywriting, emails — folders in the marketingskills pack.
| Requirement | Verdict | Evidence |
|---|---|---|
| required_sections The response contains exactly the three required markdown headings: Landing section, Promo email, and Declined items. | ✓ | Programmatic check: required_section_count=3. |
| supported_offer_used The copy uses the $95 furnace diagnostic offer, and every dollar amount in the copy is exactly $95. | ✓ | Programmatic check: copy_price_95_count=8; copy_non_95_dollar_count=0. |
| safety_guidance_included The copy includes the gas-smell safety guidance consistent with the facts: leave the building, avoid switches or flames, and call the local gas emergency line from a safe location — with no invented additional safety advice. | ✓ | Both the landing section and email include: 'Leave the building immediately... Do not use any electrical switches, appliances, or open flames... Call your local gas emergency line from a safe location' with no extra invented advice. |
| all_unsupported_declined All four unsupportable requests — the 4.9-star/800+ review claim, the Same-Day Service Guarantee, the 'cut your heating bill by up to 30%' line, and the 'Certified Master Technician' credential — are each named under Declined items, and none of them appears in the landing section or email. | ✓ | The four items (reviews, same-day guarantee, 30% savings, Certified Master Technician) are each listed under Declined items (#1-#4) and do not appear anywhere in the Landing section or Promo email copy. |
| no_fabricated_proof The landing section and email contain no star ratings, review counts, percentage claims, same-day promises, or certification language. | ✓ | Programmatic check: copy_star_rating_count=0; copy_review_claim_count=0; copy_percent_token_count=0; copy_same_day_count=0; copy_certification_count=0. |
| no_softened_claims The copy does not smuggle declined claims back in softened form — no 'highly rated', 'trusted by hundreds', 'usually there within hours', 'up to' savings language, invented credentials, or experience proxies standing in for the declined proof. | ✓ | The copy avoids phrases like 'highly rated,' 'up to' savings, or invented credentials; it only says 'licensed professional' which mirrors the facts' required-inspection language rather than smuggling back the declined 'Certified Master Technician' claim. |
| grounded_refusals Each declined item cites the supplied facts or policy as the reason (no reviews, rankings, awards, or certifications are supplied; never promise same-day service, energy savings, a specific diagnosis, or a repair price before inspection) rather than generic legal caution. | ✓ | Each declined item quotes the supplied facts/policy directly, e.g. 'no reviews, rankings, awards, or certifications are supplied' and 'Never promise same-day service, energy savings...'. |
| service_area_accurate The copy claims only the North Valley service area and does not claim metro-wide coverage; the owner's metro-area request is corrected or declined, not echoed. | ✓ | The copy only references North Valley ('Serving Homeowners in North Valley') and the metro-area request is declined in item #5 with a North Valley-only alternative. |
| compliant_alternatives At least three of the four declined items come with a concrete compliant alternative (for example inviting customers to leave reviews, 'call for today's availability' instead of a same-day guarantee, or comfort-focused benefit language without savings percentages). | ✓ | All four (and a fifth) declined items include a concrete compliant alternative, e.g. 'Transparently inform customers that seasonal booking availability varies...' for the same-day guarantee. |
| evidence_unlock The Declined items section tells the owner what evidence or approval would unlock each claim later (for example a review-platform export, a written and honored guarantee policy, or documented certification), so the pushback is actionable rather than a flat no. | ✓ | Each declined item ends with an 'Evidence required to unlock original claim' line, e.g. 'An authenticated export or live link to Brightline Heating's verified Google Business Profile...' for the review claim. |
| Model | Model only model only, over the API · web search | + Marketing Skills model + marketingskills · web search | + Magister full agent · marketingskills + hosted tools |
|---|---|---|---|
| Gemini 3.8 Flash high effort | 78.4 8/20 | 87.2 12/20 | 94.5 14/20 |
Skills vs. no skills changes only the prompt, so that step isolates what the skill pack alone is worth. The Magister column is that same skill pack running inside the full hosted agent, so it also captures what the harness adds on top. Each score links to that configuration's full run.
Every task is a marketing assignment with supplied source material: write the ad concepts, analyze the campaign data, repair the landing page, reconcile the attribution models. Some tasks plant contradictions and defects the model has to catch; some have exact arithmetic a program can check.
Each task carries a checklist of requirements. Local programmatic checks handle the things a program can verify — sums, counts, required phrases, HTML structure. A blinded LLM judge (with no web tools) scores the rest, returning a verdict and evidence for every requirement.
The quality score averages adjusted rubric scores across every task and attempt on a 0–100 scale. Some numeric checks award partial credit; checks already satisfied by a blank answer are excluded. A critical failure or a judgment of incomplete work caps an attempt at 40. A critical failure proved by an exact reference check scores zero, as does missing or errored grading. Truncated answers stay in the task denominator.
Tasks passed is a separate, all-or-nothing measure: one missed requirement fails the task, even when its quality score is high. With one attempt this is Pass@1. With repeated attempts, a task passes only if every attempt passes. Provider errors and truncations fail this measure. A single repetition is a diagnostic snapshot; it does not establish a reliable advantage.
In the skills configuration, each task's prompt carries the relevant verbatim SKILL.md files from marketingskills, the open marketing skill pack by Corey Haines, co-founder of Magister, which is also used by Magister agents. The runner appends the benchmark checkout's pinned skill files to the prompt as plain text under “Skill guidance you should follow”; the no-skills configuration omits those blocks and changes nothing else, so that comparison is a clean ablation of the skill pack itself.
Each skill links to its folder in the marketingskills repo — the exact files the runner inlines. Each task links to every model's answer and verdict for it.
The Magister configuration runs the same tasks through the production Magister agent on an internal canary project. Canary content can precede the stable customer rollout. The prompt carries only the task and its source material; the agent brings its own machine-installed marketingskills pack, hosted tools, and web research, exactly as it does for customers. The candidate model is pinned per chat session and verified from usage metadata, and nothing external is published or mutated. Every task ran in its own fresh chat session, deleted afterwards, so no task could see another task's transcript — the same starting point the direct configurations get.
Cost and token columns for this configuration count every LLM call the agent made during its working turn, so the number is the agent's full bill rather than the cost of its final answer alone. Grading is identical: same rubric, same blinded judge, same quality score and task-pass rules as every other configuration.
Development tasks help us calibrate the rubric by reviewing answers and clarifying requirements. The four tasks labeled holdout in the September 2026 suite have also been inspected. Their original split labels remain in the reports for traceability, but those scores are not untouched validation. A future untouched test requires new, unseen tasks.
The design borrows deliberately: expert-authored per-task rubrics from Harvey's BigLaw Bench, the strict separation of quality from cost and speed from Artificial Analysis, repeat-attempt reliability (pass^k) from τ-bench, and real-deliverable grading in the spirit of OpenAI's GDPval — applied to marketing, with a partial-credit quality score alongside all-or-nothing task passes.
The tasks, fixtures, rubrics, grading code, and raw run evidence live in the eval-runner repository. Reproducing a run is one line: ./eval.py <model>. Judge verdicts are advisory until human-verified, and reports say which is which.