← Marketing Benchmark
Run 20260904-150307-2b65f658
Sep 4, 2026harness direct-api20 tasks1 attempt/taskskills offtools on

Scoreboard

ModelScoreTasks passedDevelopmentHoldoutAttemptsCostAvg timeTokens
Gemini 3.8 Flash high effort78.48/207/161/48/20$1.37102.5s353.8k

Task grid

TaskGemini 3.8 Flash
development
ad-creatives
Meta ad concepts with channel limits
✗
ad-performance-analysis
Paid ad campaign performance analysis
✗
ads-ledger-analysis
Quarterly paid-media review from the raw daily export
✗
analytics-source-conflict
Reconcile conflicting Meta and GA4 performance numbers
✓
attribution-sensitivity
Attribution-model sensitivity analysis with duplicate and refund
✓
blog-post
SEO blog post for a local service business
✓
budget-reconciliation-trap
Reconcile an infeasible Q4 budget brief
✗
email-flow-repair
Repair an email onboarding sequence against the supplied facts
✗
factorial-ad-test
Constraint-dense 2x2 Meta creative experiment
✓
headline
Website hero headlines for a product launch
✓
landing-page-repair
Repair a landing page with planted conversion and compliance defects
T
landing-page
Single-file HTML landing page
✗
marketing-plan
90-day marketing plan for a fictional B2B SaaS
✗
shopify-meta-capi
Shopify to Meta ads conversion tracking implementation
✓
source-conflict-repair
Audit and repair a contradictory agency ad brief
✓
tracking-audit
Pre-Q4 tracking audit against the documented standards
T
holdout
launch-capacity-trap
Reconcile an over-committed launch production schedule
T
platform-limits-gauntlet
Cross-platform promo copy under hard character limits
T
rate-card-reconciliation
Audit a vendor invoice against the signed agreement
✓
unsupportable-claims-brief
Deliver a franchise brief without its unsupportable claims
✗

✓ passed attempt · ✗ failed attempt · E provider error · T truncated. Every attempt must pass for the task to count. Click a task to read the answers and judge evidence.

What failed, exactly

RequirementTaskFailed attempts
hypothesis_prediction
Each Test hypothesis names an observable KPI and its expected direction. Vague outcomes such as 'resonate', 'drive interest', or 'increase engagement' without a named metric fail.
ad-creatives1
partial_metric_denominators
Any Launch Teaser CPA or ROAS uses spend and outcomes from exactly the same complete rows, or is reported as unavailable. Dividing all four weeks of spend by only three known weeks of conversions/revenue fails.
ad-performance-analysis1
prioritized_action
Recommendations are prioritized (one clear primary move) rather than an unranked laundry list.
ad-performance-analysis1
no_invented_data
Every figure presented as coming from the export is derivable from the supplied CSV; no external benchmarks, industry averages, or invented segment data are presented as if they were in the data.
ads-ledger-analysis1
grounded_recommendation
One option is recommended with a rationale tied to the brief's or brand's stated goals, without invented performance predictions or unsupported ROI claims.
budget-reconciliation-trap1
no_new_claims
No new product fact, capability, audience, proof point, offer, or performance claim appears beyond the supplied sources.
email-flow-repair1
complete_html
The response is one complete, parseable HTML document from an HTML5 doctype through the closing html tag, with no Markdown fence.
landing-page-repair1
sections_and_order
All requested sections appear in the exact cautious-visitor decision order.
landing-page-repair1
guarantee_removed
All cure and guaranteed-improvement claims are absent, and the page explicitly says outcomes are not guaranteed.
landing-page-repair1
insurance_removed
The insurance-coverage claim is absent, and the page explicitly says insurance is not represented.
landing-page-repair1
emergency_direction
The page says it is not emergency care and directs urgent symptoms or safety concerns to local emergency services or an appropriate licensed clinician.
landing-page-repair1
conversion_clarity
The corrected page makes the educational offer, fit, included value, limitations, price, and next step concrete without replacing the planted defects with vague wellness copy.
landing-page-repair1
heading_hierarchy
The document has exactly one h1 and ordered heading levels with no skipped levels.
landing-page1
copy_capacity_confirmed
The response states that copywriter hours fit: 92 required (48 + 12 + 12 + 10 + 10) against 96 available (24 x 4). It does not claim the copywriter is over capacity.
launch-capacity-trap1
explicit_constraint_changes
Every resolution option names exactly which stated commitment it changes (which deliverables are cut or reduced, that the go-live date moves, or that new capacity must be approved) rather than silently altering or omitting one.
launch-capacity-trap1
auditable_arithmetic
Hour totals per person are numeric, follow the brief's per-unit production standards, and sum correctly in the original reconciliation and in every option.
launch-capacity-trap1
multiple_options
At least two materially different resolution options are proposed, each with a complete revised deliverable list whose copywriter hours fit 96 and designer hours fit 48 (or fit an explicitly approved capacity increase), and a go-live date outside the freeze.
launch-capacity-trap1
grounded_recommendation
One option is recommended with a rationale tied to the launch's or brand's stated goals, without invented performance predictions or unsupported claims about what the content will achieve.
launch-capacity-trap1
sequenced_90_days
The 90 days are phased (e.g. by month or milestone) with dependencies acknowledged, and the positioning ('faster, visible handoffs without adding another weekly status meeting') carries through the messaging.
marketing-plan1
mature_activation_denominator
Activation rate is defined using only trial starts whose full 14-day activation window has closed; newer trials are explicitly pending rather than counted as non-activated.
marketing-plan1
all_fields_present
All eleven labeled fields (RSA-H1, RSA-H2, RSA-H3, RSA-D1, RSA-D2, META-PRIMARY, META-HEADLINE, META-CTA, LI-INTRO, LI-CTA, SMS-BODY) are present; the first line bearing each label is the graded field.
platform-limits-gauntlet1
approved_cta_exact
META-CTA and LI-CTA are each exactly 'Review consultation availability' (a single terminal period is accepted, matching the brand facts' own punctuation).
platform-limits-gauntlet1
disclaimer_everywhere
The exact sentence 'Educational only; not medical care.' appears inside META-PRIMARY, LI-INTRO, and SMS-BODY.
platform-limits-gauntlet1
sms_optout_present
SMS-BODY contains 'Reply STOP to opt out'.
platform-limits-gauntlet1
accurate_price
Every stated dollar amount is exactly $160, and the price appears at least once.
platform-limits-gauntlet1
grounded_claims
Every claim uses only the supplied facts (45-minute virtual consultation, $160 paid directly, intake review, routine-planning conversation, written educational summary, adults organizing non-clinical bedtime habits). No outcomes, testimonials, credentials, guarantees, urgency, scarcity, discounts, or clinical framing (diagnosis or treatment) are invented.
platform-limits-gauntlet1
single_ask
Each placement carries one job: getting the reader to review consultation availability. No placement adds a competing ask such as a newsletter signup, follow, download, or second offer.
platform-limits-gauntlet1
distinct_rsa_assets
The three RSA headlines take meaningfully different angles rather than near-duplicate phrasings, and the two RSA descriptions differ in substance.
platform-limits-gauntlet1
honest_specificity
Copy leans on the concrete supplied specifics (45 minutes, virtual, written summary, non-clinical routine help) instead of generic superlatives or vague promises of better sleep.
platform-limits-gauntlet1
findings_located
Each reported defect names its specific location — the exact link/channel, tag, trigger, or setting — precisely enough that an engineer could act without re-deriving the audit.
tracking-audit1
finds_uppercase_medium
Flags the uppercase utm_medium=CPC values on the two google-search links as violating the lowercase standard and fragmenting paid-search reporting.
tracking-audit1
finds_uppercase_medium_pmax
Also flags the google-pmax link's uppercase utm_medium=CPC (the same defect on the Performance Max campaign, not only on the search campaigns).
tracking-audit1
finds_missing_source
Flags the klaviyo-welcome link as missing utm_source entirely.
tracking-audit1
finds_meta_source_inconsistency
Flags that some Meta links use utm_source=facebook while the standard (and other links) use utm_source=meta, splitting Meta reporting.
tracking-audit1
finds_campaign_spaces
Flags the meta-summer link's utm_campaign value ('2026q3 summer sale') for containing spaces / violating the hyphenated campaign format.
tracking-audit1
finds_duplicate_campaign_param
Flags the cart-recovery link carrying utm_campaign twice with different values, and ties it to the cart-recovery campaign appearing under two names.
tracking-audit1
fix_per_finding
Every reported defect comes with a concrete, implementable fix (what to change, not 'review this').
tracking-audit1
evidence_unlock
The Declined items section tells the owner what evidence or approval would unlock each claim later (for example a review-platform export, a written and honored guarantee policy, or documented certification), so the pushback is actionable rather than a flat no.
unsupportable-claims-brief1