← Marketing Benchmark
Run 20260904-151257-1cd41651
Sep 4, 2026harness direct-api20 tasks1 attempt/taskskills ontools on

Scoreboard

ModelScoreTasks passedDevelopmentHoldoutAttemptsCostAvg timeTokens
Gemini 3.8 Flash high effort87.212/209/163/412/20$1.69151.9s425.5k

Task grid

TaskGemini 3.8 Flash
development
ad-creatives
Meta ad concepts with channel limits
✓
ad-performance-analysis
Paid ad campaign performance analysis
✗
ads-ledger-analysis
Quarterly paid-media review from the raw daily export
T
analytics-source-conflict
Reconcile conflicting Meta and GA4 performance numbers
✓
attribution-sensitivity
Attribution-model sensitivity analysis with duplicate and refund
T
blog-post
SEO blog post for a local service business
✓
budget-reconciliation-trap
Reconcile an infeasible Q4 budget brief
✗
email-flow-repair
Repair an email onboarding sequence against the supplied facts
✗
factorial-ad-test
Constraint-dense 2x2 Meta creative experiment
✓
headline
Website hero headlines for a product launch
✓
landing-page-repair
Repair a landing page with planted conversion and compliance defects
✓
landing-page
Single-file HTML landing page
✗
marketing-plan
90-day marketing plan for a fictional B2B SaaS
✗
shopify-meta-capi
Shopify to Meta ads conversion tracking implementation
✓
source-conflict-repair
Audit and repair a contradictory agency ad brief
✓
tracking-audit
Pre-Q4 tracking audit against the documented standards
T
holdout
launch-capacity-trap
Reconcile an over-committed launch production schedule
T
platform-limits-gauntlet
Cross-platform promo copy under hard character limits
✓
rate-card-reconciliation
Audit a vendor invoice against the signed agreement
✓
unsupportable-claims-brief
Deliver a franchise brief without its unsupportable claims
✓

✓ passed attempt · ✗ failed attempt · E provider error · T truncated. Every attempt must pass for the task to count. Click a task to read the answers and judge evidence.

What failed, exactly

RequirementTaskFailed attempts
partial_metric_denominators
Any Launch Teaser CPA or ROAS uses spend and outcomes from exactly the same complete rows, or is reported as unavailable. Dividing all four weeks of spend by only three known weeks of conversions/revenue fails.
ad-performance-analysis1
answer_block_complete
The response ends with the required answer block and all five labeled answers A1-A5 are present and parseable.
ads-ledger-analysis1
q1_blended_cpa
A1 states the blended CPA (total spend / net conversions) for the full 14 weeks; full credit within 0.5% of the value computed from the export, zero credit beyond 10% off.
ads-ledger-analysis1
q2_wasted_spend
A2 states total spend across zero-conversion adset-days; full credit within 1%, zero credit beyond 25% off.
ads-ledger-analysis1
q3_underwater_week
A3 names the week (1-14) in which prospecting-broad's weekly net ROAS first fell below 1.0; full credit for the exact week, partial credit within one week, zero credit two or more weeks off.
ads-ledger-analysis1
q4_refund_drag
A4 states the total refunded revenue of the adset with the most refunded revenue; full credit within 1%, zero credit beyond 25% off.
ads-ledger-analysis1
q5_final_roas
A5 states the account-wide net ROAS for the final 28 days; full credit within 1%, zero credit beyond 15% off.
ads-ledger-analysis1
worst_refund_adset_named
The review names rt-lookalike as the adset with the largest refunded revenue.
ads-ledger-analysis1
prospecting_decline_called
The review identifies prospecting-broad's performance decline across the window as a trend requiring action, not merely a low average.
ads-ledger-analysis1
pb_video_flagged
The review specifically calls out the pb-video adset as continuing to spend while producing zero conversions in the final weeks (or an equivalent adset-level wasted-spend identification naming pb-video).
ads-ledger-analysis1
recommendation_grounded
Each recommendation is tied to specific figures from the analysis (named campaigns/adsets with their numbers), not generic best practice.
ads-ledger-analysis1
grounded_recommendation
One option is recommended with a rationale tied to the brief's or brand's stated goals, without invented performance predictions or unsupported ROI claims.
budget-reconciliation-trap1
no_new_claims
No new product fact, capability, audience, proof point, offer, or performance claim appears beyond the supplied sources.
email-flow-repair1
no_external_assets
The HTML uses inline CSS and has no CSS asset references, active content, scripts, images, stylesheets, fonts, media, iframes, embeds, objects, SVG, or canvas assets.
landing-page1
copy_capacity_confirmed
The response states that copywriter hours fit: 92 required (48 + 12 + 12 + 10 + 10) against 96 available (24 x 4). It does not claim the copywriter is over capacity.
launch-capacity-trap1
explicit_constraint_changes
Every resolution option names exactly which stated commitment it changes (which deliverables are cut or reduced, that the go-live date moves, or that new capacity must be approved) rather than silently altering or omitting one.
launch-capacity-trap1
auditable_arithmetic
Hour totals per person are numeric, follow the brief's per-unit production standards, and sum correctly in the original reconciliation and in every option.
launch-capacity-trap1
multiple_options
At least two materially different resolution options are proposed, each with a complete revised deliverable list whose copywriter hours fit 96 and designer hours fit 48 (or fit an explicitly approved capacity increase), and a go-live date outside the freeze.
launch-capacity-trap1
grounded_recommendation
One option is recommended with a rationale tied to the launch's or brand's stated goals, without invented performance predictions or unsupported claims about what the content will achieve.
launch-capacity-trap1
no_invented_proof
No invented customers, logos, revenue, market statistics, benchmark conversion rates stated as fact, or testimonials; assumptions are labeled as assumptions.
marketing-plan1
finds_uppercase_medium
Flags the uppercase utm_medium=CPC values on the two google-search links as violating the lowercase standard and fragmenting paid-search reporting.
tracking-audit1
finds_uppercase_medium_pmax
Also flags the google-pmax link's uppercase utm_medium=CPC (the same defect on the Performance Max campaign, not only on the search campaigns).
tracking-audit1
finds_missing_source
Flags the klaviyo-welcome link as missing utm_source entirely.
tracking-audit1
finds_meta_source_inconsistency
Flags that some Meta links use utm_source=facebook while the standard (and other links) use utm_source=meta, splitting Meta reporting.
tracking-audit1
finds_campaign_spaces
Flags the meta-summer link's utm_campaign value ('2026q3 summer sale') for containing spaces / violating the hyphenated campaign format.
tracking-audit1
finds_missing_transaction_id
Identifies that the GA4 purchase tags omit the transaction_id parameter required by the standards, so duplicate purchase events cannot be deduplicated.
tracking-audit1
finds_event_name_casing
Flags the 'Add to Cart' event name (spaces, not snake_case) as violating the naming standard and explains the add-to-cart reporting going quiet after the rename.
tracking-audit1
finds_duplicate_campaign_param
Flags the cart-recovery link carrying utm_campaign twice with different values, and ties it to the cart-recovery campaign appearing under two names.
tracking-audit1
fix_per_finding
Every reported defect comes with a concrete, implementable fix (what to change, not 'review this').
tracking-audit1