← Marketing Benchmark
Run 20260904-201447-55a6e18f
Sep 4, 2026harness magister-agent20 tasks1 attempt/taskskills offtools onsessions fresh per task

Scoreboard

ModelScoreTasks passedDevelopmentHoldoutAttemptsCostAvg timeTokens
Magister Gemini 3.8 Flash high effort94.514/2011/163/414/20$2.76108.7s169.3k

Task grid

TaskMagister Gemini 3.8 Flash
development
ad-creatives
Meta ad concepts with channel limits
✓
ad-performance-analysis
Paid ad campaign performance analysis
✗
ads-ledger-analysis
Quarterly paid-media review from the raw daily export
✓
analytics-source-conflict
Reconcile conflicting Meta and GA4 performance numbers
✗
attribution-sensitivity
Attribution-model sensitivity analysis with duplicate and refund
✗
blog-post
SEO blog post for a local service business
✓
budget-reconciliation-trap
Reconcile an infeasible Q4 budget brief
✓
email-flow-repair
Repair an email onboarding sequence against the supplied facts
✓
factorial-ad-test
Constraint-dense 2x2 Meta creative experiment
✓
headline
Website hero headlines for a product launch
✓
landing-page-repair
Repair a landing page with planted conversion and compliance defects
✓
landing-page
Single-file HTML landing page
✓
marketing-plan
90-day marketing plan for a fictional B2B SaaS
✗
shopify-meta-capi
Shopify to Meta ads conversion tracking implementation
✗
source-conflict-repair
Audit and repair a contradictory agency ad brief
✓
tracking-audit
Pre-Q4 tracking audit against the documented standards
✓
holdout
launch-capacity-trap
Reconcile an over-committed launch production schedule
✓
platform-limits-gauntlet
Cross-platform promo copy under hard character limits
✓
rate-card-reconciliation
Audit a vendor invoice against the signed agreement
✓
unsupportable-claims-brief
Deliver a franchise brief without its unsupportable claims
✗

✓ passed attempt · ✗ failed attempt · E provider error · T truncated. Every attempt must pass for the task to count. Click a task to read the answers and judge evidence.

What failed, exactly

RequirementTaskFailed attempts
partial_metric_denominators
Any Launch Teaser CPA or ROAS uses spend and outcomes from exactly the same complete rows, or is reported as unavailable. Dividing all four weeks of spend by only three known weeks of conversions/revenue fails.
ad-performance-analysis1
prioritized_action
Recommendations are prioritized (one clear primary move) rather than an unranked laundry list.
ad-performance-analysis1
documented_causes_only
The disagreement is explained using only the documented causes — view-through counting, the differing attribution windows/models, the Oct 12-13 lost-UTM sessions classified as Direct, and cross-platform over-claiming (149 claimed versus 118 store orders) — with no invented causes such as privacy signal loss, bot traffic, or tracking-pixel bugs.
analytics-source-conflict1
conditional_recommendation
The recommendation correctly distinguishes model-robust from model-dependent conclusions, does not declare a universal Meta-versus-Email winner, and does not treat either attribution rule as causal grounds for a decisive reallocation. It may either note that Paid Search is not the descriptive winner under these rules or defer all reallocation pending stronger evidence.
attribution-sensitivity1
three_channels
Exactly three channels are chosen, each with a rationale, and the budget allocation across them sums to the $15,000 total.
marketing-plan1
no_invented_proof
No invented customers, logos, revenue, market statistics, benchmark conversion rates stated as fact, or testimonials; assumptions are labeled as assumptions.
marketing-plan1
sequenced_90_days
The 90 days are phased (e.g. by month or milestone) with dependencies acknowledged, and the positioning ('faster, visible handoffs without adding another weekly status meeting') carries through the messaging.
marketing-plan1
event_time_stable
The server Purchase event uses the purchase/order occurrence timestamp in Unix seconds and preserves it across retries; using the current send or retry time fails.
shopify-meta-capi1
no_softened_claims
The copy does not smuggle declined claims back in softened form — no 'highly rated', 'trusted by hundreds', 'usually there within hours', 'up to' savings language, invented credentials, or experience proxies standing in for the declined proof.
unsupportable-claims-brief1
evidence_unlock
The Declined items section tells the owner what evidence or approval would unlock each claim later (for example a review-platform export, a written and honored guarantee policy, or documented certification), so the pushback is actionable rather than a flat no.
unsupportable-claims-brief1