| Model | Score | Tasks passed | Development | Holdout | Attempts | Cost | Avg time | Tokens |
|---|---|---|---|---|---|---|---|---|
| Gemini 3.8 Flash high effort | 78.4 | 8/20 | 7/16 | 1/4 | 8/20 | $1.37 | 102.5s | 353.8k |
| Task | Gemini 3.8 Flash |
|---|---|
| development | |
| ad-creatives Meta ad concepts with channel limits | ✗ |
| ad-performance-analysis Paid ad campaign performance analysis | ✗ |
| ads-ledger-analysis Quarterly paid-media review from the raw daily export | ✗ |
| analytics-source-conflict Reconcile conflicting Meta and GA4 performance numbers | ✓ |
| attribution-sensitivity Attribution-model sensitivity analysis with duplicate and refund | ✓ |
| blog-post SEO blog post for a local service business | ✓ |
| budget-reconciliation-trap Reconcile an infeasible Q4 budget brief | ✗ |
| email-flow-repair Repair an email onboarding sequence against the supplied facts | ✗ |
| factorial-ad-test Constraint-dense 2x2 Meta creative experiment | ✓ |
| headline Website hero headlines for a product launch | ✓ |
| landing-page-repair Repair a landing page with planted conversion and compliance defects | T |
| landing-page Single-file HTML landing page | ✗ |
| marketing-plan 90-day marketing plan for a fictional B2B SaaS | ✗ |
| shopify-meta-capi Shopify to Meta ads conversion tracking implementation | ✓ |
| source-conflict-repair Audit and repair a contradictory agency ad brief | ✓ |
| tracking-audit Pre-Q4 tracking audit against the documented standards | T |
| holdout | |
| launch-capacity-trap Reconcile an over-committed launch production schedule | T |
| platform-limits-gauntlet Cross-platform promo copy under hard character limits | T |
| rate-card-reconciliation Audit a vendor invoice against the signed agreement | ✓ |
| unsupportable-claims-brief Deliver a franchise brief without its unsupportable claims | ✗ |
✓ passed attempt · ✗ failed attempt · E provider error · T truncated. Every attempt must pass for the task to count. Click a task to read the answers and judge evidence.
| Requirement | Task | Failed attempts |
|---|---|---|
| hypothesis_prediction Each Test hypothesis names an observable KPI and its expected direction. Vague outcomes such as 'resonate', 'drive interest', or 'increase engagement' without a named metric fail. | ad-creatives | 1 |
| partial_metric_denominators Any Launch Teaser CPA or ROAS uses spend and outcomes from exactly the same complete rows, or is reported as unavailable. Dividing all four weeks of spend by only three known weeks of conversions/revenue fails. | ad-performance-analysis | 1 |
| prioritized_action Recommendations are prioritized (one clear primary move) rather than an unranked laundry list. | ad-performance-analysis | 1 |
| no_invented_data Every figure presented as coming from the export is derivable from the supplied CSV; no external benchmarks, industry averages, or invented segment data are presented as if they were in the data. | ads-ledger-analysis | 1 |
| grounded_recommendation One option is recommended with a rationale tied to the brief's or brand's stated goals, without invented performance predictions or unsupported ROI claims. | budget-reconciliation-trap | 1 |
| no_new_claims No new product fact, capability, audience, proof point, offer, or performance claim appears beyond the supplied sources. | email-flow-repair | 1 |
| complete_html The response is one complete, parseable HTML document from an HTML5 doctype through the closing html tag, with no Markdown fence. | landing-page-repair | 1 |
| sections_and_order All requested sections appear in the exact cautious-visitor decision order. | landing-page-repair | 1 |
| guarantee_removed All cure and guaranteed-improvement claims are absent, and the page explicitly says outcomes are not guaranteed. | landing-page-repair | 1 |
| insurance_removed The insurance-coverage claim is absent, and the page explicitly says insurance is not represented. | landing-page-repair | 1 |
| emergency_direction The page says it is not emergency care and directs urgent symptoms or safety concerns to local emergency services or an appropriate licensed clinician. | landing-page-repair | 1 |
| conversion_clarity The corrected page makes the educational offer, fit, included value, limitations, price, and next step concrete without replacing the planted defects with vague wellness copy. | landing-page-repair | 1 |
| heading_hierarchy The document has exactly one h1 and ordered heading levels with no skipped levels. | landing-page | 1 |
| copy_capacity_confirmed The response states that copywriter hours fit: 92 required (48 + 12 + 12 + 10 + 10) against 96 available (24 x 4). It does not claim the copywriter is over capacity. | launch-capacity-trap | 1 |
| explicit_constraint_changes Every resolution option names exactly which stated commitment it changes (which deliverables are cut or reduced, that the go-live date moves, or that new capacity must be approved) rather than silently altering or omitting one. | launch-capacity-trap | 1 |
| auditable_arithmetic Hour totals per person are numeric, follow the brief's per-unit production standards, and sum correctly in the original reconciliation and in every option. | launch-capacity-trap | 1 |
| multiple_options At least two materially different resolution options are proposed, each with a complete revised deliverable list whose copywriter hours fit 96 and designer hours fit 48 (or fit an explicitly approved capacity increase), and a go-live date outside the freeze. | launch-capacity-trap | 1 |
| grounded_recommendation One option is recommended with a rationale tied to the launch's or brand's stated goals, without invented performance predictions or unsupported claims about what the content will achieve. | launch-capacity-trap | 1 |
| sequenced_90_days The 90 days are phased (e.g. by month or milestone) with dependencies acknowledged, and the positioning ('faster, visible handoffs without adding another weekly status meeting') carries through the messaging. | marketing-plan | 1 |
| mature_activation_denominator Activation rate is defined using only trial starts whose full 14-day activation window has closed; newer trials are explicitly pending rather than counted as non-activated. | marketing-plan | 1 |
| all_fields_present All eleven labeled fields (RSA-H1, RSA-H2, RSA-H3, RSA-D1, RSA-D2, META-PRIMARY, META-HEADLINE, META-CTA, LI-INTRO, LI-CTA, SMS-BODY) are present; the first line bearing each label is the graded field. | platform-limits-gauntlet | 1 |
| approved_cta_exact META-CTA and LI-CTA are each exactly 'Review consultation availability' (a single terminal period is accepted, matching the brand facts' own punctuation). | platform-limits-gauntlet | 1 |
| disclaimer_everywhere The exact sentence 'Educational only; not medical care.' appears inside META-PRIMARY, LI-INTRO, and SMS-BODY. | platform-limits-gauntlet | 1 |
| sms_optout_present SMS-BODY contains 'Reply STOP to opt out'. | platform-limits-gauntlet | 1 |
| accurate_price Every stated dollar amount is exactly $160, and the price appears at least once. | platform-limits-gauntlet | 1 |
| grounded_claims Every claim uses only the supplied facts (45-minute virtual consultation, $160 paid directly, intake review, routine-planning conversation, written educational summary, adults organizing non-clinical bedtime habits). No outcomes, testimonials, credentials, guarantees, urgency, scarcity, discounts, or clinical framing (diagnosis or treatment) are invented. | platform-limits-gauntlet | 1 |
| single_ask Each placement carries one job: getting the reader to review consultation availability. No placement adds a competing ask such as a newsletter signup, follow, download, or second offer. | platform-limits-gauntlet | 1 |
| distinct_rsa_assets The three RSA headlines take meaningfully different angles rather than near-duplicate phrasings, and the two RSA descriptions differ in substance. | platform-limits-gauntlet | 1 |
| honest_specificity Copy leans on the concrete supplied specifics (45 minutes, virtual, written summary, non-clinical routine help) instead of generic superlatives or vague promises of better sleep. | platform-limits-gauntlet | 1 |
| findings_located Each reported defect names its specific location — the exact link/channel, tag, trigger, or setting — precisely enough that an engineer could act without re-deriving the audit. | tracking-audit | 1 |
| finds_uppercase_medium Flags the uppercase utm_medium=CPC values on the two google-search links as violating the lowercase standard and fragmenting paid-search reporting. | tracking-audit | 1 |
| finds_uppercase_medium_pmax Also flags the google-pmax link's uppercase utm_medium=CPC (the same defect on the Performance Max campaign, not only on the search campaigns). | tracking-audit | 1 |
| finds_missing_source Flags the klaviyo-welcome link as missing utm_source entirely. | tracking-audit | 1 |
| finds_meta_source_inconsistency Flags that some Meta links use utm_source=facebook while the standard (and other links) use utm_source=meta, splitting Meta reporting. | tracking-audit | 1 |
| finds_campaign_spaces Flags the meta-summer link's utm_campaign value ('2026q3 summer sale') for containing spaces / violating the hyphenated campaign format. | tracking-audit | 1 |
| finds_duplicate_campaign_param Flags the cart-recovery link carrying utm_campaign twice with different values, and ties it to the cart-recovery campaign appearing under two names. | tracking-audit | 1 |
| fix_per_finding Every reported defect comes with a concrete, implementable fix (what to change, not 'review this'). | tracking-audit | 1 |
| evidence_unlock The Declined items section tells the owner what evidence or approval would unlock each claim later (for example a review-platform export, a written and honored guarantee policy, or documented certification), so the pushback is actionable rather than a flat no. | unsupportable-claims-brief | 1 |