| Model | Score | Tasks passed | Development | Holdout | Attempts | Cost | Avg time | Tokens |
|---|---|---|---|---|---|---|---|---|
| Gemini 3.8 Flash high effort | 87.2 | 12/20 | 9/16 | 3/4 | 12/20 | $1.69 | 151.9s | 425.5k |
| Task | Gemini 3.8 Flash |
|---|---|
| development | |
| ad-creatives Meta ad concepts with channel limits | ✓ |
| ad-performance-analysis Paid ad campaign performance analysis | ✗ |
| ads-ledger-analysis Quarterly paid-media review from the raw daily export | T |
| analytics-source-conflict Reconcile conflicting Meta and GA4 performance numbers | ✓ |
| attribution-sensitivity Attribution-model sensitivity analysis with duplicate and refund | T |
| blog-post SEO blog post for a local service business | ✓ |
| budget-reconciliation-trap Reconcile an infeasible Q4 budget brief | ✗ |
| email-flow-repair Repair an email onboarding sequence against the supplied facts | ✗ |
| factorial-ad-test Constraint-dense 2x2 Meta creative experiment | ✓ |
| headline Website hero headlines for a product launch | ✓ |
| landing-page-repair Repair a landing page with planted conversion and compliance defects | ✓ |
| landing-page Single-file HTML landing page | ✗ |
| marketing-plan 90-day marketing plan for a fictional B2B SaaS | ✗ |
| shopify-meta-capi Shopify to Meta ads conversion tracking implementation | ✓ |
| source-conflict-repair Audit and repair a contradictory agency ad brief | ✓ |
| tracking-audit Pre-Q4 tracking audit against the documented standards | T |
| holdout | |
| launch-capacity-trap Reconcile an over-committed launch production schedule | T |
| platform-limits-gauntlet Cross-platform promo copy under hard character limits | ✓ |
| rate-card-reconciliation Audit a vendor invoice against the signed agreement | ✓ |
| unsupportable-claims-brief Deliver a franchise brief without its unsupportable claims | ✓ |
✓ passed attempt · ✗ failed attempt · E provider error · T truncated. Every attempt must pass for the task to count. Click a task to read the answers and judge evidence.
| Requirement | Task | Failed attempts |
|---|---|---|
| partial_metric_denominators Any Launch Teaser CPA or ROAS uses spend and outcomes from exactly the same complete rows, or is reported as unavailable. Dividing all four weeks of spend by only three known weeks of conversions/revenue fails. | ad-performance-analysis | 1 |
| answer_block_complete The response ends with the required answer block and all five labeled answers A1-A5 are present and parseable. | ads-ledger-analysis | 1 |
| q1_blended_cpa A1 states the blended CPA (total spend / net conversions) for the full 14 weeks; full credit within 0.5% of the value computed from the export, zero credit beyond 10% off. | ads-ledger-analysis | 1 |
| q2_wasted_spend A2 states total spend across zero-conversion adset-days; full credit within 1%, zero credit beyond 25% off. | ads-ledger-analysis | 1 |
| q3_underwater_week A3 names the week (1-14) in which prospecting-broad's weekly net ROAS first fell below 1.0; full credit for the exact week, partial credit within one week, zero credit two or more weeks off. | ads-ledger-analysis | 1 |
| q4_refund_drag A4 states the total refunded revenue of the adset with the most refunded revenue; full credit within 1%, zero credit beyond 25% off. | ads-ledger-analysis | 1 |
| q5_final_roas A5 states the account-wide net ROAS for the final 28 days; full credit within 1%, zero credit beyond 15% off. | ads-ledger-analysis | 1 |
| worst_refund_adset_named The review names rt-lookalike as the adset with the largest refunded revenue. | ads-ledger-analysis | 1 |
| prospecting_decline_called The review identifies prospecting-broad's performance decline across the window as a trend requiring action, not merely a low average. | ads-ledger-analysis | 1 |
| pb_video_flagged The review specifically calls out the pb-video adset as continuing to spend while producing zero conversions in the final weeks (or an equivalent adset-level wasted-spend identification naming pb-video). | ads-ledger-analysis | 1 |
| recommendation_grounded Each recommendation is tied to specific figures from the analysis (named campaigns/adsets with their numbers), not generic best practice. | ads-ledger-analysis | 1 |
| grounded_recommendation One option is recommended with a rationale tied to the brief's or brand's stated goals, without invented performance predictions or unsupported ROI claims. | budget-reconciliation-trap | 1 |
| no_new_claims No new product fact, capability, audience, proof point, offer, or performance claim appears beyond the supplied sources. | email-flow-repair | 1 |
| no_external_assets The HTML uses inline CSS and has no CSS asset references, active content, scripts, images, stylesheets, fonts, media, iframes, embeds, objects, SVG, or canvas assets. | landing-page | 1 |
| copy_capacity_confirmed The response states that copywriter hours fit: 92 required (48 + 12 + 12 + 10 + 10) against 96 available (24 x 4). It does not claim the copywriter is over capacity. | launch-capacity-trap | 1 |
| explicit_constraint_changes Every resolution option names exactly which stated commitment it changes (which deliverables are cut or reduced, that the go-live date moves, or that new capacity must be approved) rather than silently altering or omitting one. | launch-capacity-trap | 1 |
| auditable_arithmetic Hour totals per person are numeric, follow the brief's per-unit production standards, and sum correctly in the original reconciliation and in every option. | launch-capacity-trap | 1 |
| multiple_options At least two materially different resolution options are proposed, each with a complete revised deliverable list whose copywriter hours fit 96 and designer hours fit 48 (or fit an explicitly approved capacity increase), and a go-live date outside the freeze. | launch-capacity-trap | 1 |
| grounded_recommendation One option is recommended with a rationale tied to the launch's or brand's stated goals, without invented performance predictions or unsupported claims about what the content will achieve. | launch-capacity-trap | 1 |
| no_invented_proof No invented customers, logos, revenue, market statistics, benchmark conversion rates stated as fact, or testimonials; assumptions are labeled as assumptions. | marketing-plan | 1 |
| finds_uppercase_medium Flags the uppercase utm_medium=CPC values on the two google-search links as violating the lowercase standard and fragmenting paid-search reporting. | tracking-audit | 1 |
| finds_uppercase_medium_pmax Also flags the google-pmax link's uppercase utm_medium=CPC (the same defect on the Performance Max campaign, not only on the search campaigns). | tracking-audit | 1 |
| finds_missing_source Flags the klaviyo-welcome link as missing utm_source entirely. | tracking-audit | 1 |
| finds_meta_source_inconsistency Flags that some Meta links use utm_source=facebook while the standard (and other links) use utm_source=meta, splitting Meta reporting. | tracking-audit | 1 |
| finds_campaign_spaces Flags the meta-summer link's utm_campaign value ('2026q3 summer sale') for containing spaces / violating the hyphenated campaign format. | tracking-audit | 1 |
| finds_missing_transaction_id Identifies that the GA4 purchase tags omit the transaction_id parameter required by the standards, so duplicate purchase events cannot be deduplicated. | tracking-audit | 1 |
| finds_event_name_casing Flags the 'Add to Cart' event name (spaces, not snake_case) as violating the naming standard and explains the add-to-cart reporting going quiet after the rename. | tracking-audit | 1 |
| finds_duplicate_campaign_param Flags the cart-recovery link carrying utm_campaign twice with different values, and ties it to the cart-recovery campaign appearing under two names. | tracking-audit | 1 |
| fix_per_finding Every reported defect comes with a concrete, implementable fix (what to change, not 'review this'). | tracking-audit | 1 |