| Model | Score | Tasks passed | Development | Holdout | Attempts | Cost | Avg time | Tokens |
|---|---|---|---|---|---|---|---|---|
| Magister Gemini 3.8 Flash high effort | 94.5 | 14/20 | 11/16 | 3/4 | 14/20 | $2.76 | 108.7s | 169.3k |
| Task | Magister Gemini 3.8 Flash |
|---|---|
| development | |
| ad-creatives Meta ad concepts with channel limits | ✓ |
| ad-performance-analysis Paid ad campaign performance analysis | ✗ |
| ads-ledger-analysis Quarterly paid-media review from the raw daily export | ✓ |
| analytics-source-conflict Reconcile conflicting Meta and GA4 performance numbers | ✗ |
| attribution-sensitivity Attribution-model sensitivity analysis with duplicate and refund | ✗ |
| blog-post SEO blog post for a local service business | ✓ |
| budget-reconciliation-trap Reconcile an infeasible Q4 budget brief | ✓ |
| email-flow-repair Repair an email onboarding sequence against the supplied facts | ✓ |
| factorial-ad-test Constraint-dense 2x2 Meta creative experiment | ✓ |
| headline Website hero headlines for a product launch | ✓ |
| landing-page-repair Repair a landing page with planted conversion and compliance defects | ✓ |
| landing-page Single-file HTML landing page | ✓ |
| marketing-plan 90-day marketing plan for a fictional B2B SaaS | ✗ |
| shopify-meta-capi Shopify to Meta ads conversion tracking implementation | ✗ |
| source-conflict-repair Audit and repair a contradictory agency ad brief | ✓ |
| tracking-audit Pre-Q4 tracking audit against the documented standards | ✓ |
| holdout | |
| launch-capacity-trap Reconcile an over-committed launch production schedule | ✓ |
| platform-limits-gauntlet Cross-platform promo copy under hard character limits | ✓ |
| rate-card-reconciliation Audit a vendor invoice against the signed agreement | ✓ |
| unsupportable-claims-brief Deliver a franchise brief without its unsupportable claims | ✗ |
✓ passed attempt · ✗ failed attempt · E provider error · T truncated. Every attempt must pass for the task to count. Click a task to read the answers and judge evidence.
| Requirement | Task | Failed attempts |
|---|---|---|
| partial_metric_denominators Any Launch Teaser CPA or ROAS uses spend and outcomes from exactly the same complete rows, or is reported as unavailable. Dividing all four weeks of spend by only three known weeks of conversions/revenue fails. | ad-performance-analysis | 1 |
| prioritized_action Recommendations are prioritized (one clear primary move) rather than an unranked laundry list. | ad-performance-analysis | 1 |
| documented_causes_only The disagreement is explained using only the documented causes — view-through counting, the differing attribution windows/models, the Oct 12-13 lost-UTM sessions classified as Direct, and cross-platform over-claiming (149 claimed versus 118 store orders) — with no invented causes such as privacy signal loss, bot traffic, or tracking-pixel bugs. | analytics-source-conflict | 1 |
| conditional_recommendation The recommendation correctly distinguishes model-robust from model-dependent conclusions, does not declare a universal Meta-versus-Email winner, and does not treat either attribution rule as causal grounds for a decisive reallocation. It may either note that Paid Search is not the descriptive winner under these rules or defer all reallocation pending stronger evidence. | attribution-sensitivity | 1 |
| three_channels Exactly three channels are chosen, each with a rationale, and the budget allocation across them sums to the $15,000 total. | marketing-plan | 1 |
| no_invented_proof No invented customers, logos, revenue, market statistics, benchmark conversion rates stated as fact, or testimonials; assumptions are labeled as assumptions. | marketing-plan | 1 |
| sequenced_90_days The 90 days are phased (e.g. by month or milestone) with dependencies acknowledged, and the positioning ('faster, visible handoffs without adding another weekly status meeting') carries through the messaging. | marketing-plan | 1 |
| event_time_stable The server Purchase event uses the purchase/order occurrence timestamp in Unix seconds and preserves it across retries; using the current send or retry time fails. | shopify-meta-capi | 1 |
| no_softened_claims The copy does not smuggle declined claims back in softened form — no 'highly rated', 'trusted by hundreds', 'usually there within hours', 'up to' savings language, invented credentials, or experience proxies standing in for the declined proof. | unsupportable-claims-brief | 1 |
| evidence_unlock The Declined items section tells the owner what evidence or approval would unlock each claim later (for example a review-platform export, a written and honored guarantee policy, or documented certification), so the pushback is actionable rather than a flat no. | unsupportable-claims-brief | 1 |