Marketing Benchmark

Answers & verdicts

Gemini 3.8 Flash high effort

Task passed

Task the exact prompt the model was given — assignment, source material, and skill guidance

A founder asks: 'Meta says our ads printed money last month, our analyst says GA4 disagrees. What is actually true, and what should we do?' Using only the supplied reports and tracking notes, deliver: (1) the Meta totals and the blended CPA and ROAS under each source, arithmetic shown; (2) an explanation of why the two sources disagree, grounded strictly in the documented causes; (3) what can honestly be concluded about Meta-attributed purchases, including the defensible range; and (4) a recommendation that separates conclusions robust to the measurement disagreement from conclusions that depend on which source you believe, plus the next measurement step. Do not average the sources into a single number, do not invent causes the notes do not document, and do not treat either source's attribution as proof of causation. --- ## Supplied source: meta-platform-report.csv source,campaign,window,spend_usd,reported_conversions,reported_revenue_usd,attribution_setting meta_ads_manager,ARC-Prospecting,Oct 1-28,1500.00,38,3420.00,"7-day click, 1-day view" meta_ads_manager,ARC-Retargeting,Oct 1-28,900.00,22,1980.00,"7-day click, 1-day view" --- ## Supplied source: ga4-report.csv channel,window,sessions,purchases,revenue_usd,attribution_model Paid Social (Meta),Oct 1-28,1840,36,3240.00,"last non-direct click, 30-day" Organic Search,Oct 1-28,2210,41,3690.00,"last non-direct click, 30-day" Direct,Oct 1-28,960,35,3150.00,"last non-direct click, 30-day" --- ## Supplied source: tracking-notes.md # Tracking notes for the October reconciliation (analyst handoff) - Meta Ads Manager counts view-through conversions under its "1-day view" setting; GA4 never counts view-through under any setting. - Attribution differs by construction: Meta claims a conversion on a 7-day click / 1-day view window against its own ads; GA4 assigns each purchase to one channel by last non-direct click with a 30-day lookback. - Auto-tagging was broken on Oct 12-13; an estimated 8 Meta-referred sessions landed without UTMs during the outage and were classified as Direct in GA4. - The store platform (source of truth for orders) recorded 118 completed purchases in the window. GA4 recorded 112 site-wide. Summing every ad platform's self-reported conversions (Meta plus the search and affiliate platforms) gives 149 claimed conversions. - No other diagnostics are available for this window. --- ## Skill guidance you should follow --- name: analytics description: "Set up or audit tracking — GA4, GTM, Mixpanel, Segment; tracking plans, event naming, UTM conventions, conversion and attribution debugging. Rate metrics divide matching closed windows — never full-period spend by partial-period results." metadata: version: 2.2.0 --- # Analytics Tracking You are an expert in analytics implementation and measurement. Your goal is to help set up tracking that provides actionable insights for marketing and product decisions. ## Initial Assessment **Check for product marketing context first:** If `.agents/product-marketing.md` exists (or `.claude/product-marketing.md`, or the legacy `product-marketing-context.md` filename, in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task. **Apply the project's brand:** If `BRAND.md` exists, read it before asking questions and apply it to everything you produce — its voice, messaging, and audience for anything written, and its colors, typography, and logo for anything visual. Never ask for a brand fact it already answers. If it lacks something you need, name the missing fact instead of inventing one or quietly falling back to a generic default. Before implementing tracking, understand: 1. **Business Context** - What decisions will this data inform? What are key conversions? 2. **Current State** - What tracking exists? What tools are in use? 3. **Technical Context** - What's the tech stack? Any privacy/compliance requirements? --- ## Core Principles ### 1. Track for Decisions, Not Data - Every event should inform a decision - Avoid vanity metrics - Quality > quantity of events ### 2. Start with the Questions - What do you need to know? - What actions will you take based on this data? - Work backwards to what you need to track ### 3. Name Things Consistently - Naming conventions matter - Establish patterns before implementing - Document everything ### 4. Maintain Data Quality - Validate implementation - Monitor for issues - Clean data > more data ### 5. Compute, Don't Eyeball - Derive every total, rate, and delta from row-level data with a script or spreadsheet — never by scanning rows and summing mentally. One silently dropped row corrupts every figure downstream of it. - Cross-check each headline figure a second, independent way (a reconciliation total, a recomputation from a different grouping) before reporting it, and chase any mismatch to the exact row. - When a claim is about one segment — a single campaign, ad set, channel, or week — report that segment's own numbers. Never fold it into a neighboring group and let the group's total speak for it. ### 6. Match Observation Windows - A rate or efficiency metric divides a numerator and denominator that cover the same closed observation window. Never divide a full period's spend by a partial period's conversions or revenue — restate both over the weeks that actually have data, and say which weeks those are. - Cohort rates use only cohorts whose observation window has ended. A trial or signup still inside its window is pending, not failed — exclude it from the denominator and label it as pending rather than counting it against the rate. ### 7. Recommend Against the Stated Goal - Analysis ends in a recommendation, and the recommendation must be argued from the goal the brief states — the launch target, audience, revenue, or positioning objective — not only from what is operationally easiest. Name which stated goal each option serves and what it trades away. --- ## Tracking Plan Framework ### Structure ``` Event Name | Category | Properties | Trigger | Notes ---------- | -------- | ---------- | ------- | ----- ``` ### Event Types | Type | Examples | |------|----------| | Pageviews | Automatic, enhanced with metadata | | User Actions | Button clicks, form submissions, feature usage | | System Events | Signup completed, purchase, subscription changed | | Custom Conversions | Goal completions, funnel stages | **For comprehensive event lists**: See [references/event-library.md](references/event-library.md) --- ## Event Naming Conventions ### Recommended Format: Object-Action ``` signup_completed button_clicked form_submitted article_read checkout_payment_completed ``` ### Best Practices - Lowercase with underscores - Be specific: `cta_hero_clicked` vs. `button_clicked` - Include context in properties, not event name - Avoid spaces and special characters - Document decisions --- ## Essential Events ### Marketing Site | Event | Properties | |-------|------------| | cta_clicked | button_text, location | | form_submitted | form_type | | signup_completed | method, source | | demo_requested | - | ### Product/App | Event | Properties | |-------|------------| | onboarding_step_completed | step_number, step_name | | feature_used | feature_name | | purchase_completed | plan, value | | subscription_cancelled | reason | **For full event library by business type**: See [references/event-library.md](references/event-library.md) --- ## Event Properties ### Standard Properties | Category | Properties | |----------|------------| | Page | page_title, page_location, page_referrer | | User | user_id, user_type, account_id, plan_type | | Campaign | source, medium, campaign, content, term | | Product | product_id, product_name, category, price | ### Best Practices - Use consistent property names - Include relevant context - Don't duplicate automatic properties - Avoid PII in properties --- ## GA4 Implementation ### Quick Setup 1. Create GA4 property and data stream 2. Install gtag.js or GTM 3. Enable enhanced measurement 4. Configure custom events 5. Mark conversions in Admin ### Custom Event Example ```javascript gtag('event', 'signup_completed', { 'method': 'email', 'plan': 'free' }); ``` **For detailed GA4 implementation**: See [references/ga4-implementation.md](references/ga4-implementation.md) --- ## Google Tag Manager ### Container Structure | Component | Purpose | |-----------|---------| | Tags | Code that executes (GA4, pixels) | | Triggers | When tags fire (page view, click) | | Variables | Dynamic values (click text, data layer) | ### Data Layer Pattern ```javascript dataLayer.push({ 'event': 'form_submitted', 'form_name': 'contact', 'form_location': 'footer' }); ``` **For detailed GTM implementation**: See [references/gtm-implementation.md](references/gtm-implementation.md) --- ## UTM Parameter Strategy ### Standard Parameters | Parameter | Purpose | Example | |-----------|---------|---------| | utm_source | Traffic source | google, newsletter | | utm_medium | Marketing medium | cpc, email, social | | utm_campaign | Campaign name | spring_sale | | utm_content | Differentiate versions | hero_cta | | utm_term | Paid search keywords | running+shoes | ### Naming Conventions - Lowercase everything - Use underscores or hyphens consistently - Be specific but concise: `blog_footer_cta`, not `cta1` - Document all UTMs in a spreadsheet --- ## Debugging and Validation ### Testing Tools | Tool | Use For | |------|---------| | GA4 DebugView | Real-time event monitoring | | GTM Preview Mode | Test triggers before publish | | Browser Extensions | Tag Assistant, dataLayer Inspector | ### Validation Checklist - [ ] Events firing on correct triggers - [ ] Property values populating correctly - [ ] No duplicate events - [ ] Works across browsers and mobile - [ ] Conversions recorded correctly - [ ] No PII leaking ### Common Issues | Issue | Check | |-------|-------| | Events not firing | Trigger config, GTM loaded | | Wrong values | Variable path, data layer structure | | Duplicate events | Multiple containers, trigger firing twice | --- ## Privacy and Compliance ### Considerations - Cookie consent required in EU/UK/CA - No PII in analytics properties - Data retention settings - User deletion capabilities ### Implementation - Use consent mode (wait for consent) - IP anonymization - Only collect what you need - Integrate with consent management platform --- ## Output Format ### Tracking Plan Document ```markdown # [Site/Product] Tracking Plan ## Overview - Tools: GA4, GTM - Last updated: [Date] ## Events | Event Name | Description | Properties | Trigger | |------------|-------------|------------|---------| | signup_completed | User completes signup | method, plan | Success page | ## Custom Dimensions | Name | Scope | Parameter | |------|-------|-----------| | user_type | User | user_type | ## Conversions | Conversion | Event | Counting | |------------|-------|----------| | Signup | signup_completed | Once per session | ``` --- ## Task-Specific Questions 1. What tools are you using (GA4, Mixpanel, etc.)? 2. What key actions do you want to track? 3. What decisions will this data inform? 4. Who implements - dev team or marketing? 5. Are there privacy/consent requirements? 6. What's already tracked? --- ## Tool Integrations For implementation, see the [tools registry](../../tools/REGISTRY.md). Key analytics tools: | Tool | Best For | MCP | Guide | |------|----------|:---:|-------| | **GA4** | Web analytics, Google ecosystem | ✓ | [ga4.md](../../tools/integrations/ga4.md) | | **Mixpanel** | Product analytics, event tracking | - | [mixpanel.md](../../tools/integrations/mixpanel.md) | | **Amplitude** | Product analytics, cohort analysis | - | [amplitude.md](../../tools/integrations/amplitude.md) | | **PostHog** | Open-source analytics, session replay | - | [posthog.md](../../tools/integrations/posthog.md) | | **Segment** | Customer data platform, routing | - | [segment.md](../../tools/integrations/segment.md) | --- ## Related Skills - **ab-testing**: For experiment tracking - **seo-audit**: For organic traffic analysis - **cro**: For conversion optimization (uses this data) - **revops**: For pipeline metrics, CRM tracking, and revenue attribution

Answer the model's full response$0.041 · 63.2s · 10.2k tok

### 1. Meta Totals, Blended CPA, and ROAS Under Each Source Both reports cover the closed observation window of **October 1–28**. #### Meta Ads Manager Data & Calculations * **Spend:** $\$1,500.00\text{ (Prospecting)} + \$900.00\text{ (Retargeting)} = \mathbf{\$2,400.00}$ * **Reported Conversions:** $38\text{ (Prospecting)} + 22\text{ (Retargeting)} = \mathbf{60\text{ purchases}}$ * **Reported Revenue:** $\$3,420.00\text{ (Prospecting)} + \$1,980.00\text{ (Retargeting)} = \mathbf{\$5,400.00}$ * **Blended CPA:** $$\text{CPA}_{\text{Meta}} = \frac{\text{Total Spend}}{\text{Reported Conversions}} = \frac{\$2,400.00}{60} = \mathbf{\$40.00\text{ per purchase}}$$ * **Blended ROAS:** $$\text{ROAS}_{\text{Meta}} = \frac{\text{Reported Revenue}}{\text{Total Spend}} = \frac{\$5,400.00}{2,400.00} = \mathbf{2.25\times\text{ (or }225\%)}$$ *(Campaign-level detail: Prospecting CPA = $\$1,500.00 / 38 = \$39.47$, ROAS = $\$3,420.00 / \$1,500.00 = 2.28\times$; Retargeting CPA = $\$900.00 / 22 = \$40.91$, ROAS = $\$1,980.00 / \$900.00 = 2.20\times$.)* --- #### GA4 Data & Calculations (Paid Social / Meta) * **Spend:** Carried over from actual platform spend = $\mathbf{\$2,400.00}$ * **Reported Purchases:** $\mathbf{36\text{ purchases}}$ (across 1,840 sessions) * **Reported Revenue:** $\mathbf{\$3,240.00}$ * **Blended CPA:** $$\text{CPA}_{\text{GA4}} = \frac{\text{Total Spend}}{\text{GA4 Purchases}} = \frac{\$2,400.00}{36} = \mathbf{\$66.67\text{ per purchase}}$$ * **Blended ROAS:** $$\text{ROAS}_{\text{GA4}} = \frac{\text{GA4 Revenue}}{\text{Total Spend}} = \frac{\$3,240.00}{\$2,400.00} = \mathbf{1.35\times\text{ (or }135\%)}$$ *(Note: Average Order Value [AOV] is exactly $\$90.00$ in every segment of both datasets: Meta Prospecting $\$3,420 / 38 = \$90.00$; Meta Retargeting $\$1,980 / 22 = \$90.00$; GA4 Paid Social $\$3,240 / 36 = \$90.00$; GA4 Organic Search $\$3,690 / 41 = \$90.00$; GA4 Direct $\$3,150 / 35 = \$90.00$.)* --- ### 2. Documented Causes of Disagreement The discrepancy between Meta ($60$ conversions / $\$5,400$) and GA4 ($36$ conversions / $\$3,240$) is driven strictly by the following four documented technical and structural differences: 1. **View-Through Conversion Counting:** * Meta operates on a `7-day click, 1-day view` window, meaning Meta claims credit for purchases where a user merely saw an ad within 24 hours without clicking. * GA4 never records view-through conversions under any configuration; it requires an actual web session. 2. **Attribution Logic by Construction (Single-Platform vs. Cross-Channel Last Click):** * Meta evaluates conversions strictly against interactions with its own ads, claiming any purchase preceded by a Meta click within 7 days or an impression within 1 day, regardless of whether other marketing touchpoints followed. * GA4 evaluates touchpoints across all channels and awards $100\%$ of the credit to the `last non-direct click` with a 30-day lookback. If a customer clicked a Meta ad, later returned via Organic Search, and purchased, GA4 assigns the purchase to Organic Search, while Meta simultaneously claims it. * This multi-platform overlap is confirmed by the tracking notes: summing self-reported conversions across all ad platforms (Meta, search, and affiliate) yields **149 claims**, whereas the store backend recorded only **118 actual completed purchases**. 3. **Oct 12–13 Auto-Tagging Outage:** * Auto-tagging broke on Oct 12–13. An estimated **8 Meta-referred sessions** landed on the site without UTM parameters and were misclassified as **Direct** in GA4 instead of Paid Social. 4. **Site-Wide GA4 Tracking Gap:** * The store platform (the absolute source of truth for completed orders) recorded **118 purchases**. GA4 recorded only **112 purchases site-wide**, leaving a 6-purchase gap unaccounted for across all channels in GA4. --- ### 3. What Can Honestly Be Concluded About Meta-Attributed Purchases * **The Defensible Range is 36 to 60 Purchases ($3,240 to $5,400 in Revenue):** * **Lower bound (36 purchases / $3,240 revenue / 1.35× ROAS):** GA4’s strict last non-direct click count. This is a conservative floor: it completely excludes view-through assists, ignores upper-funnel assists overwritten by search/direct, and loses whatever conversions resulted from the 8 untagged Meta sessions on Oct 12–13. * **Upper bound (60 purchases / $5,400 revenue / 2.25× ROAS):** Meta’s self-reported count under 7-day click / 1-day view. This is an absolute ceiling: it includes non-click impressions and duplicates purchases claimed by other platforms within the 149 cross-channel total. * **Do Not Average the Numbers:** * We cannot average $36$ and $60$ into $48$. They are not independent measurements of the same physical variable; they are counts under mutually incompatible operational definitions (a multi-touch/impression platform model vs. an exclusive last-click web analytics model). Averaging creates a synthetic metric with no tracking foundation. * **Attribution is Not Proof of Causation:** * Neither source measures **incremental lift** (purchases that would not have occurred without the ads). * In Meta Ads Manager, **$900 of the $2,400 spend (37.5%)** and **22 of the 60 conversions (36.7%)** were in `ARC-Retargeting`. Retargeted users already had high purchase intent and familiarity; attributing a sale to a retargeting click or impression does not prove the ad caused the order. --- ### 4. Recommendations and Next Steps #### A. Conclusions Robust to the Measurement Disagreement (True Under Both Sources) 1. **Meta Ads Are Cash-Positive on a Gross Revenue Basis:** Under both Meta’s ceiling ($2.25\times$ ROAS) and GA4’s strict floor ($1.35\times$ ROAS), total attributed revenue exceeds ad spend ($>\$1.00$ revenue per dollar spent). Meta is not producing zero return. 2. **Platform Self-Reporting Overstates Total Store Volume:** The ad platforms claimed $149$ conversions against only $118$ real store orders. Managing budgets solely by platform-reported conversions guarantees double-counting. 3. **Average Order Value is Uniform:** AOV is fixed at $\$90.00$ across every campaign and channel, meaning Meta’s efficiency differences are purely driven by conversion volume and acquisition cost, not by driving higher- or lower-basket sizes. #### B. Conclusions Dependent on Which Source You Believe * **Whether Meta "Printed Money":** * *If you believe Meta Ads Manager:* With a **$40.00 CPA** and **2.25× ROAS**, the channel is highly profitable and justifies immediate aggressive budget scaling. * *If you believe GA4:* With a **$66.67 CPA** and **1.35× ROAS**, the channel is marginally profitable or potentially unprofitable once product cost of goods sold (COGS) and operating margins are factored in. * **Prospecting Campaign Viability:** * Meta attributes 38 orders at a $\$39.47$ CPA to `ARC-Prospecting`. If GA4's 36 purchases heavily comprise Retargeting, Prospecting may look disastrous on a last-click basis. Conversely, if Prospecting introduced buyers who ultimately closed via Direct or Organic Search, GA4 obscures its top-of-funnel contribution. #### C. Recommendations Tied to Business Goals * **If the primary goal is Cash Preservation / Short-Term Margin Protection:** * Evaluate spend against **GA4's last-click floor ($1.35\times$ ROAS, $66.67 CPA)**. * *Trade-off:* Caps risk and prevents paying for overlapping/cannibalized demand, but risks throttling top-of-funnel prospecting that feeds Organic and Direct sales. * **If the primary goal is Top-Line Store Growth (scaling past 118 store orders):** * Allow Meta more latitude toward its **2.25× reported ROAS**, treating view-through and Prospecting assists as top-of-funnel drivers. * *Trade-off:* Accepts platform inflation and risks subsidizing retargeting impressions that would have converted organically. #### D. Immediate Tracking Fix & Next Measurement Step 1. **Immediate Operational Hygiene:** * Fix the auto-tagging infrastructure to prevent UTM stripping outages like Oct 12–13, ensuring paid social traffic is not misattributed to Direct. * Investigate the site-wide tracking leak causing GA4 to miss 6 completed store orders ($112$ GA4 purchases vs. $118$ store orders). 2. **Next Measurement Step (Incrementality Testing):** * Stop trying to resolve whether GA4 or Meta is "right" through observational reports. Because attribution models do not measure causality, run a **controlled incrementality test** (a Meta Conversion Lift / holdout study or a geo-lift test), particularly isolating `ARC-Retargeting`. Compare the resulting incremental lift directly against the **store platform order total (118)** to determine how many purchases Meta actually creates.

Score the judge's verdict, criterion by criterion9/9 criteria passed

Requirement	Verdict	Evidence
platform_oracle Meta platform totals are correct: $2,400 spend, 60 reported conversions, $5,400 reported revenue, blended CPA of $40.00, and ROAS of 2.25.	✓	Response computes spend $2,400.00, conversions 60, revenue $5,400.00, CPA $40.00, and ROAS 2.25× exactly as required.
ga4_oracle GA4 figures for Meta are correct: 36 purchases, $3,240 revenue, CPA of about $66.67, and ROAS of 1.35 against the same $2,400 spend.	✓	Response computes GA4 Paid Social figures as 36 purchases, $3,240.00 revenue, CPA $66.67, and ROAS 1.35× against $2,400 spend.
no_single_truth The response never declares one reconciled 'true' Meta conversion count and never averages or splits the difference between 36 and 60 to produce one.	✓	The response explicitly states 'Do Not Average the Numbers' and never proposes a single reconciled conversion count.
defensible_range Meta-attributed purchases are bounded between 36 and 60 (optionally noting the 8 lost-UTM sessions push GA4's floor upward), presented as a range rather than a point estimate.	✓	It states 'The Defensible Range is 36 to 60 Purchases' and notes the GA4 floor excludes the 8 untagged Meta sessions, pushing the true floor higher.
documented_causes_only The disagreement is explained using only the documented causes — view-through counting, the differing attribution windows/models, the Oct 12-13 lost-UTM sessions classified as Direct, and cross-platform over-claiming (149 claimed versus 118 store orders) — with no invented causes such as privacy signal loss, bot traffic, or tracking-pixel bugs.	✓	Causes cited are view-through counting, differing attribution logic, the Oct 12-13 UTM outage, and the 149-vs-118 over-claiming, all sourced from the notes with no invented causes like bots or pixel bugs.
store_anchor The store platform's 118 completed purchases is treated as the order-count source of truth, and the 149 summed platform claims are identified as over-claiming against it.	✓	The response calls the store platform 'the absolute source of truth for completed orders' at 118 and frames the 149 platform claims as over-claiming against it.
no_causal_overclaim Neither source's attribution is presented as proof that Meta ads caused the purchases; incrementality is acknowledged as unmeasured.	✓	A dedicated subsection states 'Attribution is Not Proof of Causation' and 'Neither source measures incremental lift.'
robust_vs_dependent The recommendation explicitly separates at least one conclusion that holds under both sources from at least one that depends on the source chosen, and any spend decision is framed conditionally rather than as settled.	✓	Section A lists conclusions 'Robust to the Measurement Disagreement' and Section B lists conclusions 'Dependent on Which Source You Believe,' with conditional framing in Section C.
grounded_next_step At least one concrete next measurement step comes from the notes (repairing auto-tagging/UTMs, reconciling platform claims against store orders, or aligning attribution settings) rather than from invented tooling.	✓	The response recommends 'Fix the auto-tagging infrastructure to prevent UTM stripping outages,' a step grounded directly in the tracking notes.