Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Attribution & Windows: Optimization vs Reporting

Purpose of this page

This page describes how Neon Blue:

  • Attributes outcomes to experiments

  • Uses time windows for measurement

  • Separates metrics used for optimization from those used for business reporting

It is intended for teams that want to understand “how things are counted” and how Neon Blue fits with existing attribution and reporting practices.

2. Two layers of measurement

Neon Blue distinguishes between two layers:

  1. Optimization metrics

    • Used by machine learning models to decide which variant to show next

    • Typically based on upper-funnel or mid-funnel events (for example, clicks)

    • Designed to be simple, stable, and high-frequency

  2. Reporting metrics

    • Used by humans to evaluate business impact

    • Based on downstream outcomes such as orders, subscription revenue, and cancellations

    • Designed to align with how the business already measures performance

This separation allows Neon Blue to learn quickly from abundant signals, while still reporting in terms that match internal dashboards and financial views.

3. Optimization metrics

Optimization metrics are built from the event model described on the previous page. General principles:

  • Use a single primary positive event (for example, click, micro-conversion, or key subscription action).

  • Include negative events as guardrails (unsubscribes, complaints, cancellations).

  • Use short, intuitive windows so that feedback is timely.

Examples:

  • For a welcome flow:

    • Positive: user clicked any welcome email within 3 days of the first welcome send

    • Negative: user unsubscribed within 7 days

  • For an abandoned cart flow:

    • Positive: user placed an order within 3 days of the abandoned cart email

    • Negative: user unsubscribed within 7 days

These metrics form the objective functions for Neon Blue’s contextual bandit and optimization systems.

4. Reporting metrics

Reporting metrics are designed to answer questions such as:

  • How much incremental revenue did a flow produce?

  • How did Neon Blue affect subscription retention or cancellation rates?

  • What was the change in orders per user over a defined time period?

Reporting metrics may include:

  • Orders per user in a given window

  • Revenue per user in a given window

  • Subscription retention rate, churn rate, or net subscription revenue

  • Attachment of specific products or categories

These metrics are usually based on the orders table (and sometimes additional subscription tables), often combined with assignments and gates from Neon Blue’s outbound share.

5. Gate-based attribution

Instead of attributing a single order to a single email or click, Neon Blue uses a gate-based approach as the primary attribution mechanism.

Conceptually:

  • A gate represents an entry point into a program or flow.

  • When a user crosses that gate, Neon Blue records an assignment:

    • For example, control vs Neon Blue

  • Outcomes are then measured in windows following the gate entry.

Examples of gates:

  • global_program_gate – decides whether a user is in the global Neon Blue program or global holdout.

  • welcome_flow_gate – decides whether a user receives the Neon Blue version of the welcome flow vs the baseline version.

  • abandoned_cart_gate – decides which experience a user receives after abandoning a cart.

Gate-level measurement asks:

For users who entered this gate and received the Neon Blue experience, how did outcomes differ from users who entered the same gate and received the baseline experience?

This approach is:

  • Robust to complex journeys where users see multiple touchpoints

  • Compatible with both email and on-site experiences

  • Easier to align with business questions such as “what is the effect of this program over the next 7–30 days?”

6. Windows for optimization and reporting

Time windows are applied at both the optimization and reporting layers.

6.1 Optimization windows

Optimization windows are short and tied to the flow’s dynamics. Typical choices:

  • Click windows of 1–3 days after an email send

  • Unsubscribe windows of 7 days after a send

  • For on-site experiences, windows tied to the expected decision period (for example, same session or a few days)

These windows ensure that models receive feedback quickly and avoid learning from very delayed or unrelated events.

6.2 Reporting windows

Reporting windows are often longer and tailored to the business context:

  • 7-day or 14-day windows for order-based flows (for example, abandoned cart)

  • 30-day windows for subscription flows or higher-consideration purchases

  • Longer windows for lifetime-value style analysis if sufficient history is available

These windows are applied from:

  • Gate entry (for program-level or flow-level measurement), and/or

  • Specific experiment assignments (for touchpoint-level analysis when desired)

7. Integration with existing attribution models

Most organizations already have attribution logic in place, for example:

  • GA4-based first-touch or data-driven attribution

  • Last non-direct click logic based on UTM parameters

  • Higher-level channel attribution from marketing mix modeling

Neon Blue is designed to fit into these existing views rather than replace them.

Typical patterns:

  • Use existing attribution tables (for example, a user-level attribution table) as modeling features, so Neon Blue can account for acquisition source or channel type.

  • Use existing attribution definitions when presenting aggregated results by channel or campaign.

  • For core experimental measurement, rely on gate-based attribution and stable windows, since these are controlled by the Neon Blue integration.

If desired, reporting can be segmented by acquisition model, for example:

  • Neon Blue lift for users acquired via Paid Social vs Organic Search.

8. Subscriptions and long-lag outcomes

For subscription businesses, orders may occur long after a messaging event, and outcomes such as churn or downgrade are critical.

Neon Blue’s approach supports this by:

  • Using flows and gates that are specific to subscription lifecycle moments (for example, “pre-renewal reminder gate,” “winback gate”).

  • Defining reporting metrics that focus on:

    • Subscription kept vs cancelled within a defined window

    • Net subscription revenue over a longer period (for example, 30 or 60 days)

  • Continuing to track revenue and subscription events long after the initial messaging event, while still using finite windows for standard reporting.

For optimization, subscription flows may use earlier signals such as:

  • subscription_management_opened

  • subscription_addon_added

  • subscription_kept versus subscription_skipped

These upstream signals provide quicker feedback to models, while the longer-lag subscription outcomes are used for evaluation.

9. Example: measuring a welcome flow

Assume a gate named welcome_flow_gate and a corresponding experiment that determines whether a user receives the Neon Blue version or the baseline version of the welcome flow.

A typical configuration:

  • Optimization metric

    • Positive: email_click on any welcome email within 3 days of gate entry

    • Negative: email_unsubscribe within 7 days of gate entry

  • Reporting metrics

    • order_placed within 14 days of gate entry

    • Revenue per user in 30 days after gate entry

With this setup, the organization can answer:

  • Optimization:

    • Are users in the Neon Blue variant clicking more than users in the baseline variant?

    • Is the unsubscribe rate remaining within acceptable bounds?

  • Reporting:

    • Do users in the Neon Blue variant place more orders or generate more revenue per user over 14–30 days than users in the baseline variant?

Both analyses use the same gate assignments, windows, and underlying events, but focus on different metrics.

10. FAQ

Is attribution based on last-click, first-touch, or something else? Neon Blue does not enforce a single touchpoint-based attribution model. The primary method is gate-based attribution: outcomes are measured relative to gate entries for control and treatment groups, using defined time windows. Existing touchpoint-based models (last-click, first-touch, GA4 models) can still be used for segmentation and supplementary analysis.

Can the organization change reporting windows later? Yes. As long as raw assignments and events are retained, reporting windows and rollup definitions can be changed and recomputed. Optimization windows should remain stable for a given test, but reporting definitions are flexible.

Does Neon Blue optimize directly for revenue? In most configurations, optimization uses upper-funnel or mid-funnel events (such as clicks or micro-conversions) combined with guardrails. Revenue and profit impact are measured in reporting. Direct revenue optimization is possible for programs with sufficient volume and stable value definitions.

How does this relate to internal experimentation platforms? Neon Blue’s outbound Snowflake share provides gate and experiment assignments that can be joined to internal events and orders. This allows organizations to compute metrics using the same logic as existing experimentation platforms, while Neon Blue focuses on generating and selecting creative variants.

Last updated