Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Reporting, Validation, and Example Analyses

1. Purpose of this page

This page describes how to:

  • Interpret Neon Blue’s reporting

  • Reproduce results in a data warehouse or experimentation platform

  • Run common analyses using the inbound and outbound Snowflake tables

It is aimed at teams that want to build trust in the integration and embed Neon Blue data into existing reporting.

2. Reporting sources

Neon Blue reporting is based on three main inputs:

  1. Inbound data from the customer

    • User profiles, events, orders, and optional attribution tables.

  2. Neon Blue assignments

    • Gate and experiment assignments indicating which users saw which experiences.

  3. Metric definitions

    • Event-based metrics defined in terms of the inbound event stream (for example, clicked within 3 days, ordered within 7 days).

Reporting combines these components to produce lift estimates and segment-level insights.

3. Program-level reporting (global holdout)

At the program level, Neon Blue uses the global holdout to measure overall impact.

Typical questions:

  • What is the difference in orders per user between the global holdout group and users exposed to Neon Blue?

  • How much incremental revenue has the program generated over a quarter?

Analysis pattern:

  1. Use the outbound gate_assignments table to identify the global program gate and holdout group.

  2. Join to orders in a fixed window (for example, 30 days).

  3. Compute per-user metrics for holdout vs program groups.

This approach provides a high-level view of Neon Blue’s contribution.

4. Flow-level reporting (flow gates)

For each flow gate (for example, welcome_flow_gate, abandoned_cart_gate):

  1. Identify entries into the gate via gate_assignments.

  2. Use gate entry time as the start of the reporting window.

  3. Measure outcomes such as:

    • Orders within 7 or 14 days

    • Revenue per gate entry

    • Subscription events (kept vs cancelled)

This gate-based method avoids fragile touchpoint-level attribution and focuses on:

“Given a user entered this flow gate and was allocated to baseline or Neon Blue, what happened afterward?”

5. Variant-level reporting (experiments)

Variant-level analyses answer questions like:

  • Which subject line variant performs best on click rate?

  • Which banner variant leads to the most orders per exposed user?

Pattern:

  1. Use experiment_assignments to find which users saw which variant.

  2. Join to events and orders in relevant windows.

  3. Compute metrics such as:

    • Click-through rate (CTR)

    • Conversion rate

    • Revenue per user

Neon Blue’s own dashboards are computed using the same assignment and event data, so warehouse analyses should match when the same definitions are applied.

6. Validation workflow with a warehouse-native experimentation tool

Many organizations maintain an internal experimentation platform or use a third-party tool backed by the data warehouse.

A common validation pattern:

  1. Configure Neon Blue integration and let initial experiments run.

  2. Expose gate_assignments and experiment_assignments through Neon Blue’s outbound Snowflake share.

  3. Configure “analysis-only” experiments in the existing experimentation tool, using:

    • The assignment tables as exposure logs

    • Event and order tables as outcome sources

    • Metric definitions that mirror Neon Blue’s metrics

  4. Compare Neon Blue’s reported metrics to the internal tool’s metrics.

  5. Once consistent, embed Neon Blue exposure and outcome metrics into standard reports.

This process gives data teams full transparency into how results are computed.

7. Example analyses

7.1 Flow gate: orders per gate entry (7-day window)

Goal: Compare orders per gate entry for a flow gate between baseline and Neon Blue.

Steps:

  1. Filter gate_assignments to the desired flow gate.

  2. Join to orders where order_placed_at falls within 7 days of assigned_at.

  3. Aggregate orders and compute per-user metrics by variant key.

This pattern can be extended to:

  • Longer windows (14, 30 days)

  • Additional dimensions (persona, acquisition channel, etc.)

7.2 Experiment: click-through rate by variant

Goal: Compare CTR across variants in a subject line experiment.

Steps:

  1. Filter experiment_assignments to the specific experiment.

  2. Join to events where:

    • event_name = 'email_click'

    • occurred_at is within a configured window of assigned_at

  3. Count unique users with at least one click per variant and divide by unique users assigned to each variant.

This analysis replicates Neon Blue’s own variant-level metrics inside the customer’s warehouse.

7.3 Persona-based performance

Goal: Understand how performance varies by persona.

Steps:

  1. Join persona_assignments to gate_assignments or experiment_assignments on user_id.

  2. Join to events and orders.

  3. Compute metrics such as:

    • Orders per user by persona

    • CTR or conversion rate per persona

    • Incremental lift per persona group

This helps identify which personas benefit most from Neon Blue’s optimizations.

8. Recommended validation checks

Before treating Neon Blue metrics as production-ready, teams often:

  • Validate count parity:

    • Confirm that user counts per gate or experiment match expectations.

  • Validate event joins:

    • Confirm that clicks and orders join correctly to assignments.

  • Validate metric parity:

    • Reproduce a sample of Neon Blue-reported metrics in the warehouse and check for alignment.

Once these checks pass, Neon Blue data can be incorporated into standard BI and experimentation workflows.

Last updated