FAQ for Data & Engineering Teams
1. Purpose of this page
This page summarizes common questions that arise when integrating Neon Blue with a data warehouse, experimentation stack, and on-site or messaging systems.
Each answer links conceptually to the more detailed documentation pages.
2. Identity and data model
Q: What should be used as user_id?
A stable, long-lived identifier that exists across core tables. For ecommerce, this is often the platform’s customer ID. The same user_id should be used in:
user_profileseventsordersNeon Blue assignment tables
Q: How should device IDs and analytics IDs be handled?
Store them in a separate mapping table (for example, user_device_id_map) that associates device_id with user_id when known. Pageviews and analytics data can be keyed by device_id and joined to users via this mapping.
3. Events and metrics
Q: Do events need numeric values for Neon Blue to work? No. Binary events (“happened / did not happen”) are sufficient. Value fields can be added later for value-based optimization if needed.
Q: Can different flows use different goal events? Yes. Each flow can target a different optimization metric (for example, clicks for welcome, orders for abandoned cart), as long as the metrics are defined in terms of the event stream and configured consistently.
Q: How are negative events such as unsubscribes and cancellations handled? Negative events are included as guardrails in optimization metrics and in reporting. For subscription businesses, cancellations and downgrades are particularly important and should be surfaced as explicit events.
4. Holdouts, gates, and experiments
Q: Why is there a global holdout? A persistent global holdout (for example, 10% of users) provides a long-term baseline for measuring overall program impact. It is separate from flow-level or experiment-level gates.
Q: Can gates be placed anywhere in the journey? Yes. Gates can be configured at any logical boundary (for example, entry to a flow, eligibility for a specific on-site experience). Flow-level gates control who enters each flow and whether Neon Blue is applied.
Q: How do contextual bandits relate to A/B tests? Each touchpoint uses a contextual bandit that includes the original control variant and one or more Neon Blue-generated variants. The bandit allocates traffic adaptively and prunes underperforming variants over time, while A/B-style gating ensures clear treatment vs control groups.
5. On-site integration
Q: Does the site need to call Neon Blue on every pageview? On surfaces where Neon Blue participates (for example, a homepage banner), the site should:
Call a gate endpoint to determine whether Neon Blue applies.
If yes, call a variant endpoint to retrieve the creative for that user and context.
Q: How should content blocking be handled? Use a custom subdomain on the brand’s domain with a CNAME pointing to Neon Blue, and configure SSL appropriately. This reduces the likelihood of blockers treating Neon Blue calls as third-party tracking.
6. Tags and personas
Q: What does “tag” mean in Neon Blue documentation? Tags may refer to:
Experiment-related tags: gate IDs, experiment IDs, variant IDs in assignment tables.
Persona tags: inferred user preference segments.
The outbound Snowflake share includes both assignment tables and persona assignment tables.
Q: How reliable are persona tags? Persona assignments are probabilistic. They become more accurate as users participate in more experiments. Confidence scores indicate how strong the evidence is for a particular persona.
7. Reporting and validation
Q: How does Neon Blue attribute impact? Neon Blue uses gate-based attribution as the primary mechanism: outcomes are measured in windows following gate entry for control and Neon Blue groups. This avoids brittle touchpoint-level attribution and focuses on controllable entry points.
Q: Can existing experimentation platforms be used to analyze Neon Blue experiments? Yes. Assignment tables from Neon Blue can be used as exposure logs in a warehouse-native experimentation platform. Metrics can then be computed using the same event and order tables used for other experiments.
Q: What is the recommended process for validating Neon Blue metrics? Common steps:
Validate that user counts and assignments match expectations.
Join assignments to events and orders and compute metrics independently.
Compare those metrics to Neon Blue-reported metrics.
Once aligned, embed Neon Blue data into standard dashboards and reports.
8. Schema changes and evolution
Q: What happens if internal schemas change? To avoid breaking the integration:
Prefer additive changes (adding columns) to breaking changes.
Expose Neon Blue-facing views that remain stable even if underlying tables evolve.
If a schema change affects shared tables, coordinate with Neon Blue so assignment and reporting pipelines can be updated accordingly.
Last updated