> For the complete documentation index, see [llms.txt](https://docs.neonblue.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.neonblue.ai/data/holdouts-gates-and-experiment-structure.md).

# Holdouts, Gates, and Experiment Structure

#### 1. Purpose of this page

This page describes how Neon Blue structures experiments using:

* **Global holdout** groups
* **Gates** at different levels
* **Contextual bandits** at individual touchpoints

The goal is to give a clear mental model for how the system routes traffic and measures lift.

***

#### 2. Key concepts

**Holdout**\
A group of users that does not receive Neon Blue-optimized experiences. This group serves as a long-term reference to measure overall program impact. 

**Gate**\
A decision point in the user journey. When a user crosses a gate, Neon Blue determines:

* Whether the user should be included in a program or flow
* Whether the user receives the baseline experience or the Neon Blue experience

**Experiment**\
A test at a specific touchpoint (for example, an email subject line or on-site banner) where multiple creative variants are compared.

**Variant**\
One possible experience within an experiment (for example, a specific subject line or banner design). One variant is designated as the **control** (typically the current or legacy experience).

**Contextual bandit**\
An always-on experimentation algorithm that:

* Allocates more traffic to better-performing variants
* Continues to explore alternatives
* Prunes underperforming variants over time

***

#### 3. Global program structure

Neon Blue uses a layered structure:

1. **Global holdout**
   * A fixed percentage of users (commonly around 10%) is permanently held out from Neon Blue. 
   * These users continue to receive existing experiences only.
   * This makes it possible to measure long-term program lift.
2. **Global rollout gate**
   * A gate that determines whether a user is eligible for Neon Blue at all.
   * Within the non-holdout population, the global gate divides users into:
     * **Baseline** group (original non-Neon Blue experience)
     * **Neon Blue** group (eligible for Neon Blue-optimized flows)
3. **Flow-level gates**
   * Gates at the start of each flow (for example, Welcome, Abandoned Cart, Subscription Reminder).
   * Control ramp-up and allow flow-specific effect tests.
4. **Touchpoint-level bandits**
   * Each individual email or on-site placement is implemented as its own contextual bandit.
   * One arm of each bandit is the original control variant (for example, the existing email template or banner).

This structure supports both:

* Program-level questions (“Is Neon Blue beneficial overall?”)
* Flow-and-touchpoint-level questions (“Which variant of this email works best?”)

***

#### 4. Global holdout and rollout

The global holdout is designed to be:

* **Persistent** – users who are in global holdout remain there for the program’s life, ensuring a stable baseline.
* **Low-friction** – the holdout is configured once and then reused across flows. 

Within the non-holdout population, Neon Blue further divides users into:

* **Pending rollout** – users who are eligible but not yet part of the “in experiment” comparison set.
* **In experiment** – users who are actively included in the baseline vs Neon Blue comparison.
* **Sticky assignments** – once a user is assigned to baseline or Neon Blue within a given context, that assignment remains stable for analysis. 

Only the **in-experiment** baseline vs Neon Blue groups are used for lift calculations. Pending rollout users are excluded from effect measurement until their status is resolved.

***

#### 5. Flow-level gates

Flow-level gates are placed at the entry point for each lifecycle flow, for example:

* `welcome_flow_gate`
* `abandoned_cart_gate`
* `subscription_reminder_gate`

Each gate:

* Determines whether the user enters the flow at all
* Determines whether the user receives the baseline flow experience or the Neon Blue flow experience
* Serves as the anchor for flow-level measurement (“orders per gate entry,” etc.)

Gates are configurable; they can be placed wherever in the journey flow boundaries make the most sense.

***

#### 6. Touchpoint-level contextual bandits

Within each flow, there are one or more **touchpoints**:

* Individual emails in the flow
* On-site impressions (for example, homepage banners)
* Other personalized messages

Each touchpoint has its own contextual bandit:

* One arm is the existing control template or banner.
* Other arms are Neon Blue-generated variants.
* Traffic allocation evolves over time:
  * Initially, more traffic is placed on control and exploration variants.
  * As evidence accumulates, stronger variants receive a larger share of traffic.
  * Underperforming variants are pruned to avoid carrying too many low-value options.

The presence of the original control variant as one of the bandit arms ensures:

* Direct comparison between “current” and “Neon Blue” experiences within the same bandit
* Protection against regressions when new variants are introduced

***

#### 7. Summary: ordering of decisions

For any given user and touchpoint, routing follows this order:

1. **Global holdout check**
   * If user is in global holdout → always baseline / existing experience.
2. **Global rollout gate**
   * If user is not in global program → baseline experience.
   * If user is in global program → proceed to flow gate.
3. **Flow gate**
   * If user is not passed into the flow → no Neon Blue experience for that flow.
   * If user is passed → flow-specific baseline vs Neon Blue routing.
4. **Touchpoint bandit**
   * For eligible Neon Blue traffic within the flow, the bandit selects a creative variant.
   * Control template is included as one variant in the bandit.

This structure supports always-on experimentation with stable baselines and clear comparisons at multiple levels.