Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Holdouts, Gates, and Experiment Structure

1. Purpose of this page

This page describes how Neon Blue structures experiments using:

  • Global holdout groups

  • Gates at different levels

  • Contextual bandits at individual touchpoints

The goal is to give a clear mental model for how the system routes traffic and measures lift.

2. Key concepts

Holdout A group of users that does not receive Neon Blue-optimized experiences. This group serves as a long-term reference to measure overall program impact.

Gate A decision point in the user journey. When a user crosses a gate, Neon Blue determines:

  • Whether the user should be included in a program or flow

  • Whether the user receives the baseline experience or the Neon Blue experience

Experiment A test at a specific touchpoint (for example, an email subject line or on-site banner) where multiple creative variants are compared.

Variant One possible experience within an experiment (for example, a specific subject line or banner design). One variant is designated as the control (typically the current or legacy experience).

Contextual bandit An always-on experimentation algorithm that:

  • Allocates more traffic to better-performing variants

  • Continues to explore alternatives

  • Prunes underperforming variants over time

3. Global program structure

Neon Blue uses a layered structure:

  1. Global holdout

    • A fixed percentage of users (commonly around 10%) is permanently held out from Neon Blue.

    • These users continue to receive existing experiences only.

    • This makes it possible to measure long-term program lift.

  2. Global rollout gate

    • A gate that determines whether a user is eligible for Neon Blue at all.

    • Within the non-holdout population, the global gate divides users into:

      • Baseline group (original non-Neon Blue experience)

      • Neon Blue group (eligible for Neon Blue-optimized flows)

  3. Flow-level gates

    • Gates at the start of each flow (for example, Welcome, Abandoned Cart, Subscription Reminder).

    • Control ramp-up and allow flow-specific effect tests.

  4. Touchpoint-level bandits

    • Each individual email or on-site placement is implemented as its own contextual bandit.

    • One arm of each bandit is the original control variant (for example, the existing email template or banner).

This structure supports both:

  • Program-level questions (“Is Neon Blue beneficial overall?”)

  • Flow-and-touchpoint-level questions (“Which variant of this email works best?”)

4. Global holdout and rollout

The global holdout is designed to be:

  • Persistent – users who are in global holdout remain there for the program’s life, ensuring a stable baseline.

  • Low-friction – the holdout is configured once and then reused across flows.

Within the non-holdout population, Neon Blue further divides users into:

  • Pending rollout – users who are eligible but not yet part of the “in experiment” comparison set.

  • In experiment – users who are actively included in the baseline vs Neon Blue comparison.

  • Sticky assignments – once a user is assigned to baseline or Neon Blue within a given context, that assignment remains stable for analysis.

Only the in-experiment baseline vs Neon Blue groups are used for lift calculations. Pending rollout users are excluded from effect measurement until their status is resolved.

5. Flow-level gates

Flow-level gates are placed at the entry point for each lifecycle flow, for example:

  • welcome_flow_gate

  • abandoned_cart_gate

  • subscription_reminder_gate

Each gate:

  • Determines whether the user enters the flow at all

  • Determines whether the user receives the baseline flow experience or the Neon Blue flow experience

  • Serves as the anchor for flow-level measurement (“orders per gate entry,” etc.)

Gates are configurable; they can be placed wherever in the journey flow boundaries make the most sense.

6. Touchpoint-level contextual bandits

Within each flow, there are one or more touchpoints:

  • Individual emails in the flow

  • On-site impressions (for example, homepage banners)

  • Other personalized messages

Each touchpoint has its own contextual bandit:

  • One arm is the existing control template or banner.

  • Other arms are Neon Blue-generated variants.

  • Traffic allocation evolves over time:

    • Initially, more traffic is placed on control and exploration variants.

    • As evidence accumulates, stronger variants receive a larger share of traffic.

    • Underperforming variants are pruned to avoid carrying too many low-value options.

The presence of the original control variant as one of the bandit arms ensures:

  • Direct comparison between “current” and “Neon Blue” experiences within the same bandit

  • Protection against regressions when new variants are introduced

7. Summary: ordering of decisions

For any given user and touchpoint, routing follows this order:

  1. Global holdout check

    • If user is in global holdout → always baseline / existing experience.

  2. Global rollout gate

    • If user is not in global program → baseline experience.

    • If user is in global program → proceed to flow gate.

  3. Flow gate

    • If user is not passed into the flow → no Neon Blue experience for that flow.

    • If user is passed → flow-specific baseline vs Neon Blue routing.

  4. Touchpoint bandit

    • For eligible Neon Blue traffic within the flow, the bandit selects a creative variant.

    • Control template is included as one variant in the bandit.

This structure supports always-on experimentation with stable baselines and clear comparisons at multiple levels.

Last updated