# Retentioneering

Source: https://retentioneering.com/docs

**See why your metrics moved — not just that they did.**

Retentioneering is an open-source Python library for user behavior analysis. It turns raw event logs into interactive maps of how users actually move through your product: the paths they take, the loops they get stuck in, the step where they silently leave. All inside your notebook, on your own data, in a few lines of code.

```python
import retentioneering as rete

stream = rete.Eventstream(df)
stream.transition_graph()
```

## Dashboards tell you *what*. Retentioneering shows you *why*.

Amplitude, Mixpanel, and BI dashboards are great at reporting that conversion dropped or retention dipped. They are much worse at answering the question that follows: *what did users actually do differently?* Aggregate metrics flatten thousands of individual journeys into a single number — and the explanation lives in the journeys.

Retentioneering works at the level of behavior. Instead of a predefined funnel, it reconstructs the real paths users take — including the detours, back-and-forth loops, and dead ends no one designed — and gives you interactive tools to explore, segment, and compare them.

## What you can do

### Map real user journeys

Visualize how users actually navigate your product with an interactive [Transition Graph](https://retentioneering.com/docs/widgets/transition-graph), or unfold paths step by step with [Step Sankey](https://retentioneering.com/docs/widgets/step-sankey) and [Step Matrix](https://retentioneering.com/docs/widgets/step-matrix). Discover which routes really lead to purchase, where users circle back, and which screen is most often the last one before they leave — without deciding in advance what the "correct" path looks like.

### Find where journeys break

Build [funnels](https://retentioneering.com/docs/widgets/funnel) when you know the steps — and go beyond them when you don't. Anchor the analysis on any event, look at what happens right before drop-off, and trace an abstract "conversion fell" down to the concrete screen where the scenario falls apart.

### Compare groups, explain the difference

Any split becomes a [segment](https://retentioneering.com/docs/segments) — an A/B arm, an acquisition channel, a behavioral cluster discovered by [Cluster Analysis](https://retentioneering.com/docs/widgets/cluster-analysis) — and every core widget can render the behavioral *difference* between two groups. Two examples of what that unlocks:

- **Explain an anomaly.** Segment events into "anomalous period vs. normal", diff the two groups, and see at the level of raw events what actually caused the dip.
- **Open up the funnel.** Path patterns like `add_to_cart->.*->purchase` show what really happens *between* funnel levels, and a funnel segment — each path labelled with the last step it reached — reveals where exactly the users who never finished got lost.

More recipes like these live on the [Recipes](https://retentioneering.com/docs/recipes) page.

## Built for real data

- **Fast on millions of rows.** The engine is backed by [DuckDB](https://duckdb.org), so exploration stays interactive on production-scale event logs — on your laptop.
- **Your data, your rules.** Load events from pandas, CSV, or Parquet — whatever comes out of your warehouse. Custom event grouping, sessionization ([`split_sessions`](https://retentioneering.com/docs/data-processors)), and cleanup are a chained method call away, with no SaaS interface limits.
- **Notebook-native, shareable anywhere.** Widgets run in Jupyter, VS Code, and Google Colab, and every one of them exports to a standalone interactive HTML file you can drop into a message to a PM — no Python required to open it.
- **AI-ready.** The built-in [MCP server](https://retentioneering.com/docs/mcp-server) lets an LLM agent explore your eventstream with the full toolkit — ask "why did retention dip last week?" in plain language. [Agent Skills](https://retentioneering.com/docs/agent-skills) teach coding agents like Claude Code or Codex how to run and contribute analyses in this codebase.

## Who is it for

- **Product analysts** who hit the ceiling of funnel dashboards and need to see behavior, not just rates.
- **Quantitative UX researchers** connecting the "something feels broken here" intuition to quantitative evidence in event logs.
- **Growth and lifecycle teams** looking for what separates retained users from churned ones — and who to target next.
- **Experimenters** hunting for the segment where the A/B effect hides when the aggregate metric reads flat.
- **Data scientists** extracting behavioral features and interpretable patterns from raw trajectories for churn and LTV models.

## Where to start

Go from `pip install` to your first interactive transition graph in five minutes with the [Quick Start](https://retentioneering.com/docs/quick-start) — a bundled e-commerce dataset is included, so you can explore before touching your own data.

Then read [Path Analysis](https://retentioneering.com/docs/path-analysis). It is the shortest path to using the library well: every visualization here is a different aggregated representation of the same trajectories, and knowing which detail each one drops is what tells you when to trust it. After that, [Eventstream](https://retentioneering.com/docs/eventstream) covers connecting your own logs and [Widgets](https://retentioneering.com/docs/widgets) gives the full tour.

Coming from retentioneering 3.x? Version 5.0 is a ground-up rewrite and most 3.x code will not run unchanged — [Migrating from 3.x](https://retentioneering.com/docs/migration-from-3x) maps the old API onto the new one.

---

# Quick Start

Source: https://retentioneering.com/docs/quick-start

This guide walks you through a complete example — from installation to your first interactive visualization — in under five minutes.

## 1. Install

```bash
pip install retentioneering
```

## 2. Load your data

Create an [Eventstream](https://retentioneering.com/docs/eventstream) from a pandas DataFrame. By default, Eventstream expects columns named `user_id`, `event`, and `timestamp`. If your data uses different column names, pass a [schema](https://retentioneering.com/docs/eventstream#schema).

```python
import pandas as pd
import retentioneering as rete

df = pd.read_csv("events.csv")
stream = rete.Eventstream(df)
```

No CSV yet? Use the built-in sample dataset to follow along:

```python
import retentioneering as rete

stream = rete.datasets.load_ecom()
```

## 3. Explore with a widget

Open an interactive [Transition Graph](https://retentioneering.com/docs/widgets/transition-graph) — no arguments needed. Configure everything in the sidebar.

```python
stream.transition_graph()
```

Compare two user segments side by side:

```python
stream.transition_graph(diff=["platform", "mobile", "desktop"])
```

Explore user paths step by step around important events or drop-off points with [Step Sankey](https://retentioneering.com/docs/widgets/step-sankey):

```python
stream.step_sankey(path_pattern="purchase")
```

or its equivalent [Step Matrix](https://retentioneering.com/docs/widgets/step-matrix):

```python
stream.step_matrix(path_pattern="purchase")
```

## 4. Prepare your data

Use [data processors](https://retentioneering.com/docs/data-processors) to clean and shape the eventstream before visualizing:

```python
stream = (
    rete.datasets.load_ecom()
    .filter_events(drop={"event": ["checkout_bug"]})
    .rename_events({"wishlist_add": "add_to_wishlist"})
)

stream.step_sankey()
```

Every processor returns a new eventstream, so they chain into a pipeline and never modify what they were called on. Note that they also validate event names against your data: dropping or renaming an event that isn't there raises an error instead of silently doing nothing.

## Next steps

- [Path Analysis](https://retentioneering.com/docs/path-analysis) — the one page that explains what all of these widgets actually compute. Read this before the rest.
- [Eventstream](https://retentioneering.com/docs/eventstream) — schema configuration and data format
- [Widgets](https://retentioneering.com/docs/widgets) — all available visualizations and how they work
- [Data Processors](https://retentioneering.com/docs/data-processors) — full list of transformations

---

# Installation

Source: https://retentioneering.com/docs/installation

## Requirements

- Python 3.10 or later
- Jupyter, VS Code, or Google Colab

## Install the package

```bash
pip install retentioneering
```

## Environment setup

### VS Code

Works out of the box — no additional setup required.

### JupyterLab and JupyterLab Desktop

Retentioneering widgets are built on [anywidget](https://anywidget.dev). For widgets to render correctly in JupyterLab, `anywidget` must be installed in the same Python environment that runs JupyterLab itself — not just in the kernel.

If you are using **JupyterLab Desktop**, open the built-in Extension Manager and make sure `anywidget` is listed there. If it is not, install it into the JupyterLab Desktop environment:

```bash
pip install anywidget
```

Then restart JupyterLab.

### Google Colab

Install the package at the top of your notebook:

```bash
!pip install retentioneering
```

## Verify the installation

```python
import retentioneering
print(retentioneering.__version__)
```

---

# Path Analysis

Source: https://retentioneering.com/docs/path-analysis

A real event log contains from a hundred to a million paths. Although you may want to look at individual paths, mostly you look at them as a whole. Every visualization in this library is an aggregated representation of the same trajectories that keeps some aspects of each path. Retentioneering builds three such representations: by step, by transition, by milestone. This page demonstrates each of them and explains how to choose the right one for your question.

## A path is the unit of analysis

A raw event log has one row per event. The journey appears once you group rows by who did them and sort by time. That ordered sequence is a *path*. What counts as a path is your choice, not the library's. Typically, you want to switch between user-level and session-level paths. See [Eventstream key concepts](https://retentioneering.com/docs/eventstream#key-concepts) for how to declare and switch between path definitions.

Those five paths are the running example for the rest of this page:

![A raw event log with one row per event, grouped and sorted into five ordered paths](https://retentioneering.com/docs-demos/img/paths.svg "Left: a raw event log, one row per event. Right: the same rows grouped by user and sorted by time.")

Loaded into an [Eventstream](https://retentioneering.com/docs/eventstream) — the object every tool on this page is called on — they look like this:

```python
import pandas as pd
import retentioneering as rete

paths = {
    "u1": ["home", "catalog", "cart", "purchase"],
    "u2": ["home", "catalog", "cart", "purchase"],
    "u3": ["home", "catalog", "catalog", "cart"],
    "u4": ["home", "search", "cart"],
    "u5": ["home", "search", "search"],
}
df = pd.DataFrame(
    [
        {"user_id": path_id, "event": event, "timestamp": f"2024-01-01 10:0{step}"}
        for path_id, events in paths.items()
        for step, event in enumerate(events)
    ]
)
stream = rete.Eventstream(df)
```

Every widget shown below is that `stream` — so each one renders exactly the five paths above, and you can check its numbers by hand. See [Eventstream](https://retentioneering.com/docs/eventstream) for loading your own data, naming columns and declaring a schema.

You will also meet two events that appear in no event log: `path_start` and `path_end`. Retentioneering automatically brackets every path with them depending on the `path_col` argument that is given to the widget.

## Representation 1: by step

A natural way to visualize paths is to stack them up so they would form a **trajectory tree**.

![Five paths merged into a trajectory tree, each node carrying the number of paths passing through it](https://retentioneering.com/docs-demos/img/trajectory-tree.svg "The five example paths stacked into a trajectory tree: identical beginnings share a branch, and each node carries the number of paths passing through it.")

On five paths it is the clearest view available: every route is visible, and each branch carries the number of paths on it. On real data it stops working. Branching compounds at every step, so even a few thousand paths over a few dozen event names give a tree with nearly as many leaves as there are paths. Nothing repeats often enough to be legible, and no layout rescues it.

The way out is to give up *which branch* a path came from and keep *where in the path it is*: group the paths by step and look at the distribution of events at each step. Everything at the same depth carrying the same event name collapses into one number.

![The step matrix of the five example paths: one row per event, one column per step](https://retentioneering.com/docs-demos/img/step-matrix.svg "The same tree counted by step. The two cart nodes at step 3 — one reached via catalog, one via search — merge into a single cell: 2 + 1 = 3 of 5 paths.")

That table is the [Step Matrix](https://retentioneering.com/docs/widgets/step-matrix): one row per event, one column per step, each cell is the share of paths sitting on that event at that step. **Every column sums to 1**, because every path is somewhere at every step — including the paths that are already over, which pile up in `path_end`. And **a cell counts paths, not events**: e.g. `cart` at step 3 is `0.6` because three of the five paths had `cart` as their third event.

```python
stream.step_matrix()
```

[Step Sankey](https://retentioneering.com/docs/widgets/step-sankey) draws the exact same numbers as flows between adjacent columns instead of as a heatmap — the two widgets share a single headless method, `step_sankey_data()`. Use the matrix to compare one event across many steps, the Sankey to watch volume move from one step to the next.

```python
stream.step_sankey()
```

**Blind spot: which branch a path came from.** The two ways of reaching `cart` at step 3 — via `catalog` and via `search` — merge into a single `0.6` cell, and nothing downstream can tell them apart. In particular, a Sankey ribbon does not track individual paths across more than one hop: following three ribbons in a row does *not* give you the share of users who took that three-event route. See in the next section how to capture narrower sub-sequences with a `path_pattern`.

### Aligning on any event

Steps are counted from the beginning of each path by default. That is the right frame for onboarding questions and the wrong one for almost everything else — if the thing you care about happens at step 4 for some users and step 40 for others, aligning at the start smears it across many columns.

`path_pattern` can re-anchor the columns on an event instead. Every path is shifted until that event sits in column 0. The steps before/after it get negative/positive numbers correspondingly:

![Five paths shifted so that each path's cart event sits in column zero, with negative column numbers before it](https://retentioneering.com/docs-demos/img/path-pattern.svg 'Every path shifted until its cart event sits in column 0. u5 never reaches it at all and drops out of the view.')

Now column −1 answers "what do users do immediately before the cart?" (75% `catalog`, 25% `search`) and column +1 answers "what happens right after?" (50% `purchase`, 50% the path ends). Neither question is answerable from the start-anchored view.

```python
stream.step_matrix(path_pattern="cart")
```

Note the denominator: `u5` never reaches `cart`, so it is not in this picture at all, and the shares are out of the four paths that do. Re-anchoring silently redefines the population to "paths that match the pattern" — which is usually what you want, and always worth knowing.

Two more anchors worth having:

- `path_pattern="path_end"` — [centers on where paths stop](https://retentioneering.com/docs/widgets/step-matrix#path-pattern---drop-off-points), so the columns to its left are the last things users did before leaving.
- `path_pattern="catalog->.*->purchase"` — [several anchors in sequence](https://retentioneering.com/docs/widgets/step-matrix#path-pattern---funnel-patterns), where `.*` matches any run of events. This allows you to explore steps surrounding events in a funnel pattern.

## Representation 2: by transition

Now drop position altogether and look at one transition at a time. Every path breaks down into the pairs of events that follow each other in it (bigrams, if you know the term from text analysis). For example, `u3`, `home → catalog → catalog → cart`, gives the pairs `home→catalog`, `catalog→catalog` and `catalog→cart`. Transition graph is a graph where nodes are unique events, edges are transitions between them, and the edge weight is some numerical characteristic of the transition, e.g. transition count.

![Five paths on the left; on the right their adjacent event pairs counted into a transition graph with weighted edges and self-loops](https://retentioneering.com/docs-demos/img/event-aggregation.svg "Building a transition graph from paths. Edge weight is the number of transitions.")

```python
stream.transition_graph(edge_weight="count")
```

This buys two things the step representation cannot give you:

- **Loops become visible.** `u3`'s two consecutive `catalog` events and `u5`'s two `search` events are pairs like any other, so they turn into self-loops. Users going in circles are one of the most common findings in real event logs, and this is the representation that shows them directly.
- **The picture stays finite.** However long the paths, the graph has one node per distinct event — so it works as a map of the product, while the step representation grows a column per step.

A transition can be scored by more than a count: by how many distinct paths made it, by a transition probability (a.k.a. [Markov transition probabilities](https://en.wikipedia.org/wiki/Discrete-time_Markov_chain), the default option), by how long it takes. See the full list is on the [Transition Graph](https://retentioneering.com/docs/widgets/transition-graph#edge-weights) page.

**Blind spot: when it happened.** A `cart → purchase` edge of weight 2 says two transitions happened somewhere; it does not say whether it was on step 3 or step 30, or whether they happened in the same path, or what exactly preceded them (the latter explains why Markov chains are called "memoryless" models).

## Representation 3: by milestone

A [Funnel](https://retentioneering.com/docs/widgets/funnel) keeps only the events you name, and asks of each path: did it reach these, in this order? Everything in between is ignored.

```python
stream.funnel(steps=["home", "cart", "purchase"])
```

Of the five paths, all five hit `home`, four reach `cart` (`u5` never does) and two go on to `purchase` — 100% → 80% → 40%.

**Blind spot: everything that is not a milestone** — which is exactly why a funnel is easy to explain to stakeholders and unable to tell you *why* the drop happened. See [how to analyze what happens between funnel levels](https://retentioneering.com/docs/recipes#open-up-the-funnel).

## Comparing groups of paths

Every representation above summarizes one population of paths, while most real questions are comparative: this cohort against that one, before against after, converted against not. Define the groups as a [segment](https://retentioneering.com/docs/segments) — an A/B arm, a platform, an acquisition channel, a time window, a behavioral cluster, a rule over your own columns — and every core widget will render `group1 − group2` in place of one group's numbers ([diff mode](https://retentioneering.com/docs/widgets#diff-mode)).

Three important cases that are widely used in practice:

- **An anomalous period.** A dynamic segment splits the eventstream into "inside the window" and "outside", and a diffed widget shows which steps or transitions actually degraded there — see [Find a root cause in an anomalous period](https://retentioneering.com/docs/recipes#find-a-root-cause-in-an-anomalous-period).
- **How far a path got in a funnel.** `add_segment(..., funnel_events=[...])` labels every path with the deepest funnel level it completed *in order*, so "stalled at `cart`" and "converted" become two comparable groups — see [Open up the funnel](https://retentioneering.com/docs/recipes#open-up-the-funnel).
- **The user's state.** A user is not the same person in their first session and in their fiftieth. `split_sessions` numbers each path's sessions, and a rule over that number turns it into named stages — first session, getting familiar, experienced — so the diff compares the *same* people at different points of their lifetime — see [Compare newcomers with experienced users](https://retentioneering.com/docs/recipes#compare-newcomers-with-experienced-users).

Because both groups are still summarized from raw events, the difference lands on a concrete event, step or transition instead of on another aggregate rate — which is what makes this a root-cause tool rather than a reporting one.

**More than two levels?** A segment often has more levels than that — six acquisition channels, dozens of countries or regions — and diffing every pair is not a plan. [Segment Overview](https://retentioneering.com/docs/widgets/segment-overview) puts [metrics](https://retentioneering.com/docs/path-metrics) in rows and segment levels in columns, so one heatmap tells you which level is the outlier and on which metric. That is the big picture; the diff is the drill-down that follows it.

**Need behavioral segments?** Every example above starts from a rule you already had in mind. When you don't have one, [Cluster Analysis](https://retentioneering.com/docs/widgets/cluster-analysis) goes the other way round — it groups paths by their [metrics](https://retentioneering.com/docs/path-metrics), suggests a splitting, and lets you inspect what each cluster actually contains. Once clustered, it becomes an ordinary segment column: from there it is the same `diff` and the same three representations as everything else on this page.

## Preparing the data

Event logs rarely arrive in a shape worth analysing. Event names are whatever the tracking code happened to emit, passing-by users and bots sit next to meaningful ones, sessions are not marked, etc. [Data processors](https://retentioneering.com/docs/data-processors) are small methods built for exactly this kind of work — filtering, renaming, collapsing, splitting, labelling. Each one returns a new `Eventstream`, so they chain into a readable pipeline and never modify what they were called on:

```python
stream = (
    rete.Eventstream(df)
    .filter_events(drop={"event": ["checkout_bug"]})
    .split_sessions(timeout="30m")
    .truncate_paths(start_event="path_start", end_event="purchase")
)
```

Processors validate event names against your data, so a misspelled event raises instead of silently filtering nothing — see [Data Processors](https://retentioneering.com/docs/data-processors#event-names-are-validated).

### When the picture is unreadable

Retentioneering's widgets are built for high-cardinality event logs — hundreds of distinct event names — but applied head-on to a raw log they can still come out as a hairball of a graph or a plate of Sankey spaghetti. That is rather a data-preparation signal, not a rendering problem. Consider to try the following data processors:

- [`filter_events`](https://retentioneering.com/docs/data-processors/filter-events) / [`drop_events`](https://retentioneering.com/docs/data-processors/drop-events) — remove technical and bot events that split the paths without meaning anything.
- [`filter_paths`](https://retentioneering.com/docs/data-processors/filter-paths) — remove short paths or paths that do not follow the pattern you are interested in.
- [`collapse_events`](https://retentioneering.com/docs/data-processors/collapse-events) — merge related events (all the checkout steps into one `checkout`), or squash runs of repeats.
- [`truncate_paths`](https://retentioneering.com/docs/data-processors/truncate-paths) — cut every path down to the window you are actually asking about.

Also, treating sessions as paths might help: [Path as a sequence of sessions](https://retentioneering.com/docs/recipes#path-as-a-sequence-of-sessions).

## Which tool for which question

| Question | Tool | Why |
|---|---|---|
| How does the product connect up? Where do users loop? | [Transition Graph](https://retentioneering.com/docs/widgets/transition-graph) | The only representation that shows repetition and cycles |
| What happens right before / right after event X? | [Step Matrix](https://retentioneering.com/docs/widgets/step-matrix) or [Step Sankey](https://retentioneering.com/docs/widgets/step-sankey) with `path_pattern="X"` | Position relative to an anchor is preserved |
| How do the first N steps unfold? | Step Sankey, no pattern | Start-anchored steps are the question |
| Where do users leave? | Step Matrix with `path_pattern="path_end"` | Anchors on the ending |
| What share reaches these milestones in order? | [Funnel](https://retentioneering.com/docs/widgets/funnel) | Ignores everything between them, on purpose |
| Do two groups behave differently? | Any of the above with [`diff`](https://retentioneering.com/docs/widgets#diff-mode) | Renders group1 − group2 in place |
| Which of many segment levels stands out? | [Segment Overview](https://retentioneering.com/docs/widgets/segment-overview) | Scans every level on metrics at once, so you know which pair to diff |
| Which behaviors exist at all? | [Cluster Analysis](https://retentioneering.com/docs/widgets/cluster-analysis) | Groups paths by metrics instead of by shape, and saves the result as a segment |

## Asking an agent instead

The same techniques introduced on this page could be used by an LLM agent with the help of the built-in [MCP server](https://retentioneering.com/docs/mcp-server) or [skills](https://retentioneering.com/docs/agent-skills).

## Next steps

- [Widgets](https://retentioneering.com/docs/widgets) — the shared behaviour of every visualization: diff mode, headless twins, HTML export.
- [Segments](https://retentioneering.com/docs/segments) — how to define the groups you compare.
- [Recipes](https://retentioneering.com/docs/recipes) — these ideas applied to concrete product questions.
- [Data processors](https://retentioneering.com/docs/data-processors) — how to prepare the data for analysis.
- [MCP server](https://retentioneering.com/docs/mcp-server) — connecting an agent, the full tool list, and what it can and cannot do.

---

# Eventstream

Source: https://retentioneering.com/docs/eventstream

`Eventstream` is the central object in retentioneering. It wraps your event data and exposes all widgets and data processors as methods.

> Throughout this documentation, `stream` refers to an `Eventstream` instance. All code examples assume you have created one as shown below.

## Key concepts

**Path** is the unit of analysis — an ordered sequence of events that all analysis tools (transition graph, step matrix, funnel, path metrics) operate on. What counts as a path is defined by the `path_col` you choose, not fixed by the library:

- `path_col="user_id"` — a path is the whole **user journey**. If you come from Amplitude or Mixpanel, this is the closest match to how those tools group events by user.
- `path_col="session_id"` — a path is a single **session**.

Every widget and most data processors accept a `path_col` override, so the same eventstream can be analysed at user grain and at session grain without rebuilding it — **but `path_col` must be one of the columns declared in `path_cols`** (see [schema](#schema) below); passing any other column raises an error.

`path_cols` must be listed **coarsest grain first**: every value of `path_cols[i+1]` must belong to exactly one value of `path_cols[i]` (e.g. every `session_id` belongs to exactly one `user_id`, so `path_cols=["user_id", "session_id"]` is correct). This nesting is validated against your data when the `Eventstream` is created — a schema declared the wrong way round (or a `session_id` that isn't actually unique per user) raises `SchemaConfigError` immediately, rather than producing silently-wrong analysis. See [ADR-0004](https://github.com/retentioneering/retentioneering-tools/blob/master/docs/adr/0004-schema-and-grain-neutral-paths.md) for why.

Need to group by something that *isn't* a nested grain of your path — a device type, a campaign, an arbitrary cohort? That's not a `path_col`; use a segment column instead (see below).

**Step** is a position within a path — its 1st event, its 2nd, and so on. Steps are what [Step Matrix and Step Sankey](https://retentioneering.com/docs/path-analysis) count over, and they are always relative to an anchor: the start of the path by default, or any event you choose.

**Segment** is a way to split paths into meaningful groups. A split is defined by a *segment column* that maps each event to a group: for example, a `country` column assigns one of the segment levels `US`, `DE`, `FR` to every event of a path. Segments can be static (acquisition channel, user age group, etc.) or dynamic — changing along the path, like weekend/weekday or an evolving user state (new/returning/loyal). Segment columns are declared in the [schema](#schema) (`segment_cols`) or created with the [Add Segment](https://retentioneering.com/docs/data-processors/add-segment) data processor, and drive all segment-aware tools. See the [Segments](https://retentioneering.com/docs/segments) page for the full story.

### Coming from Amplitude, Mixpanel or GA4

Most of the vocabulary translates, with one structural difference worth knowing up front: those tools are built around *users*, while retentioneering is built around *paths* — and a path is whatever grain you declare, so the same eventstream can be analysed per user and per session without reloading it.

| There | Here | Note |
|---|---|---|
| User / user timeline | **Path** with `path_col="user_id"` | The default. `path_col="session_id"` switches the whole analysis to session grain. |
| User property, cohort, A/B arm | **Segment column** | One segment column describes a whole *split* (`US`/`DE`/`FR`), not a single group. Values live per event, so a segment can also change along a path. |
| Event property | **Custom column** | Any extra column rides along; promote it to a segment with [`add_segment`](https://retentioneering.com/docs/data-processors/add-segment) when you want to compare by it. |
| Pathfinder / Journeys / Path exploration | [Transition Graph](https://retentioneering.com/docs/widgets/transition-graph) | Plus [Step Sankey](https://retentioneering.com/docs/widgets/step-sankey) for the step-by-step view. |
| Funnel | [Funnel](https://retentioneering.com/docs/widgets/funnel) | Ordered and unique-path-counted; see its page for the exact conventions. |
| Segment comparison | [Diff mode](https://retentioneering.com/docs/widgets#diff-mode) | Every core widget renders group1 − group2 directly. |
| Session (defined server-side) | [`split_sessions`](https://retentioneering.com/docs/data-processors/split-sessions) | You choose the timeout or the boundary events, and can change your mind later. |

There is no dashboard state to configure and no sampling: everything is a method call on an object you hold, computed over the full log on your machine.

## Creating an Eventstream

By default, Eventstream expects columns named `user_id`, `event`, and `timestamp`. If your data uses different column names, pass a [schema](#schema).

```python
import pandas as pd
import retentioneering as rete

df = pd.read_csv("events.csv")
stream = rete.Eventstream(df)
```

You can also pass a CSV path directly:

```python
stream = rete.Eventstream("events.csv")
```

## Expected data format

Each row in your DataFrame represents a single event. At minimum, you need a path identifier column, an event name column, and a timestamp column.

| user_id | event | timestamp |
|---|---|---|
| user_1 | page_view | 2024-01-01 10:00:00 |
| user_1 | add_to_cart | 2024-01-01 10:02:00 |
| user_1 | purchase | 2024-01-01 10:05:00 |

## Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `df` | `DataFrame \| str` | required | Event data as a pandas DataFrame or a path to a CSV file. |
| `schema` | `dict \| None` | `None` | Schema configuration. See below. |
| `preprocess` | `bool` | `True` | When `True`, parses timestamps, casts categoricals, and sorts rows. Set to `False` if your DataFrame is already preprocessed. |

## Schema

The schema tells retentioneering which columns in your DataFrame correspond to paths, events, timestamps, and segments. Pass it as a dict to the `schema` parameter.

```python
stream = rete.Eventstream(df, schema={
    "path_cols": ["user_id"],
    "event_cols": ["event"],
    "timestamp_col": "timestamp",
    "segment_cols": ["country", "plan"],
})
```

| Field | Default | Description |
|---|---|---|
| `path_cols` | `["user_id"]` | Columns that identify a path, ordered **coarsest grain first** (e.g. `["user_id", "session_id"]`). The first column is the primary/default path ID and every later column must nest inside all earlier ones — validated against your data at construction time. `path_col` overrides passed to tools must be one of these columns. |
| `event_cols` | `["event"]` | Columns that contain event names. The first column is the primary event column. |
| `timestamp_col` | `"timestamp"` | The timestamp column. |
| `segment_cols` | `[]` | Columns treated as segmentations, available in widgets and metrics. See [Key concepts](#key-concepts). |
| `custom_cols` | `None` | Extra columns you may need for working with the eventstream. Left as `None`, every column not covered by the rest of the schema is included automatically. Set to a list — even `[]` — and only those columns (plus the ones already covered by the schema) are kept; anything else is dropped. |

## Sample dataset

retentioneering ships with a synthetic e-commerce dataset you can use to try the library without your own data. It contains six months of user sessions on a consumer electronics store, with several embedded behavioral patterns designed to showcase the analysis tools.

```python
import retentioneering as rete

stream = rete.datasets.load_ecom()
```
which is equivalent to:

```python
import retentioneering as rete

df = rete.datasets.load_ecom(as_dataframe=True)
stream = rete.Eventstream(df, schema={
    "path_cols": ["user_id", "session_id"],
    "segment_cols": [
        "platform",
        "acquisition_channel",
        "user_cohort",
        "user_lifecycle",
    ],
})
```

## Inspecting your data

`stream.describe()` is a quick sanity check on what got loaded: dataset shape, schema, date range, event frequency, and path length/duration statistics.

```python
stream.describe()
```

Parameters:

- `percentiles` — percentiles (0-1) reported in `path_stats`. Default `(0.25, 0.5, 0.75, 0.9, 0.99)`.
- `top_events` — number of most frequent events to include in `event_frequency`. Default `20`; pass `None` to include every unique event, unranked and unlimited (e.g. when building a full event rename mapping, where the default cap would silently drop long-tail events).

Returns a dict:

| Key | Contents |
|---|---|
| `schema` | `event_col`, `path_col`, `path_cols`, `segment_cols`, `timestamp_col` |
| `shape` | `n_events`, `n_paths`, `n_unique_events` |
| `date_range` | `min`, `max`, `span` |
| `event_frequency` | `DataFrame` of `event`/`count`/`share`, sorted descending, limited to `top_events` rows (default 20; pass `top_events=None` for the full, unranked list). `.attrs["truncated"]` and `.attrs["n_total_events"]` say whether/how much this was cut down |
| `path_stats` | dict keyed by each entry of `path_cols`, each a `DataFrame` (`DataFrame.describe()` shape: count/mean/std/min/percentiles/max) with `length`/`duration` columns |
| `segments` | `DataFrame` of `segment_col`/`value`/`count`/`share`, one row per segment value across all segment columns |

## Other Eventstream methods

Beyond the [data processors](https://retentioneering.com/docs/data-processors) and [widgets](https://retentioneering.com/docs/widgets), the object exposes a handful of methods that don't fit either category.

### Getting your data back out

```python
df = stream.to_dataframe()                      # plain pandas copy
df = stream.to_dataframe(exclude_start_end=False)  # keep path_start / path_end rows
```

Processors never mutate an eventstream, so a `to_dataframe()` result is a snapshot you can hand to any other library. `stream.get_event_counts()` returns a `{event: count}` dict, and `stream.get_segment_levels()` returns `{segment_col: [levels]}` — handy before writing a `diff=` or a rename mapping.

### Per-path metrics as a feature table

`get_metrics()` runs the same [path metrics](https://retentioneering.com/docs/path-metrics) registry that powers Segment Overview and clustering, and returns one row per path — a ready-to-join feature table for churn or LTV models.

```python
features = stream.get_metrics([
    {"metric": "length"},
    {"metric": "duration"},
    {"metric": "active_days"},
    {"metric": "event_count_bulk"},
    {"metric": "matches_pattern", "metric_args": {"pattern": "home->.*->purchase"}},
])
```

Metric configs here take no `agg` field — these are raw per-path values, not aggregates. `event_count_bulk` / `has_event_bulk` expand into one column per event.

`get_metric_distribution()` is the related one-off: it returns the distribution of a single metric for one segment value against another (or against everything else, with `complement=True`), which is what the Segment Overview widget draws when you click a cell.

### Reproducing an eventstream

Every processor call is recorded, so a derived eventstream can describe how it was built:

```python
prepared = stream.filter_events(drop={"event": ["checkout_bug"]}).truncate_paths(
    start_event="path_start", end_event="purchase"
)
prepared.recipe()
# [{"type": "filter_events", "drop": {...}},
#  {"type": "truncate_paths", "start_event": "path_start", "end_event": "purchase"}]
```

`Eventstream.from_recipe(df, recipe)` replays that list onto a base DataFrame, rebuilding an identical eventstream — useful for moving a prepared pipeline between notebooks, storing it next to a report, or handing it to the [MCP server](https://retentioneering.com/docs/mcp-server), whose preprocessor steps use exactly this format. `stream.fingerprint` (a property) is a content hash for checking two eventstreams really are the same, and `stream.equals(other)` compares them directly.

---

# Widgets

Source: https://retentioneering.com/docs/widgets

Widgets are interactive visualizations that run directly in your notebook. Each widget connects to your eventstream, computes the relevant analysis, and renders an interactive UI where you can adjust parameters and explore the data without writing additional code.

## Creating a widget

Every widget is available as a method on the [Eventstream](https://retentioneering.com/docs/eventstream) object. In the examples throughout this section, `stream` refers to an `Eventstream` instance.

```python
stream.transition_graph()
stream.step_sankey()
stream.step_matrix()
stream.funnel()
stream.segment_overview()
stream.cluster_analysis()
```

All parameters are optional. You can call any widget without arguments and configure it interactively using the sidebar that appears on the right side of the widget.

## Common parameters

All widgets share the following parameters:

| Parameter | Type | Description |
|---|---|---|
| `path_col` | `str \| None` | Override the path ID column from the schema. |
| `height` | `int` | Widget height in pixels. |
| `sidebar_open` | `bool` | Whether the settings sidebar starts open. Default: `True`. |
| `state_file` | `str \| None` | JSON file the widget state is bound to. See [Saving widget state](#saving-widget-state). |

Each widget's parameters split into two groups. **Data parameters** change the
computed result — they are exactly the arguments of the widget's headless
`*_data()` twin (see [Headless mode](#headless-mode)). **Display parameters**
(`height`, `sidebar_open`, ...) only affect how the widget is rendered. The
per-widget documentation pages use the same two groups.

## Interactive configuration

Every widget has a settings sidebar where you can adjust all parameters interactively. Changes take effect immediately when you click **Apply**. You do not need to re-run the notebook cell.

Passing arguments and configuring the widget in the sidebar are equivalent. For example, these two are the same:

```python
# Configure via arguments
stream.funnel(steps=["catalog", "add_to_cart", "purchase"])

# Configure interactively — open the widget, add the same steps in the sidebar
stream.funnel()
```

Arguments are a convenient way to reproduce a specific configuration without clicking through the UI. They are also useful when sharing notebooks — a reader can see the configuration at a glance without opening the widget.

## Diff mode

Transition Graph, Step Sankey, Step Matrix, and Funnel support diff mode, which overlays two groups in the same visualization so you can compare their behavior directly. The `diff` parameter takes one of two shapes, depending on how you want to define the two groups.

Every diff value is `group1 − group2`: a positive value (shown in red) means group1 is higher, a negative value (shown in blue) means group2 is higher. Tooltips spell out the exact values with an explicit `+`/`-` sign.

### By segment

Pass a 3-element tuple or list `(segment_col, value1, value2)` to compare two values of a segment column:

```python
stream.funnel(
    steps=["catalog", "add_to_cart", "purchase"],
    diff=("user_lifecycle", "loyal", "new"),
)
```

Use the reserved value `<REST>` as `value2` to compare a segment against everyone else:

```python
stream.transition_graph(diff=("acquisition_channel", "paid_search", "<REST>"))
```

This form is also available interactively in the widget sidebar: pick a segment column and two of its values from the dropdowns.

### By path-id groups

Pass a 2-element tuple or list `(path_ids1, path_ids2)` — each side any iterable of path IDs (list, tuple, set, ...) — to compare two explicit, arbitrary groups of paths, without going through a segment column:

```python
stream.step_matrix(diff=(["user_0000", "user_0001"], ["user_0002", "user_0003", "user_0004"]))
```

This form is programmatic only — there is no sidebar UI for picking path-id groups. Tooltips and captions show generic "Group 1" / "Group 2" labels instead of segment names or raw IDs.

## Headless mode

Every widget has a corresponding **headless** method that runs the same computation and returns the raw data instead of rendering a widget. This is useful for building custom visualizations, exporting results, or running analysis in automated pipelines.

Headless methods follow the naming pattern `<widget>_data()`:

| Widget | Headless |
|---|---|
| `stream.transition_graph()` | [`stream.transition_graph_data()`](https://retentioneering.com/docs/widgets/transition-graph#headless-mode) |
| `stream.step_sankey()` | [`stream.step_sankey_data()`](https://retentioneering.com/docs/widgets/step-sankey#headless-mode) |
| `stream.step_matrix()` | [`stream.step_matrix_data()`](https://retentioneering.com/docs/widgets/step-matrix#headless-mode) — alias of `step_sankey_data()`; both widgets share one computation |
| `stream.funnel()` | [`stream.funnel_data()`](https://retentioneering.com/docs/widgets/funnel#headless-mode) |
| `stream.segment_overview()` | [`stream.segment_overview_data()`](https://retentioneering.com/docs/widgets/segment-overview#headless-mode) |
| `stream.cluster_analysis()` | [`stream.cluster_analysis_data()`](https://retentioneering.com/docs/widgets/cluster-analysis#headless-mode) |

```python
# Get transition matrix as a DataFrame
tm = stream.transition_graph_data(edge_weight="proba_out")

# Get funnel results as a dict
result = stream.funnel_data(steps=["catalog", "add_to_cart", "purchase"])
```

Headless methods accept the same parameters as their widget counterparts, excluding those that are needed for visualization only, like `height`.

### Headless data in diff mode

Diff mode changes what each widget's headless `*_data()` twin returns (see [Headless mode](#headless-mode)):

- `transition_graph_data()` returns `(diff, group1, group2)` instead of a single matrix — three DataFrames, where `diff = group1 - group2`.
- `step_sankey_data()` / `step_matrix_data()` return `(combined, group1, group2)` instead of a single DataFrame — three DataFrames, where `combined = group1 - group2`. With `path_pattern` (multiple anchor blocks), each of the three is instead a tuple with one DataFrame per block, and `combined_blocks[i] = group1_blocks[i] - group2_blocks[i]`.
- `funnel_data()` keeps the same `{"steps": [...]}` shape, but each step dict gets `funnel1_unique_paths`, `funnel1_conversion_rate`, `funnel1_step_conversion_rate`, `funnel2_unique_paths`, `funnel2_conversion_rate`, `funnel2_step_conversion_rate`, `delta_unique_paths`, `delta_conversion_rate`, and `delta_step_conversion_rate` instead of the plain `unique_paths`/`conversion_rate`/`step_conversion_rate` keys (`delta_* = funnel1_* - funnel2_*`).

## Exporting to HTML

Every widget has an `export_html()` method that writes the current widget as a standalone, self-contained HTML file — no Python kernel, notebook, or network access is required to open it. This is useful for sharing a result with people who don't have Jupyter, or for archiving a snapshot of an analysis.

```python
widget = stream.transition_graph(edge_weight="proba_out")
widget.export_html("transition_graph.html")
```

`export_html()` takes:

| Parameter | Type | Description |
|---|---|---|
| `path` | `str` | Destination file path. |
| `title` | `str` | Title shown in the browser tab. Defaults to the widget's name, e.g. `"Transition Graph"`. |
| `analysis` | `str \| None` | Optional analysis text rendered alongside the widget. Supports basic markdown (bold, italic, bullet lists, tables, headings). Wrap an event name in square brackets, e.g. `` [basket] ``, to turn it into a link that focuses that event in the widget. |
| `sidebar_open` | `bool \| None` | Whether the settings sidebar starts open in the exported file. Defaults to the widget's current `sidebar_open` value. |

The export captures the widget's full state at the time `export_html()` is called — computed results, current parameters, and display preferences like node layout or sort order — so the file reproduces exactly what you saw in the notebook. Because there's no kernel behind the exported file, controls that would require recomputation (e.g. changing `edge_weight`) are disabled; layout, zoom, and other purely visual interactions still work.

```python
stream.funnel(steps=["catalog", "add_to_cart", "purchase"]).export_html(
    "funnel.html",
    title="Checkout funnel",
    analysis="Drop-off at [add_to_cart]: most users leave before reaching checkout.",
)
```

## Saving widget state

Every widget accepts a `state_file` parameter — a path to a JSON file the
widget state is bound to, e.g. `stream.transition_graph(state_file="checkout.json")`.
The file is loaded if it exists and created otherwise, and every subsequent
change is auto-saved to it. It captures the full widget configuration — data
and display parameters plus extras like node layout, filters, sorting, scroll
position, and zoom (results are recomputed, not saved) — so re-running the
cell restores the widget exactly as you left it. Explicitly passed arguments
override the loaded state.

---

# Data Processors

Source: https://retentioneering.com/docs/data-processors

Data processors transform an [Eventstream](https://retentioneering.com/docs/eventstream) and return a new one. They are available as methods on the `Eventstream` object. In the examples throughout this section, `stream` refers to an `Eventstream` instance.

Processors can be chained:

```python
stream = (
    rete.datasets.load_ecom()
    .add_start_end_events()
    .filter_events(drop={"event": ["checkout_bug"]})
    .rename_events({"wishlist_add": "add_to_wishlist"})
)
```

Each processor returns a new `Eventstream`, so the original is never modified.

## Event names are validated

Processors check every event name you pass against the events actually present in the eventstream, and raise a `PreprocessingConfigError` listing the available names when one doesn't match:

```python
stream.filter_events(drop={"event": ["bot_ping"]})
# PreprocessingConfigError: [filter_events] Value(s) ['bot_ping'] not found in
# column 'event'. Available values: ['account_page', 'add_to_cart', 'cart', ...]
```

This is deliberate. A typo in an event name would otherwise turn into a filter that quietly keeps everything, or a rename that quietly does nothing — and you would find out several steps later, from a chart that looks plausible and is wrong. Use `stream.get_event_counts()` (or [`describe()`](https://retentioneering.com/docs/eventstream#inspecting-your-data)) to get the exact spelling of every event before writing a pipeline.

## Overview

### Adding events

- [`add_events`](https://retentioneering.com/docs/data-processors/add-events) — insert synthetic events derived from existing events or a SQL query.
- [`add_start_end_events`](https://retentioneering.com/docs/data-processors/add-start-end-events) — prepend a `path_start` and append a `path_end` synthetic event to each path.

### Renaming, editing & merging events

- [`rename_events`](https://retentioneering.com/docs/data-processors/rename-events) — rename events using a mapping dict.
- [`edit_events`](https://retentioneering.com/docs/data-processors/edit-events) — rename and/or delete events in a single operation.
- [`collapse_events`](https://retentioneering.com/docs/data-processors/collapse-events) — merge consecutive or grouped events into a single representative event.
- [`urls_to_events`](https://retentioneering.com/docs/data-processors/urls-to-events) — turn a raw URL column into structured event names using a URL path tree.

### Filtering, truncating & sampling

- [`filter_events`](https://retentioneering.com/docs/data-processors/filter-events) — keep only rows that match a column filter, a Python predicate, or a SQL query.
- [`drop_events`](https://retentioneering.com/docs/data-processors/drop-events) — remove events from the eventstream by name.
- [`filter_paths`](https://retentioneering.com/docs/data-processors/filter-paths) — keep only paths that satisfy a metric condition.
- [`truncate_paths`](https://retentioneering.com/docs/data-processors/truncate-paths) — trim each path to the window between two anchor events.
- [`sample_paths`](https://retentioneering.com/docs/data-processors/sample-paths) — randomly sample paths (and all their events).

### Sessions & lifecycle

- [`split_sessions`](https://retentioneering.com/docs/data-processors/split-sessions) — split each path into sub-sessions and add session ID and index columns.
- [`to_daily_states`](https://retentioneering.com/docs/data-processors/to-daily-states) — convert the eventstream into daily lifecycle-state events (new, current, at-risk, dormant, ...).

### Segments & clustering

- [`add_segment`](https://retentioneering.com/docs/data-processors/add-segment) — add a new categorical segment column to the eventstream.
- [`drop_segment`](https://retentioneering.com/docs/data-processors/drop-segment) — remove a segment column from the eventstream.
- [`add_clusters`](https://retentioneering.com/docs/data-processors/add-clusters) — cluster paths using ML and add a new segment column with integer cluster labels.
- [`rename_segment_levels`](https://retentioneering.com/docs/data-processors/rename-segment-levels) — rename levels within an existing segment column.

---

# Segments

Source: https://retentioneering.com/docs/segments

Segmentation is how you slice an eventstream into comparable groups: mobile vs desktop, paying vs free, during an incident vs before it. Almost every non-trivial analysis is a comparison, and segments are the mechanism that defines what is being compared.

In retentioneering a segment is a way to split paths, or parts of paths, into meaningful groups. A split is defined by a *segment column* that maps each event to a group: for example, a `country` column assigns a country to every event of a path, splitting the eventstream into segment levels `US`, `DE`, `FR`, and so on. Note the difference from Amplitude-style tools, where "creating a segment" means defining a single group of users; here one segment column describes a whole split at once.

```python
stream = Eventstream(df, schema={
    "segment_cols": ["country", "plan"],
})
```

Columns already present in your data become segments by listing them in the schema's `segment_cols` up front. If the `Eventstream` already exists — the column just rode along as a `custom_col` (see [Eventstream](https://retentioneering.com/docs/eventstream#schema)) — call `add_segment` with no `rules`/`func`/`sql`/`funnel_events` argument to promote it in place, keeping its existing values:

```python
stream.add_segment("returned")
```

New segment columns are derived with [Add Segment](https://retentioneering.com/docs/data-processors/add-segment) and [Add Clusters](https://retentioneering.com/docs/data-processors/add-clusters), have their levels renamed with [Rename Segment Levels](https://retentioneering.com/docs/data-processors/rename-segment-levels), and are removed with [Drop Segment](https://retentioneering.com/docs/data-processors/drop-segment).

## Static and dynamic segments

Segment values are stored per event row. This gives two kinds of segments:

- **Static** — the value is constant for the whole path: `country`, `acquisition_channel`, an A/B test arm, a cluster label, the deepest funnel step reached in order. A static segment answers "*which paths* behave differently?"
- **Dynamic** — the value changes along the path, because it is assigned per event: `weekend` / `weekday` depending on each event's timestamp, `inside` / `outside` an incident window, a user state that evolves from `new` through `returning` to `loyal`. A dynamic segment answers "*which parts of a path* behave differently?"

The row-level modes of `add_segment` (`rules`, `sql`, `func`) can produce either kind — a static segment is simply one whose value happens to be constant within each path. The `funnel_events` mode and `add_clusters` always produce static, per-path segments.

Dynamic segments are handled naturally by the segment-aware tools. [Segment Overview](https://retentioneering.com/docs/widgets/segment-overview) splits each path into its within-segment fragments, so the same user's behavior inside and outside a time window lands in separate columns of the heatmap. The `in_segment` [path metric](https://retentioneering.com/docs/path-metrics) offers `any` / `all` / `event_share` modes to express "the path ever touched this segment", "the path stayed in it entirely", or "at least a given share of its events were in it".

## Creating segments

[Add Segment](https://retentioneering.com/docs/data-processors/add-segment) supports five modes — exactly one must be used per call:

```python
# rules — ordered CASE-WHEN conditions
stream.add_segment(
    "region",
    rules=[
        ["country", "=", "US", "domestic"],
        ["country", "in", "('GB', 'DE', 'FR')", "europe"],
        ["other"],
    ],
)

# sql — a DuckDB SELECT returning one label per row
stream.add_segment(
    "device",
    sql="SELECT CASE WHEN platform = 'mobile' THEN 'mobile' ELSE 'web' END FROM eventstream",
)

# func — any Python function over the raw DataFrame
stream.add_segment("power_user", func=lambda df: df["user_id"].map(power_users_lookup))

# funnel_events — the deepest funnel step each path completed in order;
# skipping or reordering a step keeps it out of that step's group (out_of_funnel
# if even the first step was never completed in order)
stream.add_segment("funnel", funnel_events=["add_to_cart", "checkout_start", "purchase"])

# time_range — binary "inside" vs "outside" an inclusive timestamp interval
stream.add_segment("incident", time_range=("2024-03-10", "2024-03-17"))
```

[Add Clusters](https://retentioneering.com/docs/data-processors/add-clusters) is a special case: it clusters paths by behavioral metrics with ML and stores the cluster labels as a new static segment, so clusters immediately work everywhere ordinary segments do.

## Common patterns

### Inside vs outside an anomalous period

When something unusual happens — a payment gateway incident, a broken release, a marketing spike, a bot attack — the first question is "how did behavior change?" A dynamic segment over the timestamp cleanly separates the anomalous window from normal operation. The `time_range` mode covers this directly — pass the `(start, end)` bounds and every event is labeled `inside` or `outside`:

```python
stream.add_segment("incident", time_range=("2024-03-10", "2024-03-17"))
```

For anything the binary inside/outside split doesn't cover — more than two buckets, or a boundary rule other than a plain inclusive interval — fall back to `sql`:

```python
stream.add_segment(
    "incident",
    sql="""
        SELECT CASE
            WHEN timestamp < '2024-03-10' THEN 'before'
            WHEN timestamp <= '2024-03-17' THEN 'during'
            ELSE 'after'
        END
        FROM eventstream
    """,
)
```

Because the segment is dynamic, a user active both during and outside the window contributes to both groups — you are comparing *behavior in the period* against *behavior outside it*, not "users who happened to be around that week" against everyone else. Pass it to a widget's diff mode (`diff=("incident", "inside", "outside")`) to see exactly which transitions or funnel steps degraded, or to [Segment Overview](https://retentioneering.com/docs/widgets/segment-overview) to scan many metrics at once.

### Weekend vs weekday

Time-of-week (or time-of-day) segments reveal rhythm in product usage — different intent, different conversion, different session shape:

```python
stream.add_segment(
    "day_type",
    sql="""
        SELECT CASE WHEN isodow(timestamp) IN (6, 7) THEN 'weekend' ELSE 'weekday' END
        FROM eventstream
    """,
)
```

The same pattern with `hour(timestamp)` gives working-hours vs evening segments.

### User state

A user is not the same person in their first session and in their fiftieth. A state segment assigns each event the user's stage at that moment, so you can compare how the *same* users navigate at different stages of their lifetime:

- The bundled [e-commerce dataset](https://retentioneering.com/docs/eventstream#sample-dataset) ships a `user_lifecycle` segment that evolves from `new` through `returning` to `loyal` session by session.
- [To Daily States](https://retentioneering.com/docs/data-processors/to-daily-states) takes the state idea further: it converts the eventstream into one row per path per calendar day, labelled with engagement states (`new`, `current`, `reactivated`, `at_risk_wau`, `dormant`, ...), which is a natural input for retention and lifecycle analysis.
- Any custom state logic fits in `add_segment`'s `func` mode — compute the state per row in pandas and return the labels.

Combined with the `in_segment` metric, state segments answer questions like "what share of paths ever reached the `loyal` state" (`mode="any"`) or "which paths spent most of their events in `dormant`" (`mode="event_share"`).

## Using segments in analysis

Once a segment column exists, every segment-aware tool picks it up:

- **Diff mode** — [Transition Graph](https://retentioneering.com/docs/widgets/transition-graph), [Step Sankey](https://retentioneering.com/docs/widgets/step-sankey), [Step Matrix](https://retentioneering.com/docs/widgets/step-matrix), and [Funnel](https://retentioneering.com/docs/widgets/funnel) accept `diff=(segment_col, value1, value2)` to render the difference between two segments; `value2` may be `<REST>` to compare one segment against everyone else.
- **[Segment Overview](https://retentioneering.com/docs/widgets/segment-overview)** — an interactive heatmap with metrics as rows and segment values as columns: the fastest way to scan where segments differ before diving into a specific widget. See its documentation page for parameters and examples.
- **`in_segment` path metric** — turns segment membership into a per-path value usable in [Path Metrics](https://retentioneering.com/docs/path-metrics) contexts, including as a clustering feature in [Add Clusters](https://retentioneering.com/docs/data-processors/add-clusters).

---

# Path Metrics

Source: https://retentioneering.com/docs/path-metrics

Path metrics are scalar values computed per path (see [Key concepts](https://retentioneering.com/docs/eventstream#key-concepts) for what counts as a path). All metric-accepting tools share this single registry. Metrics are used in three places:

- [Segment Overview](https://retentioneering.com/docs/widgets/segment-overview) — metrics are aggregated per segment and displayed as a heatmap.
- [Cluster Analysis](https://retentioneering.com/docs/widgets/cluster-analysis) — metrics are used as features for clustering, and separately as overview metrics for the resulting clusters.
- [Filter Paths](https://retentioneering.com/docs/data-processors/filter-paths) — metrics are used as conditions to keep or drop entire paths.

## Available metrics

| Metric | Description | metric_args |
|---|---|---|
| `length` | Total number of events in the path. | — |
| `duration` | Time in seconds between the first and last event. | — |
| `active_days` | Number of distinct calendar days with at least one event. Optionally restricted to specific events. | `active_events`: list[str] (optional) |
| `event_count` | Number of times a single event occurred. | `event`: str (required) |
| `has_event` | 1 if a single event occurred at least once, 0 otherwise. | `event`: str (required) |
| `event_count_bulk` | Number of times each event occurred, expanded into one column per event. Omit `events` (or pass `None`) to count every event in the stream — an explicit empty list is invalid. | `events`: list[str] (optional; omit/`None` for all events) |
| `has_event_bulk` | 1 if each event occurred at least once, 0 otherwise, expanded into one column per event. Omit `events` (or pass `None`) to check every event in the stream — an explicit empty list is invalid. | `events`: list[str] (optional; omit/`None` for all events) |
| `has_all_events` | 1 if **all** of the given events occurred at least once (AND semantics), 0 otherwise. | `events`: list[str] (required, non-empty) |
| `has_any_event` | 1 if **any** of the given events occurred at least once (OR semantics), 0 otherwise. | `events`: list[str] (required, non-empty) |
| `time_between` | Time in seconds between the first occurrences of two events. Returns null if either event is missing. Use `path_start` or `path_end` as anchors. | `start_event`: str, `end_event`: str |
| `first_event_time` | Unix timestamp of the first event in the path. | — |
| `matches_pattern` | 1 if the path matches a sequence pattern, 0 otherwise. Events are separated by `->` and matched as whole tokens (not substrings); `.*` matches any sequence of whole events. Example: `home->.*->purchase`. | `pattern`: str |
| `in_segment` | Checks whether path events belong to a segment value. Mode `any`: at least one event has the value. `all`: all events have the value. `event_share`: at least a threshold share of events have the value. If multiple segment values are selected, a separate metric is created for each value. | `segment_name`: str, `segment_value`: str or list[str], `mode`: `"any"` \| `"all"` \| `"event_share"`, `threshold`: float (for `event_share`) |

`event_count`/`has_event` are strict single-event metrics — one number per path,
comparable directly in a Filter Paths condition. Passing a list of events is a
different, separate concern handled by three distinct metrics rather than by
overloading `events`:

- `event_count_bulk`/`has_event_bulk` are a **shorthand for multiple metrics** —
  they expand into one column per event, for Segment Overview/Cluster Analysis.
  They cannot be used inside a Filter Paths (or `collapse_events` case)
  condition, since a condition needs exactly one comparable value per path.
- `has_all_events`/`has_any_event` are genuine single-valued metrics that
  combine a list of events into one 0/1 result (AND/OR respectively) — these
  **can** be used in Filter Paths conditions.

## Metric config format

Metrics appear in two different config formats depending on where they are used:

- **Segment Overview** (`metrics`) and **Cluster Analysis** (`overview_metrics`) — each metric requires an `agg` field that defines how per-path values are aggregated across paths in a segment.
- **Cluster Analysis** (`features`) — no `agg` field. The raw per-path values are used directly as clustering features.
- **Filter Paths** — metrics appear inside a condition tree with comparison operators. See the [Filter Paths condition format](#filter-paths-condition-format) section below.

| Key | Required | Description |
|---|---|---|
| `metric` | yes | Metric name from the table above. |
| `metric_args` | depends | Additional arguments for the metric. Required for `event_count`, `has_event`, `event_count_bulk`/`has_event_bulk` (unless the wildcard is intended), `has_all_events`/`has_any_event`, `time_between`, `matches_pattern`, and `in_segment`; optional for `active_days`. |
| `agg` | yes (Segment Overview `metrics` and Cluster Analysis `overview_metrics` only) | Aggregation function. See aggregations below. |

```python
metrics=[
    {"metric": "length", "agg": "mean"},
    {"metric": "duration", "agg": "median"},
    {"metric": "event_count", "metric_args": {"event": "purchase"}, "agg": "mean"},
    {"metric": "event_count_bulk", "metric_args": {"events": ["add_to_cart", "purchase"]}, "agg": "mean"},
    {"metric": "time_between", "metric_args": {"start_event": "path_start", "end_event": "purchase"}, "agg": "median"},
    {"metric": "matches_pattern", "metric_args": {"pattern": "home->.*->purchase"}, "agg": "mean"},
]
```

## Aggregations

Aggregations apply to `metrics` in Segment Overview and `overview_metrics` in Cluster Analysis. They are not used for clustering `features` or in Filter Paths conditions.

| Value | Description |
|---|---|
| `mean` | Mean value across all paths in the segment. |
| `median` | Median value (50th percentile). |
| `q5` | 5th percentile. |
| `q25` | 25th percentile. |
| `q75` | 75th percentile. |
| `q95` | 95th percentile. |
| `complement_distance` | Wasserstein distance between this segment's distribution and all other segments combined. Higher means more distinctive from the rest. **Segment Overview only** — requires a segment column to define what "the rest" means. |

## Filter Paths condition format

In [Filter Paths](https://retentioneering.com/docs/data-processors/filter-paths), metrics are used inside a condition tree rather than a flat list. Each leaf node specifies a metric, a comparison operator, and a value:

```python
# Single condition
{"op": ">", "metric": "length", "value": 5}

# Combine conditions
{
    "op": "and",
    "args": [
        {"op": ">", "metric": "length", "value": 3},
        {"op": "=", "metric": "has_event", "metric_args": {"event": "purchase"}, "value": True},
    ]
}

# A top-level list is shorthand for AND — equivalent to the tree above
[
    {"op": ">", "metric": "length", "value": 3},
    {"op": "=", "metric": "has_event", "metric_args": {"event": "purchase"}, "value": True},
]

# Keep paths that contain every one of several events (AND)
{"op": "=", "metric": "has_all_events", "metric_args": {"events": ["add_to_cart", "purchase"]}, "value": True}

# Keep paths that contain at least one of several events (OR)
{"op": "=", "metric": "has_any_event", "metric_args": {"events": ["promo_view", "discount_applied"]}, "value": True}
```

Supported operators: `=` (or `==`), `!=`, `>`, `<`, `>=`, `<=`. Logical nodes use `and`, `or`, `not` with an `args` list.

`event_count_bulk`/`has_event_bulk` cannot appear in a condition leaf — they expand into
multiple columns, and a condition needs exactly one comparable value per path. Use the
non-bulk `event_count`/`has_event` for a single event, or `has_all_events`/`has_any_event`
for a multi-event AND/OR condition.

---

# MCP Server (beta version)

Source: https://retentioneering.com/docs/mcp-server

retentioneering ships a built-in [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server that exposes your eventstream to any MCP-compatible AI agent. Once connected, the agent can analyse your behavioural data by calling [tools](#available-tools) — you simply ask questions in plain language.

The MCP server runs **locally** in the Jupyter kernel, so it requires a local Python environment. It will not work in cloud notebooks such as Google Colab.

There is a second, unrelated MCP server that serves *this documentation* rather than your data, needs no installation, and is described at the bottom of this page: [Documentation MCP server](#documentation-mcp-server).

## Starting the server

You can either start the server with no context so the agent will have to upload the eventstream itself to analyse it:
```python
import retentioneering as rete

rete.mcp.serve(port=8765)
```

or with context to analyse an eventstream that is already in the kernel's memory:

```python
import retentioneering as rete

stream = rete.datasets.load_ecom()
rete.mcp.serve(stream, context={
    "description": "E-commerce store. Main KPI — purchase conversion.",
    "events": {
        "purchase":    "Completed a purchase",
        "add_to_cart": "Added an item to cart",
    }
  },
  port=8765
)
```

The `stream` argument is the [Eventstream](https://retentioneering.com/docs/eventstream) you want the agent to analyse. It stays in the kernel's memory — the agent reads from it via tool calls without copying data anywhere. The optional `context` dict adds semantic information that the agent uses to write better analysis.

## Connecting an agent

Any agent that supports MCP over SSE can connect using the server URL `http://localhost:8765/sse` (adjust the port if you changed it). Setup instructions for popular agents:

- [Claude Code](https://docs.anthropic.com/en/docs/claude-code/mcp) — run `claude mcp add --scope user --transport sse retentioneering http://localhost:8765/sse`
- [Codex](https://developers.openai.com/codex/mcp) — add the server URL to Settings → MCP Servers → Add Server (Streamable HTTP)
- [Cursor](https://cursor.com/docs/mcp) — add the server under Settings → Tools & MCPs → New MCP Server

  ```json
  {
    "mcpServers": {
      "retentioneering": {
        "url": "http://localhost:8765/sse"
      }
    }
  }
  ```

## Example questions

Once connected, you can ask the agent questions like:

- What are the most common user flows in this data?
- Why was there a spike in conversion around January 20-22?
- Why do users drop-off between `basket` and `shipping_details` in the checkout funnel?
- How do new and loyal users behave differently?
- Build a transition graph and analyse the main bottlenecks.

The agent will call the appropriate tools, build visualisations, and generate a self-contained HTML report with interactive charts and clickable annotations.

See also [Agent Skills](https://retentioneering.com/docs/agent-skills) — instruction packages for coding agents (Claude Code, Codex) that run analyses directly in your own code, rather than over MCP.

## Available tools

| Tool | Description |
|---|---|
| `load_data(path, schema, context)` | Load an eventstream from a local CSV and set it as the session's base stream — required first call if `serve()` was started with no `stream` (data-agnostic mode); also usable later to switch datasets. |
| `describe()` | Schema, event list, path counts, timestamp range. |
| `reset_base_stream()` | Reset the active stream to the original eventstream passed to `serve()`. |
| `playbook(scenario)` | Step-by-step recipes for common analysis patterns. |
| `describe_tool(tool)` | Full parameter reference for any preprocessor. |
| `update_base_stream(preprocessors)` | Filter or transform the stream for the session. |
| `add_transition_graph(label, ...)` | Compute a transition graph and register it as a report tab. |
| `add_step_matrix(label, ...)` | Compute a step matrix and register it as a report tab. |
| `add_segment_overview(label, ...)` | Compute a segment overview and register it as a report tab. |
| `check_analysis(analysis)` | Validate analysis text before export. |
| `export_report(title, analysis, path)` | Generate a self-contained HTML report. |

## Documentation MCP server

Everything above describes the server that runs **on your machine, over your
data**. There is a second one that runs **on our side, over this
documentation**:

```
https://retentioneering.com/docs/mcp
```

It needs no installation, no kernel and no data — connect it to any MCP client
and the agent can search and read these pages while it writes retentioneering
code.

Connecting is the same procedure as [above](#connecting-an-agent), with the
URL instead of `localhost` — for example, in Claude Code:

```bash
claude mcp add --scope user --transport http retentioneering-docs https://retentioneering.com/docs/mcp
```

Three tools are available: `search_docs(query, limit)` returns the most relevant
documentation sections with deep links, `get_doc_page(path)` returns one page in
full, and `list_doc_pages()` returns the table of contents.

If you would rather not connect a server at all, the same content is published
as plain text following the [llms.txt convention](https://llmstxt.org):
[`/llms.txt`](https://retentioneering.com/llms.txt) is the annotated table of
contents, and [`/llms-full.txt`](https://retentioneering.com/llms-full.txt) is
the entire documentation as one file (~200 KB, roughly 50k tokens — small
enough to hand to an agent wholesale).

---

# Agent Skills

Source: https://retentioneering.com/docs/agent-skills

retentioneering ships two Agent Skills — task-oriented instruction packages that coding agents (Claude Code, Codex, Cursor, and other compatible tools) load automatically when a task matches, so the agent follows a tested workflow instead of improvising one from scratch each time.

This is different from the [MCP server](https://retentioneering.com/docs/mcp-server): the MCP server runs inside a local Jupyter kernel, so using it means keeping a kernel process alive for the agent to connect to. Agent Skills carry no such requirement — they teach an agent to write and run retentioneering code directly, e.g. against your own files in its normal working environment, with no server to keep running.

## Available skills

| Skill | Use when |
|---|---|
| `retentioneering-product-analytics` | You have event-level data (user, event, timestamp columns) and ask an agent why users convert, churn, loop, or abandon a flow. Drives the full workflow: inspect the log → pick a recipe → execute → validate → interpret. |
| `retentioneering-contributing` | You ask an agent to turn a bug, friction report, or feature idea into an issue or pull request against this repository. Drives: capture → validate → minimal reproduction → issue/PR. |

The skills are duplicated under two directories in the repository, kept identical by design:

- `.claude/skills/` — loaded automatically by Claude Code.
- `.agents/skills/` — the generic/Codex-compatible location, per the open Agent Skills spec.

Each skill is a directory with a `SKILL.md` entry point (objective, activation criteria, step-by-step workflow) plus `references/` (deep material, read on demand) and, for the product-analytics skill, a `scripts/inspect_event_log.py` profiler.

## Using a skill

There are two ways to give an agent access to a skill:

- **Regular: import the skill's `SKILL.md`.** Point your agent at the `SKILL.md` of the
  skill you want (from `.claude/skills/` or `.agents/skills/` in this repository), without
  cloning anything else alongside it. The agent follows the skill's workflow but runs
  retentioneering itself from the version installed from PyPI.
- **Advanced: run the agent from inside a clone of the repository.** Clone
  `retentioneering-tools` and run the agent from within the checkout. The agent still
  picks up the skills from `.claude/skills/` or `.agents/skills/`, but now runs
  retentioneering directly against the library code in the checkout instead of a PyPI
  install — useful when you're developing against an unreleased change or want the skill
  and the library code to always match exactly.

## Contributing to the skills

Skills are versioned alongside the library and welcome the same kind of contributions as the code — see [Contributing](https://github.com/retentioneering/retentioneering-tools/blob/master/CONTRIBUTING.md). If you change the public API, the product-analytics skill's `references/api-map.md` is version-verified against the checkout and should be re-checked; update both `.claude/skills/` and `.agents/skills/` copies together.

---

# Recipes

Source: https://retentioneering.com/docs/recipes

Short recipes that map common product questions to retentioneering tools. Each one is a starting point, not a full tutorial — follow the links for details on every tool involved. Throughout this page, `stream` is an [Eventstream](https://retentioneering.com/docs/eventstream) instance.

## Find a root cause in an anomalous period

A KPI metric dipped last week and you need to find the root cause? Create a dynamic [segment](https://retentioneering.com/docs/segments) that separates the anomalous window from normal operation, then [diff](https://retentioneering.com/docs/widgets#diff-mode) the two groups on any widget:

```python
stream = stream.add_segment("incident", time_range=("2024-03-10", "2024-03-17"))
stream.transition_graph(diff=("incident", "inside", "outside"))
```

The graph highlights exactly which transitions degraded during the window. [Segment Overview](https://retentioneering.com/docs/widgets/segment-overview) with the same segment scans many metrics at once when you don't yet know where to look. See [Segments](https://retentioneering.com/docs/segments#inside-vs-outside-an-anomalous-period) for why a *dynamic* segment is the right tool here.

## Open up the funnel

A [Funnel](https://retentioneering.com/docs/widgets/funnel) `add_to_cart` → `shipping_details` → `purchase` tells you 40% of paths drop between `shipping_details` and `purchase` — but not what those users did instead. Two moves recover the lost context.

First, look *between* the levels: trim each path to the window between two funnel steps and map what actually happens there:

```python
stream.truncate_paths(start_event="shipping_details", end_event="purchase").transition_graph()
```

Second, follow the users who never made it. The `funnel_events` mode of [Add Segment](https://retentioneering.com/docs/data-processors/add-segment) labels each path with the deepest funnel step it completed *in order* — reaching a step out of sequence, or skipping an earlier one, doesn't count — so drop-offs at each level become comparable groups:

```python
labelled = stream.add_segment("funnel", funnel_events=["add_to_cart", "shipping_details", "purchase"])
labelled.transition_graph(diff=("funnel", "shipping_details", "purchase"))
```

Also, you can use [Step Matrix](https://retentioneering.com/docs/widgets/step-matrix) or [Step Sankey](https://retentioneering.com/docs/widgets/step-sankey) along with the `path_pattern` parameter to explore paths around the funnel steps:

```python
stream.step_matrix(path_pattern="add_to_cart->.*->shipping_details->.*->purchase")
stream.step_sankey(path_pattern="add_to_cart->.*->shipping_details->.*->purchase")
```

## See which paths lead to conversion

Instead of assuming the intended route, reconstruct the routes that actually end in the target event. Trim every converting path to the window from its start to the first conversion, and drop the rest:

```python
stream.truncate_paths(start_event="path_start", end_event="purchase").step_sankey()
```

[Step Sankey](https://retentioneering.com/docs/widgets/step-sankey) unfolds the converging routes step by step; the [Transition Graph](https://retentioneering.com/docs/widgets/transition-graph) shows the same data as a map.

## Find what leads to churn

Conversion has a mirror image: which early behavior separates the users who came back from the ones who never did? Mark the outcome, label every path with it, then look only at the window you could have acted in.

```python
labelled = (
    stream
    # a synthetic event after 14 days of silence marks the moment a path went cold
    .add_events("churned", churn={"inactivity_days": 14})
    # ... which makes every path either churned or retained
    .add_segment(
        "outcome",
        sql="""
            SELECT CASE
                WHEN COUNT(*) FILTER (WHERE event = 'churned') OVER (PARTITION BY user_id) > 0
                THEN 'churned' ELSE 'retained'
            END
            FROM eventstream
        """,
    )
)

early = (
    labelled
    # keep each path's first 7 days — the onboarding window you can still influence
    .filter_events(
        sql="""
            SELECT * FROM eventstream
            QUALIFY timestamp < MIN(timestamp) OVER (PARTITION BY user_id) + INTERVAL 7 DAY
        """
    )
    # drop the marker itself: compare behavior, not the label you just derived
    .filter_events(drop={"event": ["churned"]})
)

early.step_matrix(diff=("outcome", "churned", "retained"))
early.transition_graph(diff=("outcome", "churned", "retained"))
```

The order matters: the label comes from the *whole* path, the comparison from its *first days*. That gap is the point — whatever the diff highlights happened before the outcome was decided, so it is a candidate cause rather than a symptom. Widen or narrow the window to find when the two groups start to diverge.

## Read an A/B test beyond the headline metric

The test metric reads flat, but you suspect behavior shifted somewhere the aggregate masks. Declare the test arm as a segment column and compare behavior, not just rates:

```python
stream.segment_overview(segment_col="ab_arm")            # scan many metrics for any difference
stream.step_matrix(diff=("ab_arm", "test", "control"))   # inspect where journeys diverge
```

To hunt for heterogeneity — an effect that exists only for part of the audience — narrow the stream first and diff again:

```python
stream\
    .filter_events(keep={"platform": ["mobile"]})\
    .step_matrix(diff=("ab_arm", "test", "control"))
```

## Compare acquisition channels

Users from different channels arrive with different intent, and their journeys show it. With the channel declared in the schema's [segment_cols](https://retentioneering.com/docs/eventstream#schema), every widget can compare one channel against all the others using the `<REST>` shorthand:

```python
stream.step_matrix(diff=("acquisition_channel", "paid_search", "<REST>"))
```
Also, you can use [Segment Overview](https://retentioneering.com/docs/widgets/segment-overview) to get a quick overview of the differences between the channels
comparing multiple metrics – such as path length, event count, conversion rate, and time to purchase – at once.

```python
stream.segment_overview(
    segment_col="acquisition_channel",
    metrics=[
        {"metric": "length"},
        {"metric": "event_count_bulk"},
        # has_event mean is equivalent to conversion rate
        {"metric": "has_event", "metric_args": {"event": "purchase"}},
        # median time to purchase (in seconds)
        {"metric": "time_between", "metric_args": {"start_event": "path_start", "end_event": "purchase"}, "agg": "median"},
    ]
)
```

## Compare newcomers with experienced users

A user is not the same person in their first session and in their fiftieth. Session number is a ready-made state: [Split Sessions](https://retentioneering.com/docs/data-processors/split-sessions) numbers each path's sessions, and a rule over that number turns the number into named stages.

The `split_sessions` call below assumes sessions aren't marked yet — drop it if your log already carries a session column (the bundled [e-commerce dataset](https://retentioneering.com/docs/eventstream#sample-dataset) does, and also ships a ready-made `user_lifecycle` segment along the same lines).

```python
labelled = (
    stream
    .split_sessions(timeout="30m")
    .add_segment(
        "experience",
        rules=[
            ["session_index", "=", 1, "first session"],
            ["session_index", "<=", 5, "getting familiar"],
            ["experienced"],
        ],
    )
)

labelled.transition_graph(diff=("experience", "first session", "experienced"))
labelled.segment_overview(segment_col="experience", metrics=[{"metric": "length"}])
```

Because the segment is dynamic, the *same* user contributes to several stages — you are comparing behavior at different points of a lifetime, not one cohort of people against another. Experienced users skipping the steps newcomers grind through is the usual finding, and the steps they skip are the ones worth shortening. [Segments](https://retentioneering.com/docs/segments#user-state) covers other state definitions, including the lifecycle labels from [To Daily States](https://retentioneering.com/docs/data-processors/to-daily-states).

## Discover your behavior types

How many kinds of users does the product actually have? [Cluster Analysis](https://retentioneering.com/docs/widgets/cluster-analysis) clusters paths by behavioral [path metrics](https://retentioneering.com/docs/path-metrics) and profiles each cluster interactively:

```python
stream.cluster_analysis()
```

Once the clusters look meaningful, persist them with [Add Clusters](https://retentioneering.com/docs/data-processors/add-clusters) — the labels become an ordinary segment, usable in diff mode and every other segment-aware tool:

```python
stream = stream.add_clusters(
    "behavior",
    features=[
        {"metric": "length"},
        {"metric": "active_days"},
        {"metric": "has_event", "metric_args": {"event": "purchase"}},
    ],
    n_clusters=4,
)
```

Note the difference in `n_clusters`: the widget searches a range (`"3-8"` by default) and picks the best split by silhouette score, while `add_clusters` materializes one specific clustering and therefore needs an exact number. You don't have to copy it by hand — the widget's "Save Clusters" button writes the matching `add_clusters(...)` call for you, and the headless twin returns the same thing under `best_params`:

```python
features = [{"metric": "length"}, {"metric": "active_days"}]

result = stream.cluster_analysis_data(features=features)
result["best_params"]                     # e.g. {"n_clusters": 3} — the winning split

stream = stream.add_clusters("behavior", features=features, **result["best_params"])
```

`best_params` carries only the searched-over parameters, so pass the same `features` alongside it.

## Measure time to activation

How long does it take a new user to reach the key action? The `time_between` [path metric](https://retentioneering.com/docs/path-metrics) computes it per path in one call:

```python
import pandas as pd

metrics = stream.get_metrics([
    {"metric": "time_between", "metric_args": {"start_event": "path_start", "end_event": "purchase"}},
])

# get_metrics() returns a DataFrame — one row per path, one column per metric
time_to_purchase = metrics["time_from_path_start_to_purchase"]
pd.to_timedelta(time_to_purchase, unit="s").median()
```

Paths that never reached `purchase` come back as `NaN`, so the median is over converting paths only — `time_to_purchase.notna().mean()` is the conversion rate that goes with it.

## Path as a sequence of sessions

A raw eventstream often has hundreds of granular events, which makes every path
long and hard to read so the widgets become overloaded and branchy. Instead of removing or collapsing raw events,
we can collapse each session down to one event named after its behavior. Ultimately, the resulting
eventstream has as many unique events as session types, and is far easier to
explore.

First, split paths into sessions if the eventstream doesn't have them yet:

```python
stream = stream.split_sessions(timeout="30m")
```

Then cluster sessions — not paths — by passing the session column as
`path_col` to [Cluster Analysis](https://retentioneering.com/docs/widgets/cluster-analysis), and use the
widget to find a splitting that makes sense and see what each cluster
actually contains.

```python
stream.cluster_analysis(
    path_col="session_id",
    features=[{"metric": "length"}, {"metric": "event_count_bulk"}],
    overview_metrics=[{"metric": "length"}, {"metric": "event_count_bulk"}],
)
```

Once the split looks right, label the clusters right in the UI and click "Save Clusters" to persist the clusters which will become a new column in the eventstream. This is equivalent to running [Add Clusters](https://retentioneering.com/docs/data-processors/add-clusters) with the same `path_col`, `features`, and `n_clusters` you settled on.

Finally, collapse each session into a single event named after its type with
[Collapse Events](https://retentioneering.com/docs/data-processors/collapse-events):

```python
stream = stream.collapse_events(session_col="session_id", session_type_col="session_type")
```

The new eventstream has one synthetic event per session, and the number of unique
events equals the number of session types from clustering — small enough to
read a [Step Matrix](https://retentioneering.com/docs/widgets/step-matrix) or [Transition Graph](https://retentioneering.com/docs/widgets/transition-graph) at a glance.

## Extract behavioral features for ML

Churn and LTV models improve when they see *behavior*, not just demographics. `get_metrics()` turns any set of [path metric](https://retentioneering.com/docs/path-metrics) configs into a clean per-path feature table, ready to join with your training data:

```python
features = stream.get_metrics([
    {"metric": "length"},
    {"metric": "duration"},
    {"metric": "active_days"},
    {"metric": "event_count_bulk"},
    {"metric": "matches_pattern", "metric_args": {"pattern": "home->.*->purchase"}},
])
```

---

# Migrating from 3.x

Source: https://retentioneering.com/docs/migration-from-3x

retentioneering 5.0 is a ground-up rewrite, not an incremental release. If you are following a tutorial, a blog post, or a notebook written for 3.3.0, most of the code in it will not run as-is. This page maps the old API onto the new one so you can tell, method by method, whether something was renamed, replaced, or dropped.

If you are new to the library, you can skip this page entirely — start with the [Quick Start](https://retentioneering.com/docs/quick-start).

## What changed, in one paragraph

The pandas-based engine was replaced with a DuckDB-backed one, so the same analyses run at production-log scale on a laptop. The old iframe + CDN-loaded widgets were replaced with [anywidget](https://anywidget.dev)-based ones whose JavaScript is open source, lives in this repository, and ships inside the wheel — nothing is downloaded at runtime, and widgets now work in VS Code and Cursor as well as Jupyter. The `data_processor` / `preprocessor` / `params_model` machinery is gone: data processors are now plain methods with plain keyword arguments. And two things are new with no 3.x counterpart — an [MCP server](https://retentioneering.com/docs/mcp-server) that lets an LLM agent drive the analysis, and the [Segment Overview](https://retentioneering.com/docs/widgets/segment-overview) widget.

`CHANGELOG.md` in the repository holds the exhaustive, itemized delta under `[5.0.0]`.

## Should you upgrade?

Upgrade if you want the new engine, the new widgets, agent support, or ongoing fixes. **Stay on 3.x** if your work depends on the Preprocessing Graph, `Cohorts`, `StatTests`, or `Sequences` — none of them exist in 5.x yet (see [What is gone](#what-is-gone) below).

The 3.x engine is preserved on the [`3.x` branch](https://github.com/retentioneering/retentioneering-tools/tree/3.x) for reference and patches:

```bash
pip install "retentioneering<4"     # the 3.x line
pip install -U retentioneering      # 5.x
```

The two versions install under the same package name, so they cannot coexist in one environment — use separate virtualenvs if you need both.

## Concepts that shifted

Four changes explain most of the surprises when porting old code.

**Paths, not users.** 3.x was built around users. 5.x is built around *paths*, and a path is whatever grain you declare: `path_col="user_id"` reproduces the old behavior, `path_col="session_id"` switches the same eventstream to session grain without rebuilding it. See [Key concepts](https://retentioneering.com/docs/eventstream#key-concepts).

**Everything is immutable.** Every data processor returns a **new** `Eventstream` and never modifies the one it was called on. Code written against 3.x's in-place semantics needs to capture the result:

```python
stream = stream.filter_events(drop={"event": ["checkout_bug"]})   # not just stream.filter_events(...)
```

This is also why `copy()` is gone — there is nothing to protect against.

**Segments replace ad-hoc grouping.** A *segment column* describes a whole split at once (`US` / `DE` / `FR`), not one group, and its values live per event — so a segment can change along a path. Segments drive comparison everywhere: every core widget takes `diff=(segment_col, value1, value2)`. See [Segments](https://retentioneering.com/docs/segments).

**Every widget has a headless twin.** `stream.step_matrix()` renders; `stream.step_matrix_data()` returns the DataFrame behind it, with the same data arguments. If you used 3.x tools to get numbers rather than pictures, the `*_data()` methods are what you want. See [Headless mode](https://retentioneering.com/docs/widgets#headless-mode).

## Renamed or re-signatured

Same concept, different call shape.

| 3.3.0 | 5.x | Note |
|---|---|---|
| `to_dataframe(copy=False)` | [`to_dataframe(exclude_start_end=True)`](https://retentioneering.com/docs/eventstream#getting-your-data-back-out) | The result is always a snapshot, so there is no `copy` flag. |
| `filter_events(func)` | [`filter_events(keep=, drop=, func=, sql=)`](https://retentioneering.com/docs/data-processors/filter-events) | `func` is now one of four alternative modes; `keep`/`drop` cover the common cases without a lambda. |
| `rename(rules: list[dict])` | [`rename_events(mapping: dict)`](https://retentioneering.com/docs/data-processors/rename-events) | A plain `{old: new}` dict. |
| `collapse_loops(suffix, time_agg)` | [`collapse_events(consecutive=, event_groups=, ...)`](https://retentioneering.com/docs/data-processors/collapse-events) | Loop-squashing is now one mode of a general merging processor. |
| `group_events` / `group_events_bulk` | [`collapse_events(event_groups=...)`](https://retentioneering.com/docs/data-processors/collapse-events) or [`rename_events`](https://retentioneering.com/docs/data-processors/rename-events) | Renaming several events to one name is `rename_events`; merging runs of them is `collapse_events`. |
| `split_sessions(timeout, delimiter_events, ...)` | [`split_sessions(timeout=, separator=, start_event=, end_event=, ...)`](https://retentioneering.com/docs/data-processors/split-sessions) | `timeout` now needs an explicit unit — `"30m"` or a `pd.Timedelta`. Bare numbers are rejected. |
| `truncate_paths(drop_before, drop_after, ...)` | [`truncate_paths(start_event, end_event)`](https://retentioneering.com/docs/data-processors/truncate-paths) | Window anchors are the `start_event`/`end_event` pair everywhere in 5.x. |
| `drop_paths()` | [`filter_paths(condition)`](https://retentioneering.com/docs/data-processors/filter-paths) | A condition tree over [path metrics](https://retentioneering.com/docs/path-metrics) — `{"op": ">", "metric": "length", "value": 5}` — instead of fixed thresholds. |
| `add_positive_events` / `add_negative_events` | [`add_events(name, source_events=[...])`](https://retentioneering.com/docs/data-processors/add-events) | One processor for synthetic events; the positive/negative distinction was only naming. |
| `label_new_users` / `label_lost_users` / `label_cropped_paths` | [`add_segment`](https://retentioneering.com/docs/data-processors/add-segment), [`add_events(churn=...)`](https://retentioneering.com/docs/data-processors/add-events), [`to_daily_states`](https://retentioneering.com/docs/data-processors/to-daily-states) | Labelling is a segment; a churn marker is a synthetic event; lifecycle states are their own processor. |
| `clusters()` (stateful `fit()`/`extract_features()`) | [`cluster_analysis()`](https://retentioneering.com/docs/widgets/cluster-analysis) + [`add_clusters()`](https://retentioneering.com/docs/data-processors/add-clusters) | Explore interactively, then persist the split as a segment column. |
| `transition_matrix()` | [`transition_graph_data()`](https://retentioneering.com/docs/widgets/transition-graph#headless-mode) | Same matrix, now the headless twin of the graph. |
| `describe()` / `describe_events()` | [`describe()`](https://retentioneering.com/docs/eventstream#inspecting-your-data) | One dict: schema, shape, date range, event frequency, path length/duration stats. |
| `add_custom_col()` / `index_events()` | `schema={"custom_cols": [...]}` | Extra columns ride along automatically; declare them only if you want strict control. |
| `append_eventstream()` | `pd.concat(...)` before constructing | Combine the frames, then build one `Eventstream`. |
| `pipe()` | plain chaining, or [`recipe()`](https://retentioneering.com/docs/eventstream#reproducing-an-eventstream) | Processors chain directly; `recipe()`/`from_recipe()` replay a pipeline elsewhere. |

Widget classes were reimplemented from scratch, but kept their names and are reached the same way — as methods on the eventstream: `transition_graph`, `step_matrix`, `step_sankey`, `funnel`, and `cluster_analysis` (3.x's `Clusters`).

## What is gone

No equivalent in 5.x today. These were cut deliberately to get the rewrite shipped, and may return — requests and pull requests are welcome on the [issue tracker](https://github.com/retentioneering/retentioneering-tools/issues).

| Gone | Closest thing available now |
|---|---|
| **Preprocessing Graph** (visual no-code pipeline builder) | Chained data processors in code; for agents, the [MCP server](https://retentioneering.com/docs/mcp-server)'s preprocessor lists. |
| **Cohorts** | [`add_segment`](https://retentioneering.com/docs/data-processors/add-segment) over a signup-period column plus [Segment Overview](https://retentioneering.com/docs/widgets/segment-overview); [`to_daily_states`](https://retentioneering.com/docs/data-processors/to-daily-states) for lifecycle-state retention. |
| **StatTests** | Nothing built in — pull per-path values with [`get_metrics()`](https://retentioneering.com/docs/eventstream#per-path-metrics-as-a-feature-table) and run your own test in `scipy`. |
| **Sequences** | The `matches_pattern` [path metric](https://retentioneering.com/docs/path-metrics) and `path_pattern` on [Step Matrix](https://retentioneering.com/docs/widgets/step-matrix) / [Step Sankey](https://retentioneering.com/docs/widgets/step-sankey) answer pattern questions, but there is no n-gram frequency table. |
| `timedelta_hist`, `user_lifetime_hist`, `event_timestamp_hist` | [`get_metrics()`](https://retentioneering.com/docs/eventstream#per-path-metrics-as-a-feature-table) returns `duration`, `time_between`, `first_event_time` and friends per path — plot them with your own matplotlib/plotly call. `describe()` gives the percentiles directly. |

## Naming conventions

5.x follows one vocabulary throughout ([ADR-0008](https://github.com/retentioneering/retentioneering-tools/blob/master/docs/adr/0008-naming-conventions.md)), which is worth skimming before you port a pipeline:

- Column arguments are always `path_col`, `event_col`, `timestamp_col`, `session_col`, `segment_col`.
- Window anchors are always the `start_event` / `end_event` pair.
- Durations are strings with an explicit unit (`"30m"`) or a `pd.Timedelta`; time *outputs* are seconds.
- `<REST>` is the diff sentinel for "every other value of this segment".
- Data processors are verb-first (`filter_events`), widgets are nouns (`funnel`), headless twins are `<widget>_data`.

## Requirements

5.x requires **Python 3.10 or later** (3.3.0 supported 3.8–3.11). Widgets render in Jupyter, JupyterLab, VS Code, Cursor, and Google Colab — see [Installation](https://retentioneering.com/docs/installation) for the one JupyterLab caveat.

## Next steps

- [Quick Start](https://retentioneering.com/docs/quick-start) — the 5.x pipeline end to end in five minutes.
- [Path Analysis](https://retentioneering.com/docs/path-analysis) — what each visualization actually computes, and when to trust it.
- [Eventstream](https://retentioneering.com/docs/eventstream) — schema, path grain, and the methods that aren't processors or widgets.

---

# Tracking

Source: https://retentioneering.com/docs/tracking

retentioneering collects anonymous usage analytics to help improve the library. This page explains exactly what is tracked and how to opt out.

## What we track

We track **method calls and widget actions only** — the fact that a method was called and whether it succeeded, never the data it operated on.

**Eventstream creation** (`eventstream_created`). The properties collected are exactly:

- dataset shape: the number of rows and the number of columns,
- the number of path columns, segment columns, and event columns declared in the schema — counts only, never the column names themselves.

This event does **not** carry the `default_args`/`non_default_args` split described below — `Eventstream(df, schema=...)` itself isn't wrapped, so there's no per-argument report of what you passed to the constructor, only these derived shape metrics.

**Data processor calls** (`dp_filter_events`, `dp_collapse_events`, ...) — which processor was called.

**Widget and headless method calls** (`widget_transition_graph`, `headless_step_matrix`, ...) — which visualization or `*_data` method was used.

**MCP server start** (`mcp_serve`) — whether a semantic-layer `context` was supplied (`has_context`, a boolean, never its contents).

For every tracked method call other than `eventstream_created` we also record:

- **Parameter names, never values.** Parameters that have a default are split into two lists, `default_args` and `non_default_args` — so we learn, for example, that `funnel_events` was customized, but never what it was set to. Values are never sent, since they may contain column names, event names, or query text.
- **Call status.** If the call raises, the event carries `status: "error"` plus `error_type` — either the library's stable internal error code (for `retentioneering.exceptions.RetentioneeringError` subclasses, e.g. `EMPTY_EVENTSTREAM`) or the exception's class name — never the exception message, which could echo your data back (e.g. a DuckDB error quoting the offending SQL or column name). On success the event carries `status: "success"`.

Every event also records whether the call was made by your code or by an agent through the MCP server.

## What we do not track

We **never** collect any sensitive data from your eventstream — no event names, no user identifiers, no path contents, no segment values, and no business metrics. We also never collect method parameter values or exception messages. Your data stays entirely on your machine.

We don't collect anything about your environment beyond: an anonymous device identifier (a one-way hash derived from the machine id, or a random UUID stored in `~/.retentioneering/config.json`, from which nothing about you or your machine can be recovered), the library version, OS and OS version, Python version, the runtime environment (Jupyter, VS Code, Google Colab, or script), the Jupyter kernel id when running inside a notebook, and — for plain scripts — a one-way hash of the invoked script's path and a separate one-way hash of the full command line (arguments included). The latter two let you tell apart events from different scripts or automated agents sharing one machine (which would otherwise share the same device identifier above); the actual path and arguments are never sent, only the hashes.

## Opting out

Set the environment variable `RETENTIONEERING_NO_TRACK=1` before starting your notebook kernel:

```bash
# In your shell profile (.zshrc, .bashrc)
export RETENTIONEERING_NO_TRACK=1

# Or at the top of a notebook cell
import os
os.environ["RETENTIONEERING_NO_TRACK"] = "1"
```

### Google Colab

Add a secret named `RETENTIONEERING_NO_TRACK` with value `1` in Colab → Settings → Secrets, then enable notebook access for it. The secret persists across all Colab sessions automatically.

---

# Transition Graph

Source: https://retentioneering.com/docs/widgets/transition-graph

Displays an interactive directed graph where nodes are unique events and edges represent transitions
between them. Edge weights can show transition probabilities, counts, or time-based metrics.
Supports diff mode to compare two user segments side by side.

## How it works

Take every adjacent pair of events in every path and count the pairs
([Path Analysis](https://retentioneering.com/docs/path-analysis) puts this next to the other summaries).
Each distinct event becomes one node, no matter where in the path it occurred,
and each pair becomes a weighted edge.

![Five paths on the left; on the right their adjacent event pairs counted into a transition graph with weighted edges and self-loops](https://retentioneering.com/docs-demos/img/event-aggregation.svg "Building a transition graph from paths. Edge weight is the number of transitions.")

What this buys you, and what it costs:

- **Loops become visible.** An event repeating back to back turns into a
  self-loop, and a cycle between screens becomes an actual cycle on the graph.
  No step-based view shows this, and going in circles is one of the most common
  things real users do.
- **The picture stays finite.** One node per distinct event, however long the
  paths — so the graph works as a map of the product.
- **Position is gone.** A `cart → purchase` edge says the transition happened,
  not whether it happened on step 3 or step 30. When timing within the path is
  the question, use [Step Matrix](https://retentioneering.com/docs/widgets/step-matrix) or
  [Step Sankey](https://retentioneering.com/docs/widgets/step-sankey) instead.

The default edge weight, `proba_out`, is a conditional probability: of everything
that happens *after* the source event, what share goes to the target. Edges out
of a node therefore sum to 1, and the graph reads as a Markov-style map. Switch
to `unique_paths` or `count` when you care about volume rather than likelihood,
or to `time_median` to find the slow transitions — see
[Edge weights](#edge-weights) below.

Edges are filtered before drawing (by default, each node's few strongest
outgoing edges), otherwise a real eventstream renders as a hairball. That filter
is a *display* choice — the underlying matrix always has every transition, and
`transition_graph_data()` returns it in full.

## Usage

```python
stream.transition_graph()
stream.transition_graph(edge_weight="count", diff=("user_lifecycle", "loyal", "new"))
stream.transition_graph(state_file="checkout_graph.json")
stream.transition_graph(
    views=[{"name": "Checkout", "focus": {"type": "node", "event": "cart"}}],
    view="Checkout",
)
```

## Examples

### Basic
```python
stream.transition_graph()
```

### Edge weight
```python
stream.transition_graph(edge_weight="unique_paths")
```

### Diff mode
```python
stream.transition_graph(diff=["platform", "mobile", "desktop"])
```

### Views
```python
stream.transition_graph(
    views=[
        {
            "name": "Cart",
            "focus": {"type": "node", "event": "cart"},
            "edgeFilter": {"mode": "topk", "k": 2}
        },
        {
            "name": "Checkout path",
            "focus": {
                "type": "path",
                "nodes": ["cart", "shipping_details", "payment_details", "purchase"]
            }
        },
        {
            "name": "Big picture",
            "edgeFilter": {"mode": "range", "range": [0.10, 1]},
            "eventCountFilter": [500, 7000]
        },
    ],
    view="Checkout path",
    sidebar_open=False
)
```

### Ego view
```python
stream.transition_graph(
    views=[
        {
            "name": "Cart",
            "focus": {"type": "node", "event": "cart"},
            "egoNode": "cart"
        },
    ],
    view="Cart"
)
```


## Parameters

### Data

Data parameters change the computed result. They are exactly the arguments of
the widget's headless twin `stream.transition_graph_data()` — see
[headless mode](#headless-mode) below.

| Parameter | Type | Description |
|---|---|---|
| `edge_weight` | `{"proba_out", "proba_in", "count", "unique_paths", "share_of_total", "avg_per_path", "time_median", "time_q95"}, default "proba_out"` | Value shown on edges. See the [Edge Weights](https://retentioneering.com/docs/widgets/transition-graph#edge-weights) section for more details. |
| `diff` | `tuple or list, optional` | Draws a comparative chart for a pair of segments; see [Diff mode](https://retentioneering.com/docs/widgets#diff-mode). `(segment_col, value1, value2)` or `(path_ids1, path_ids2)`; `value2` may be `<REST>`, meaning "every other value of `segment_col`". |
| `path_col` | `str, optional` | Path ID column override; defaults to `schema.path_col`. |

### Display

Display parameters only affect how the widget is rendered.

| Parameter | Type | Description |
|---|---|---|
| `height` | `int, default 500` | Widget height in pixels. |
| `sidebar_open` | `bool, default True` | Whether the sidebar starts open. |
| `views` | `list of dict, optional` | Named visual presets rendered as pills above the graph. Each view is a dict with the keys `name`, `focus`, `edgeFilter`, `eventCountFilter`, `hiddenEvents`, `viewport` (all optional except `name`) and describes only how the graph is *displayed* — never the computed data. See the [Views](https://retentioneering.com/docs/widgets/transition-graph#views) section for a detailed description of every key. |
| `view` | `dict or str, optional` | View applied once after the graph is built: a view dict (`name` not required), or the name of an entry in `views`. See the [Views](https://retentioneering.com/docs/widgets/transition-graph#views) section. |
| `state_file` | `str, optional` | JSON file the widget state is bound to; see [Saving widget state](https://retentioneering.com/docs/widgets#saving-widget-state). |

### Edge weights

- `"proba_out"` — probability of the transition among all transitions out of the source event.
- `"proba_in"` — probability of the transition among all transitions into the target event.
- `"count"` — number of times the transition occurred.
- `"unique_paths"` — number of distinct paths containing the transition.
- `"share_of_total"` — share of this transition among all transitions in the eventstream.
- `"avg_per_path"` — average number of occurrences per path.
- `"time_median"` / `"time_q95"` — median / 95th-percentile time between the two events (in seconds).

### Views

`views` is a list of named **visual presets** rendered as pills above the
graph; `view` applies one of them (or an inline preset) right after the graph
is built. A view describes only how the graph is *displayed* — focus,
filters, hidden events, viewport — never the computed data, so the same view
works identically in the live widget, in exported HTML (including the
`#view=<base64url>` URL fragment) and in MCP report tabs
(`[Tab:view=Name]` links in the analysis text).

Applying a pill always starts from the Default state — views do not inherit
each other's settings — and the Default pill returns exactly the state the
widget had before the first view was applied. Any manual interaction
(clicking a node, moving a filter) deselects the active pill.

| Key | Type | Description |
|---|---|---|
| `name` | `str` | Pill label; also the handle for `view="Name"` and `[Tab:view=Name]` links. Required for pills, optional for inline `view=` presets. |
| `focus` | `dict, optional` | Dim everything except the focused element(s): a node, an edge, or a path. See [focus](#focus). |
| `edgeFilter` | `dict, optional` | Edge filter override — the Auto/Manual control at the bottom right. See [edgeFilter](#edgefilter). |
| `eventCountFilter` | `[min, max], optional` | Event-count (node) filter in absolute occurrence counts — the sidebar's Event Count slider. |
| `hiddenEvents` | `list of str, optional` | Events hidden in this view. |
| `viewport` | `str or dict, optional` | Camera control. See [viewport](#viewport). |
| `nodePositions` | `dict, optional` | Exact node coordinates (`{"event": {"x": …, "y": …}}`) the view pins on apply. Rarely needed by hand: manual arrangements persist on their own (per-dataset) and travel with exports. |
| `egoNode` | `str, optional` | Opens the [Ego view](#ego-view) modal for this event on apply. Independent of `focus` — a view can set both, or just this. Not captured by the Copy view link button (the modal covers the toolbar while it's open) — write it by hand. |

#### focus

- `{"type": "node", "event": "cart"}` — focus one event: everything else dims
  and the viewport fits the node together with its neighborhood. One
  direction is shown at a time by default: outgoing transitions,
  `"direction": "in"` for incoming, `"direction": "both"` for both at once
  (no color code — direction is read off the existing arrowheads). In the
  widget, clicking a node shows its outgoing transitions; clicking that same
  node again cycles to incoming, then to both, then back to outgoing. Diff
  mode always shows both directions (red/blue).
- `{"type": "edge", "source": "cart", "target": "checkout"}` — focus one
  transition: the edge and its two nodes stay bright (the edge is shown even
  when the edge filter would hide it), the weight label is forced on, and the
  viewport fits the pair.
- `{"type": "path", "nodes": ["main", "catalog", "cart"]}` — highlight a
  route of two or more events: consecutive transitions are drawn as an amber
  dashed overlay (visible through the edge filter), weight labels appear
  along the route, and everything outside the route dims. Segments with no
  direct transition in the data are skipped.

Omitting `focus` means "no focus" — the view only applies filters and the
viewport.

#### edgeFilter

- `{"mode": "topk", "k": 3}` — the Auto mode: keep each node's `k` strongest
  outgoing edges (plus every node's single strongest incoming edge, so no
  node is left unconnected).
- `{"mode": "range", "range": [0.05, 1]}` — the Manual mode: a weight range
  normalized to 0..1. For probability weights the bounds are the
  probabilities themselves; for other weight types they are fractions of the
  largest weight on the graph.

#### viewport

- `"fit"` — fit the whole graph.
- `"fit-focus"` — fit the focused node/edge/path; this is the default
  whenever `focus` is present.
- `{"zoom": 1.4, "pan": {"x": 350, "y": 220}}` — an exact camera position.

The easiest way to author a view is the **Copy view link** toolbar button:
arrange the graph by hand and click it — the current state (including the
exact viewport) is copied as a JSON snippet for `views=[...]` in the live
widget, or as a shareable `#view=` URL in exported HTML. To author a *path*
view by mouse: click the route's first event, then Cmd/Ctrl+click the
following events **in route order** — after every click, the outgoing edges
of the route's last event stay lit, so the candidates for the next click are
always visible. Every Cmd/Ctrl+click *appends*, so repeated events build
loops and cycles (`A→B→B`, `A→B→A→C`); Cmd/Ctrl+**double-click on the
route's last event** removes it — a backspace-like undo. Then Copy view link serializes the selection as
`{"type": "path", ...}`. Note that transitions are directed: the route
follows the click order, and segments with no direct transition in the data
are skipped when the view is applied.

### Route statistics

While a path of two or more events is selected in the live widget, a badge
at the top shows statistics for that exact route — how many paths contain
the sequence `A→B→…→N` as strict consecutive transitions (no other events
in between). The badge's metric selector offers: **unique paths** (and their
share of all paths), **traversals** (total occurrences; a path may traverse
a route several times), **avg per path**, **median** / **p95 route time**
(from the route's first to last event within one traversal), and
**P(route)** — the product of the route's `P(next | source)` transition
probabilities. The default metric follows the current edge weight.

### Ego view

With a node focused, the **Ego view** toolbar button expands its
neighborhood into a modal mini-sankey: the event in the center, incoming
transitions on the left, outgoing on the right, sorted with ribbon
thickness proportional to the value. The sides answer the two ego
questions directly, independent of the edge weight selected on the graph:
the left side shows each source's **share of the arrivals** at the event
(`proba_in`), the right side each target's **share of the exits** from it
(`proba_out`) — both sides sum to 100%, and the tooltip carries the raw
transition counts. Unlike the graph, the same event can appear on both
sides, and a self-loop shows up on each (marked ↻). The view is not
limited by the edge filter — it always shows the strongest transitions of
the focused event (up to 12 per side). Clicking a neighbor re-centers the
view on it, so you can walk the graph one neighborhood at a time; Esc or
clicking outside closes the modal. In diff mode the ribbons instead show
the displayed diff values with the red/blue color code, and hovering one
shows the same per-group breakdown tooltip as a graph edge — both group
values and the diff between them.

## Headless mode

`stream.transition_graph_data()`

Compute the transition **matrix** between events (headless): an
events x events DataFrame where cell `[source, target]` holds the selected
`edge_weight` for the `source -> target` transition. This is the data
behind the `transition_graph` widget.

---

# Step Matrix

Source: https://retentioneering.com/docs/widgets/step-matrix

Heatmap of what users are doing at each step of their path: cell
`[event, step]` is the share of paths on that event at that step, so every
column sums to 1 (to 0 in diff mode). Rows are events, columns are steps
from the anchor — `path_start` by default, or a `path_pattern` event, in
which case the columns to its left are negative.

Same numbers as [Step Sankey](https://retentioneering.com/docs/widgets/step-sankey), drawn as a table.

## How it works

Stack every path up, slice the stack by step, and count: what share of paths is
sitting on each event at step 1, at step 2, and so on? That table is the Step
Matrix — one of the three aggregated representations of trajectories described in
[Path Analysis](https://retentioneering.com/docs/path-analysis).

![Five paths merged into a trajectory tree, each node carrying the number of paths passing through it](https://retentioneering.com/docs-demos/img/trajectory-tree.svg "Five example paths stacked into a trajectory tree: identical beginnings share a branch, and each node carries the number of paths passing through it.")

![The step matrix of those five paths: one row per event, one column per step](https://retentioneering.com/docs-demos/img/step-matrix.svg "The same tree counted by step. The two cart nodes at step 3 — one reached via catalog, one via search — merge into a single cell: 2 + 1 = 3 of 5 paths.")

Reading it:

- **A cell is a share of paths, not of events.** Row `cart`, column 3 = 0.6 means
  60% of all paths had `cart` as their third event.
- **Every column sums to 1** (in diff mode, to 0), because every path is
  somewhere at every step. Paths that have ended sit in `path_end`, which is why
  that row fills up towards the right.
- **Branches merge.** Two groups of users reaching `cart` at step 3 by different
  routes land in the same cell. The matrix tells you *what* happens at each step,
  not which route got there.

Step Matrix and [Step Sankey](https://retentioneering.com/docs/widgets/step-sankey) are two renderings of
exactly the same numbers — they share one headless method. The matrix is easier
for reading a single event across steps and for spotting rare events; the Sankey
is easier for following volume.

### Anchoring on an event

By default step 1 is each path's own first event. `path_pattern` re-anchors the
matrix on an event instead: every path is shifted until that event sits in
column 0, and the steps before it get negative numbers.

![Five paths shifted so that each path's cart event sits in column zero, with negative column numbers before it](https://retentioneering.com/docs-demos/img/path-pattern.svg 'Five example paths under path_pattern="cart": every path shifted until its cart event sits in column 0. Paths that reach the anchor later stick out to the left; paths that never reach it drop out of the view.')

Column −1 then answers "what leads into this event?" and column +1 "what follows
it?" — questions the start-anchored view smears across many columns whenever the
event happens at different steps for different users.

A pattern with several anchors (`"add_to_cart->.*->purchase"`, where `.*` matches
any run of events) draws one block of columns per anchor, side by side, so you
can walk a funnel and still see each step's neighbourhood. When a block's edge
isn't a real path boundary the widget draws it serrated, signalling that paths
continue beyond the visible range.

## Usage

```python
stream.step_matrix(path_pattern="purchase")
stream.step_matrix(path_pattern="add_to_cart->.*->purchase")
stream.step_matrix(diff=("user_lifecycle", "new", "loyal"))
```

## Examples

### Basic
Without a `path_pattern`, the matrix is anchored at `path_start`: column 0 is the first
step of every path, and each column to the right is one step further in.

```python
stream.step_matrix()
```

### Path pattern - central event
A single event as `path_pattern` re-centers the matrix on that event instead of
`path_start`: column 0 is always `purchase`, negative columns are the steps leading up
to it, positive columns are what follows.

```python
stream.step_matrix(path_pattern="purchase")
```

### Path pattern - funnel patterns
Chaining several anchors with `.*` renders one matrix block per anchor, side by side,
so you can follow a multi-step funnel across blocks.

```python
stream.step_matrix(path_pattern="payment_details->.*->shipping_details->.*->purchase", step_window=2)
```

### Path pattern - drop-off points
`path_pattern="path_end"` centers on where paths end, so the columns to the left show
what users tend to do right before dropping off (or completing their path).

```python
stream.step_matrix(path_pattern="path_end")
```

### Diff mode
Passing `diff` splits paths into two segments and shows the difference between them at
each step, making it easy to spot where the segments' behavior diverges.

```python
stream.step_matrix(diff=["platform", "mobile", "desktop"])
```


## Parameters

### Data

Data parameters change the computed result. They are exactly the arguments of
the widget's headless twin `stream.step_matrix_data()` — see
[headless mode](#headless-mode) below.

| Parameter | Type | Description |
|---|---|---|
| `max_steps` | `int, default 10` | Number of path steps to compute on each side of the anchor. |
| `diff` | `tuple or list, optional` | Draws a comparative chart for a pair of segments; see [Diff mode](https://retentioneering.com/docs/widgets#diff-mode). `(segment_col, value1, value2)` or `(path_ids1, path_ids2)`; `value2` may be `<REST>`. |
| `path_col` | `str, optional` | Path ID column override; defaults to `schema.path_col`. |
| `path_pattern` | `str, optional` | Restrict/split paths using a `"->"`-separated sequence of anchor events, where `.*` matches any run of events, e.g. `"add_to_cart->.*->purchase"`. Without a pattern, shows the whole path from `path_start` to `path_end`. Multiple anchors render one matrix block per anchor, side by side. A pattern that doesn't start at `path_start` or end at `path_end` shows a serrated edge, signalling paths continue beyond the visible range. To see the neighborhood around a single event: `path_pattern="add_to_cart"`. |

### Display

Display parameters only affect how the widget is rendered.

| Parameter | Type | Description |
|---|---|---|
| `step_window` | `int, default 3` | Number of step columns shown around each anchor. |
| `height` | `int, default 600` | Widget height in pixels. |
| `sidebar_open` | `bool, default True` | Whether the sidebar starts open. |
| `state_file` | `str, optional` | JSON file the widget state is bound to; see [Saving widget state](https://retentioneering.com/docs/widgets#saving-widget-state). |

## Headless mode

`stream.step_matrix_data()`

Alias for `step_sankey_data` — Step Matrix and Step Sankey render the
same underlying per-step data, so both widgets share one headless method.

---

# Step Sankey

Source: https://retentioneering.com/docs/widgets/step-sankey

Flow diagram of what users are doing at each step of their path: block
height is the share of paths on that event at that step (columns sum to 1,
or to 0 in diff mode), and ribbons show how volume moves between two
adjacent steps. Ribbons do not chain — paths arriving by different routes
merge at every column.

Same numbers as [Step Matrix](https://retentioneering.com/docs/widgets/step-matrix), drawn as flows.

## How it works

Step Sankey draws the same numbers as the
[Step Matrix](https://retentioneering.com/docs/widgets/step-matrix): all paths stacked up, sliced by step,
counted per event (see [Path Analysis](https://retentioneering.com/docs/path-analysis)). Both widgets share
one headless method — the difference is only in the rendering.
A column of the matrix becomes a column of stacked blocks whose heights are those
shares, and the ribbons between two columns show how volume moves from one step
to the next.

![The step matrix of five example paths: one row per event, one column per step](https://retentioneering.com/docs-demos/img/step-matrix.svg "The numbers behind both widgets, on five example paths: the share of paths on each event at each step. Step Sankey draws each column as stacked blocks and the moves between columns as ribbons.")

Reading it:

- **Block height is a share of paths** at that step; every column sums to 1 (to 0
  in diff mode).
- **A ribbon is a transition between two adjacent steps** — how many paths that
  were on event A at step *k* are on event B at step *k+1*.
- **Ribbons do not chain.** Following three ribbons in a row does *not* give the
  share of users who took that three-event route: at every column, paths arriving
  by different routes merge into the same block. For an exact sequence use
  `path_pattern`, the `matches_pattern` [path metric](https://retentioneering.com/docs/path-metrics), or the
  Transition Graph's route statistics badge.

Use the Sankey when the question is about volume flowing forward, and the matrix
when you want to read one event across many steps or catch rare events that a
thin ribbon would hide.

`path_pattern` re-anchors the diagram on an event, so that column 0 is always
that event and negative columns are what led up to it — see
[anchoring](https://retentioneering.com/docs/widgets/step-matrix#anchoring-on-an-event) for the full picture.

## Usage

```python
stream.step_sankey(max_steps=15, path_pattern="add_to_cart->.*->purchase")
stream.step_sankey(diff=("acquisition_channel", "paid_search", "<REST>"))
```

## Examples

### Basic
```python
stream.step_sankey()
```

### Path pattern - central event
A single event as `path_pattern` re-centers the diagram on that event instead of
`path_start`: column 0 is always `purchase`, negative columns are the steps
leading up to it, positive columns are what follows.

```python
stream.step_sankey(path_pattern="purchase")
```

### Path pattern - funnel patterns
Chaining several anchors with `.*` renders one diagram block per anchor, side by side,
so you can follow a multi-step funnel across blocks.

```python
stream.step_sankey(path_pattern="payment_details->.*->shipping_details->.*->purchase", step_window=2)
```

### Path pattern - drop-off points
`path_pattern="path_end"` centers on where paths end, so the columns to the left show
what users tend to do right before dropping off (or completing their path).

```python
stream.step_sankey(path_pattern="path_end")
```

### Diff mode
```python
stream.step_sankey(diff=["platform", "mobile", "desktop"])
```



## Parameters

### Data

Data parameters change the computed result. They are exactly the arguments of
the widget's headless twin `stream.step_sankey_data()` — see
[headless mode](#headless-mode) below.

| Parameter | Type | Description |
|---|---|---|
| `max_steps` | `int, default 10` | Number of path steps to compute. |
| `diff` | `tuple or list, optional` | Draws a comparative chart for a pair of segments; see [Diff mode](https://retentioneering.com/docs/widgets#diff-mode). `(segment_col, value1, value2)` or `(path_ids1, path_ids2)`; `value2` may be `<REST>`. |
| `path_col` | `str, optional` | Path ID column override; defaults to `schema.path_col`. |
| `path_pattern` | `str, optional` | Same syntax as `step_matrix`'s `path_pattern`. |

### Display

Display parameters only affect how the widget is rendered.

| Parameter | Type | Description |
|---|---|---|
| `step_window` | `int, default 3` | Number of step columns shown around each anchor. |
| `height` | `int, default 500` | Widget height in pixels. |
| `sidebar_open` | `bool, default True` | Whether the sidebar starts open. |
| `state_file` | `str, optional` | JSON file the widget state is bound to; see [Saving widget state](https://retentioneering.com/docs/widgets#saving-widget-state). |

## Headless mode

`stream.step_sankey_data()`

Compute per-step event-share matrices for Step Matrix / Step Sankey (headless).

Both widgets render the same underlying data — Step Matrix as a heatmap,
Step Sankey as a flow diagram.

---

# Funnel

Source: https://retentioneering.com/docs/widgets/funnel

Interactive conversion funnel for Jupyter notebooks.

A path is counted at step N if it contains that step's event after
already passing through all previous steps. Supports diff mode to
compare two segments side by side. `steps` is also editable from the
widget's sidebar without re-running the cell.

## How it works

A funnel keeps only the events you name and asks of each path: did it reach
these, in this order? Everything in between is ignored — which is what makes a
funnel easy to agree on with stakeholders, and unable to tell you why the drop
happened.

Two conventions worth being explicit about, because funnels in different tools
count differently:

- **Order matters, gaps do not.** A path counts at step N only if it already
  passed every earlier step, in sequence. Other events may happen in between,
  any number of times.
- **Paths are counted once.** The numbers are unique paths, not event
  occurrences — buying twice does not count twice.

The result carries two conversion rates per step, and mixing them up is the most
common misreading:

| Field | Denominator | Reads as |
|---|---|---|
| `conversion_rate` | all paths in the eventstream | "share of everyone who got this far" |
| `step_conversion_rate` | the previous step | "share of those who got here from the step before" |

Note the denominator of `conversion_rate`: *all* paths, including those that
never entered the funnel at all. If most of your eventstream never touches the
funnel, filter first ([`filter_paths`](https://retentioneering.com/docs/data-processors/filter-paths)) so
the rate answers the question you meant.

### When the funnel stops being enough

The moment you want to know *why* users dropped between two levels, put the
discarded detail back on a narrower question:

```python
# what actually happens between two levels
stream.truncate_paths(start_event="add_to_cart", end_event="purchase").transition_graph()

# turn "how far did this path get" into a segment, then compare the groups
labelled = stream.add_segment("funnel", funnel_events=["add_to_cart", "shipping_details", "purchase"])
labelled.step_matrix(diff=("funnel", "shipping_details", "purchase"))
```

See [Path Analysis](https://retentioneering.com/docs/path-analysis) for how a funnel compares with the
step and transition representations of the same paths.

## Usage

```python
stream.funnel(steps=["catalog", "add_to_cart", "purchase"])
stream.funnel(steps=["add_to_cart", "purchase"], diff=("user_lifecycle", "loyal", "new"))
```

## Examples

### Basic
```python
stream.funnel(
    steps=["catalog", "product_view", "add_to_cart", "purchase"],
)
```

### Diff mode
```python
stream.funnel(
    steps=["catalog", "product_view", "add_to_cart", "purchase"],
    diff=["platform", "mobile", "desktop"]
)
```

## Parameters

### Data

Data parameters change the computed result. They are exactly the arguments of
the widget's headless twin `stream.funnel_data()` — see
[headless mode](#headless-mode) below.

| Parameter | Type | Description |
|---|---|---|
| `steps` | `list of str, optional` | Ordered event names defining the funnel steps. |
| `diff` | `tuple or list, optional` | Draws a comparative chart for a pair of segments; see [Diff mode](https://retentioneering.com/docs/widgets#diff-mode). `(segment_col, value1, value2)` or `(path_ids1, path_ids2)`; `value2` may be `<REST>`. |
| `path_col` | `str, optional` | Path ID column override; defaults to `schema.path_col`. |

### Display

Display parameters only affect how the widget is rendered.

| Parameter | Type | Description |
|---|---|---|
| `height` | `int, default 420` | Widget height in pixels. |
| `sidebar_open` | `bool, default True` | Whether the sidebar starts open. |
| `state_file` | `str, optional` | JSON file the widget state is bound to; see [Saving widget state](https://retentioneering.com/docs/widgets#saving-widget-state). |

## Headless mode

`stream.funnel_data()`

Compute funnel conversion metrics and return a dict (headless).

---

# Segment Overview

Source: https://retentioneering.com/docs/widgets/segment-overview

Interactive segment comparison heatmap for Jupyter notebooks.

Rows are metrics, columns are segment values. Click a cell to see that
metric's distribution for the segment; shift-click a second cell in the
same row to compare two distributions side by side. `segment_col` and
`metrics` are also editable from the widget's sidebar without
re-running the cell.

## How it works

[Diff mode](https://retentioneering.com/docs/widgets#diff-mode) compares two groups in depth. Segment
Overview does the opposite: it compares *every* level of a segment at once, on
several metrics, shallowly — so you know which pair is worth diffing before you
spend a widget on it. Six acquisition channels across five metrics is one
heatmap here and fifteen diffed graphs otherwise.

The computation is two steps. Each path gets a value for each of your
[path metrics](https://retentioneering.com/docs/path-metrics) — `length`, `duration`, conversion via
`has_event`, time to purchase via `time_between`. Then, for every level of
`segment_col`, those per-path values are collapsed into one number by the
metric's `agg`. Metrics become rows, segment levels become columns, and the
colour is the metric's value read across the row.

Which is why **`agg` is required here and nowhere else**. A distribution has no
single summary: `mean` follows the tail, `median` follows the bulk, and for
skewed quantities like session duration the two can point in opposite
directions. Adding the same metric twice under different aggregations is a
normal thing to do — `q95` next to `median` says whether a segment is slower
overall or just has a heavier tail.

Two rows are always present without being asked for: `segment_size` and
`segment_share`. Read them first. A level holding 0.4% of the paths will
happily post the most extreme value in every row, and it means nothing.

**`complement_distance`** is the odd one out among the aggregations. Instead of
summarizing a level's values, it measures the
[Wasserstein distance](https://en.wikipedia.org/wiki/Wasserstein_metric) between
that level's whole distribution and the pooled distribution of every other
level — "how unlike the rest is this group?" as a single number. It catches
differences the mean hides, such as one level splitting into fast and slow
sub-populations whose average lands exactly on everyone else's.

Clicking a cell opens the underlying distribution rather than the number;
shift-clicking a second cell in the same row overlays two of them, which is
usually enough to tell a real shift from a couple of outliers.

**Dynamic segments split paths, not just group them.** Metrics are computed per
(path, segment level) pair, so a user active both inside and outside an incident
window contributes a fragment to each column — you are comparing *behavior
during* the window against *behavior outside* it, not one set of users against
another. For a static segment (channel, A/B arm, cluster label) each path
belongs to exactly one level and this collapses back to the obvious reading. See
[Static and dynamic segments](https://retentioneering.com/docs/segments#static-and-dynamic-segments).
## Usage

```python
stream.segment_overview(
    segment_col="plan",
    metrics=[
        {"metric": "length", "agg": "mean"},
        {"metric": "event_count", "metric_args": {"event": "purchase"}, "agg": "mean"},
    ],
)
```

## Examples

### Basic
```python
stream.segment_overview(
    segment_col="platform",
    metrics=[
        {"metric": "length", "agg": "mean"},
        {"metric": "event_count", "metric_args": {"event": "purchase"}, "agg": "mean"},
    ],
)
```

## Parameters

### Data

Data parameters change the computed result. They are exactly the arguments of
the widget's headless twin `stream.segment_overview_data()` — see
[headless mode](#headless-mode) below.

| Parameter | Type | Description |
|---|---|---|
| `segment_col` | `str, optional` | Segment column to split by; must be one of `schema.segment_cols`. Required (directly or via the sidebar) before the widget computes anything. |
| `metrics` | `list of dict, optional` | Metric configurations, each with a `"metric"` key, optional `"metric_args"`, and an `"agg"` key (`"mean"`, `"median"`, `"q5"`, `"q25"`, `"q75"`, `"q95"`, or `"complement_distance"`) controlling how per-path values roll up across a segment. See the [Path Metrics](https://retentioneering.com/docs/path-metrics) documentation page for the metric reference. |
| `path_col` | `str, optional` | Path ID column override; defaults to `schema.path_col`. |

### Display

Display parameters only affect how the widget is rendered.

| Parameter | Type | Description |
|---|---|---|
| `height` | `int, default 480` | Widget height in pixels. |
| `sidebar_open` | `bool, default True` | Whether the sidebar starts open. |
| `state_file` | `str, optional` | JSON file the widget state is bound to; see [Saving widget state](https://retentioneering.com/docs/widgets#saving-widget-state). |

## Headless mode

`stream.segment_overview_data()`

Compute aggregated metrics across segment values (headless).

---

# Cluster Analysis

Source: https://retentioneering.com/docs/widgets/cluster-analysis

An interactive tool for finding an optimal splitting of paths by behavioral metrics.
Allows you to inspect clusters in a [Segment Overview](https://retentioneering.com/docs/widgets/segment-overview)-style heatmap
and offers the best possible splitting from the silhouette score perspective.
Once the splitting looks right, you can label the clusters and save them as a new segment column
of the eventstream right from the UI by clicking "Save Clusters".

## How it works

Every representation in [Path Analysis](https://retentioneering.com/docs/path-analysis) summarizes paths by
their *shape* — which events, in which order. Clustering goes the other way: it
describes each path by a handful of numbers, then groups paths whose numbers
look alike. Two users who never visited the same screen can still land in the
same cluster if they were equally brief, equally repetitive, or equally unlikely
to buy.

The pipeline is three steps, and each maps to one argument:

1. **Describe.** `features` turns every path into a row of numbers, using the
   shared [path metrics](https://retentioneering.com/docs/path-metrics) registry — `length`, `duration`,
   `active_days`, a per-event count, whether a pattern matched. This choice
   *is* the analysis: clusters can only differ along dimensions you measured.
2. **Scale.** `scaler` puts those numbers on a comparable footing. Without it a
   `duration` in seconds (six digits) drowns out a `length` in events (two
   digits), and the clustering silently becomes "group by duration". Hence the
   `"minmax"` default.
3. **Split.** `n_clusters` fixes the number of groups, or names a range to
   search. Given a range, the widget clusters at every size in it and keeps the
   one with the best **silhouette score** — a measure of how much tighter paths
   sit within their own cluster than to the nearest other one. It is a hint
   about structure, not a verdict: a mediocre score across the whole range
   usually means the paths form a continuum rather than distinct groups.

An optional fourth knob, [NMF](#nmf), sits between steps 2 and 3.

What comes back is not the clusters themselves but a way to read them: the
overview heatmap puts `overview_metrics` in rows and clusters in columns,
exactly like [Segment Overview](https://retentioneering.com/docs/widgets/segment-overview), so you can see
what actually distinguishes cluster 2 from cluster 3 and give it a name. Note
that `features` and `overview_metrics` are independent on purpose — profiling
clusters on metrics you did *not* cluster by is how you find out whether the
split means anything beyond its own inputs.

Clusters are not a special kind of object. Once a split looks right, "Save
Clusters" (or [`add_clusters`](https://retentioneering.com/docs/data-processors/add-clusters)) writes the
labels into the eventstream as an ordinary [segment](https://retentioneering.com/docs/segments) column, and
from there every segment-aware tool — diff mode, Segment Overview, the
`in_segment` metric — works on it unchanged.

**Choosing a cluster count is a judgment call, not a computation.** Prefer the
smallest number of clusters you can still describe in words; a split you cannot
name is a split you cannot act on.
## Usage

```python
stream.cluster_analysis(
    features=[{"metric": "length"}, {"metric": "duration"}, {"metric": "event_count_bulk"}],
    n_clusters="3-6",
)
```

## Examples

### Basic
```python
stream.cluster_analysis(
    features=[
        {"metric": "event_count_bulk", "metric_args": {"events": ["catalog", "product_view", "add_to_cart", "purchase"]}},
    ],
    n_clusters=3,
)
```

## Parameters

### Data

Data parameters change the computed result. They are the arguments of the
widget's headless twin `stream.cluster_analysis_data()` — see
[headless mode](#headless-mode) below — with three exceptions, all of which the
twin accepts and this constructor does not:

- `nmf_components` — see [NMF](#nmf) below. It is *not* missing from the widget:
  the sidebar has an NMF toggle and a component-count field, and the result
  arrives as the H-matrix / W Cluster Means tabs. It is only unavailable as a
  constructor argument, so set it in the sidebar or use the headless twin.
- `min_cluster_size`, `cluster_selection_epsilon` — tuning knobs for
  `method="hdbscan"`. The method itself can be passed to the widget, but there
  is no sidebar control to switch it and no way to tune these two from the
  widget at all. Use the headless twin for hdbscan work.

| Parameter | Type | Description |
|---|---|---|
| `features` | `list of dict, optional` | Metric configurations used as clustering features (see the [Path Metrics](https://retentioneering.com/docs/path-metrics)). If omitted, the sidebar starts pre-filled with a wildcard `event_count_bulk` metric (one column per event in the eventstream) — that pre-fill is a starting point to edit, not something that runs on its own: passing `features` explicitly (or clicking "Apply" in the sidebar) is what actually triggers clustering. |
| `method` | `{"kmeans", "hdbscan"}, default "kmeans"` | Clustering algorithm. |
| `scaler` | `{"minmax", "std"}, optional` | Feature scaler applied before clustering; default `"minmax"`. |
| `n_clusters` | `int, list of int, or str, optional` | Number of clusters. A single int fixes the cluster count; a list of ints or a range string (e.g. `"3-8"`) runs a silhouette-scored grid search over that range and picks the best. Defaults to `"3-8"`. |
| `overview_metrics` | `list of dict, optional` | Metrics shown in the overview heatmap after clustering (independent of `features`). If omitted, the sidebar starts pre-filled with a wildcard `event_count_bulk` metric here too (mean count per event); same as `features`, it only takes effect once you click "Apply" or pass the argument explicitly. Both `features` and `overview_metrics` accept metric configs from the same [Path Metrics](https://retentioneering.com/docs/path-metrics) registry. |
| `path_col` | `str, optional` | Path ID column override; defaults to `schema.path_col`. |

### Display

Display parameters only affect how the widget is rendered.

| Parameter | Type | Description |
|---|---|---|
| `height` | `int, default 520` | Widget height in pixels. |
| `sidebar_open` | `bool, default True` | Whether the sidebar starts open. |
| `state_file` | `str, optional` | JSON file the widget state is bound to; see [Saving widget state](https://retentioneering.com/docs/widgets#saving-widget-state). |

### NMF

Turning on **NMF** in the sidebar (or passing `nmf_components` to the headless
twin) inserts a dimensionality-reduction step between scaling and clustering:
the feature matrix is factorized into the requested number of non-negative
components, and paths are clustered on those components instead of on the raw
metrics. It is independent of the clustering algorithm — the usual pairing is
NMF with k-means.

Reach for it when `features` is wide and correlated — a wildcard
`event_count_bulk` over a few dozen events is the standard case. Twenty
near-duplicate count columns let the loudest events dominate the distance
calculation; a handful of components spreads the signal out. Like `n_clusters`,
`nmf_components` accepts a single number or a range to grid-search.

The payoff is interpretability, and it arrives as two extra tabs:

- **H-matrix** — how each component is built out of your features. This is what
  names a component: one loading heavily on `search` and `filter_results` is
  "browsing intensity", whatever the algorithm called it.
- **W Cluster Means** — how strongly each cluster expresses each component. Read
  it together with the H-matrix and each cluster gets a description in terms of
  behavior rather than a number.

The cost is that clusters no longer sit in the space of your original metrics,
so `overview_metrics` (which always report raw per-path values) stays the honest
check on whether the split means anything.

### From the widget to a segment column

The widget explores; [`add_clusters`](https://retentioneering.com/docs/data-processors/add-clusters)
persists. The two differ in one place: `cluster_analysis` searches a range of
cluster counts (`"3-8"` by default) and picks a winner, while `add_clusters`
materializes one specific clustering and therefore needs an exact
`n_clusters`.

You don't have to transcribe it. "Save Clusters" in the sidebar writes the
matching `add_clusters(...)` call — optionally with your renamed cluster labels —
and headlessly the same value comes back as `best_params`:

```python
features = [{"metric": "length"}, {"metric": "active_days"}]

result = stream.cluster_analysis_data(features=features)
result["best_params"]        # e.g. {"n_clusters": 3}

stream = stream.add_clusters("behavior", features=features, **result["best_params"])
```

`best_params` holds only the parameters that were searched over, so pass the
same `features` alongside it.

## Headless mode

`stream.cluster_analysis_data()`

Run cluster analysis headlessly and return a dict of results.

Pass lists for n_clusters / nmf_components / min_cluster_size to trigger
grid search with silhouette scoring. For the kmeans method (the default),
n_clusters defaults to `"3-8"` if omitted — including for
nmf_components-only searches.

`best_params` holds the concrete parameter values actually used to produce
`overview_df` (the winning combination when searching, or just the fixed
values passed in otherwise) — pass it straight to `add_clusters` to
materialize the same clustering as a segment column.

---

# Filter Events

Source: https://retentioneering.com/docs/data-processors/filter-events

Keep only rows that match a column filter, a Python predicate, or a SQL query.

Exactly one of `keep`, `drop`, `func`, or `sql` must be provided. If all are
`None` the eventstream is returned unchanged.

## Usage

```python
stream.filter_events(keep={"event": ["purchase", "add_to_cart"]})
stream.filter_events(drop={"event": ["system_event"], "platform": ["bot"]})
stream.filter_events(sql="SELECT * FROM eventstream WHERE event NOT LIKE 'system_%'")
```

## How it works

`filter_events` works on **rows**, not on paths: a path whose every event is
filtered out disappears, but a path can also be left with only some of its
events. To keep or drop whole paths instead, use
[Filter Paths](https://retentioneering.com/docs/data-processors/filter-paths).

The `keep` and `drop` forms are not mirror images once you list more than one
column, and that catches people out. Starting from this toy eventstream:

| user_id | event | platform |
|---|---|---|
| u1 | home | ios |
| u1 | cart | ios |
| u1 | purchase | ios |
| u2 | home | web |
| u2 | bot_ping | web |
| u2 | cart | web |

```python
stream.filter_events(keep={"event": ["home", "cart"], "platform": ["ios"]})
```

`keep` combines columns with **AND** — a row survives only if it matches every
entry. Here that means "iOS rows that are also `home` or `cart`":

| user_id | event | platform |
|---|---|---|
| u1 | home | ios |
| u1 | cart | ios |

```python
stream.filter_events(drop={"event": ["bot_ping"], "platform": ["web"]})
```

`drop` combines columns with **OR** — a row is removed if it matches any entry.
Here that drops `bot_ping` *and* everything on web, which takes u2 out entirely:

| user_id | event | platform |
|---|---|---|
| u1 | home | ios |
| u1 | cart | ios |
| u1 | purchase | ios |

For anything the two dict forms can't express, `func` takes a pandas predicate
over the raw DataFrame and `sql` takes a DuckDB `SELECT` reading from the
`eventstream` table alias.

## Parameters

| Parameter | Type | Description |
|---|---|---|
| `keep` | `dict, optional` | `{column: values}` mapping. Keeps rows where each listed column contains one of the listed values. Multiple columns combine with AND: a row is kept only if it matches every entry. Example: `{"event": ["purchase", "add_to_cart"]}`. |
| `drop` | `dict, optional` | Same `{column: values}` format, but removes the matching rows instead. Multiple columns combine with OR: a row is removed if it matches any entry (the exact complement of `keep`). |
| `func` | `callable, optional` | A function that accepts the raw pandas DataFrame and returns a boolean Series. Rows where the Series is `True` are kept. |
| `sql` | `str, optional` | DuckDB SQL SELECT statement that reads from the `eventstream` table alias and returns all original columns. Example: `"SELECT * FROM eventstream WHERE event NOT LIKE 'system_%'"`. |

---

# Filter Paths

Source: https://retentioneering.com/docs/data-processors/filter-paths

Keep only paths that satisfy a metric condition.

The condition is a tree of comparison nodes connected by `and` / `or` / `not`
branch nodes. Per-path metrics are computed once and the condition is evaluated
in SQL.

Raises `EmptyEventstreamError` when no paths match.

## Usage

```python
# Keep paths that contain at least one purchase
stream.filter_paths({"op": ">", "metric": "event_count", "value": 0, "metric_args": {"event": "purchase"}})

# Keep paths that contain a promo_view or a discount_applied event
stream.filter_paths({"op": "=", "metric": "has_any_event", "value": True,
                      "metric_args": {"events": ["promo_view", "discount_applied"]}})

# Keep paths longer than 3 events that match a funnel pattern
# (a top-level list means AND)
stream.filter_paths([
    {"op": ">", "metric": "length", "value": 3},
    {"op": "=", "metric": "matches_pattern", "value": True,
     "metric_args": {"pattern": "registration->.*->purchase"}},
])
```

## How it works

Every path is scored by the [path metrics](https://retentioneering.com/docs/path-metrics) named in the
condition, and then kept or dropped **as a whole** — unlike
[Filter Events](https://retentioneering.com/docs/data-processors/filter-events), this never leaves a path
with some of its events missing.

Before — three paths of different lengths, one of which converts:

- u1: `home → cart → purchase`
- u2: `home → cart`
- u3: `home`

```python
stream.filter_paths({"op": ">", "metric": "length", "value": 2})
```

After — only u1 has more than two events, so u2 and u3 are gone, all of their
events with them:

- u1: `home → cart → purchase`

```python
stream.filter_paths({"op": "=", "metric": "has_event", "value": True,
                     "metric_args": {"event": "purchase"}})
```

After — the same single path survives, this time because it is the only one
containing a `purchase` anywhere:

- u1: `home → cart → purchase`

Conditions combine into a tree, and a plain list is shorthand for AND:

```python
# longer than 2 events AND reached purchase
stream.filter_paths([
    {"op": ">", "metric": "length", "value": 2},
    {"op": "=", "metric": "has_event", "value": True,
     "metric_args": {"event": "purchase"}},
])
```

If nothing matches, the call raises `EmptyEventstreamError` rather than
returning an empty eventstream.

## Parameters

| Parameter | Type | Description |
|---|---|---|
| `condition` | `dict or list` | Condition tree of leaf (comparison) and branch (`and`/`or`/`not`) nodes; a plain list is shorthand for AND. Metrics used in a leaf must produce exactly one value per path, which rules out `has_event_bulk` and `event_count_bulk`. |
| `path_col` | `str, optional` | Path ID column override; defaults to `schema.path_col`. |
| `event_col` | `str, optional` | Event column override; defaults to `schema.event_col`. |

### Condition node keys

- `op` — for a leaf, a comparison operator: `>`, `>=`, `<`, `<=`, `=` (or `==`), `!=`. For a branch, one of `and`, `or`, `not`.
- `args` (branch nodes only) — list of child nodes. `[cond1, cond2]` ≡ `{"op": "and", "args": [cond1, cond2]}`.
- `metric` — metric name (see the [Path Metrics](https://retentioneering.com/docs/path-metrics) documentation page for the full list).
- `value` — threshold value.
- `metric_args` (optional) — dict of extra arguments for the metric. `has_event`/`event_count` take a single `event` string; for a multi-event AND/OR condition use `has_all_events`/`has_any_event` with an `events` list.

Supported comparison operators: `=` (or `==`), `!=`, `>`, `<`, `>=`, `<=`. See
[Path Metrics](https://retentioneering.com/docs/path-metrics#filter-paths-condition-format) for the full
condition grammar and the list of metrics you can compare.

---

# Add Events

Source: https://retentioneering.com/docs/data-processors/add-events

Insert synthetic events derived from existing events or a SQL query.

Exactly one of `source_events`, `sql`, or `churn` must be provided.
The new event rows are appended to the eventstream; original rows are kept.

## Usage

```python
stream.add_events("session_start", source_events=["login", "app_open"])
stream.add_events("churned", churn={"inactivity_days": 30})
stream.add_events("churned", churn={"inactivity_days": 30, "active_events": ["purchase"]})
```

## How it works

New events are marked `synthetic` and inserted at the same timestamp as the row
that produced them, ordered immediately after it. Nothing is removed.

### source_events

One synthetic event per **occurrence** of any listed source event — not one per
path. This is the usual way to give several equivalent events a single name you
can anchor on later.

Before:

- u1: `login → browse → login → app_open`
- u2: `browse → app_open`

```python
stream.add_events("session_start", source_events=["login", "app_open"])
```

After — u1 gets three `session_start` events, because it has three matching
source events:

- u1: `login → session_start → browse → login → session_start → app_open → session_start`
- u2: `browse → app_open → session_start`

### churn

Marks the point where a path goes quiet: a synthetic event is inserted after
the last activity that is followed by a gap of at least `inactivity_days`.
Reaching the end of the observation window counts as such a gap, so paths that
simply stop also get the event.

Before — u1 stops after 2 January, while u2 comes back after two months:

| user_id | event | timestamp |
|---|---|---|
| u1 | purchase | 2024-01-01 |
| u1 | browse | 2024-01-02 |
| u2 | browse | 2024-01-01 |
| u2 | browse | 2024-03-01 |

```python
stream.add_events("churned", churn={"inactivity_days": 30})
```

After — u1 churns at the end of its own activity, u2 churns *mid-path*, at the
event that preceded the long gap:

| user_id | event | timestamp |
|---|---|---|
| u1 | purchase | 2024-01-01 |
| u1 | browse | 2024-01-02 |
| u1 | **churned** | 2024-01-02 |
| u2 | browse | 2024-01-01 |
| u2 | **churned** | 2024-01-01 |
| u2 | browse | 2024-03-01 |

Pass `active_events` to count only meaningful actions as activity — with
`churn={"inactivity_days": 30, "active_events": ["purchase"]}` a user who keeps
browsing but stops buying still churns.

### sql

For anything the two modes above don't cover, `sql` takes a DuckDB `SELECT`
over the `eventstream` table alias returning rows in the eventstream schema;
each returned row becomes one synthetic event.

## Parameters

| Parameter | Type | Description |
|---|---|---|
| `name` | `str` | Name of the synthetic event to create. |
| `source_events` | `list of str, optional` | List of existing event names. Every occurrence of any of them gets a synthetic event at the same timestamp, ordered right after the source row — a path with three matching events gets three synthetic events. |
| `sql` | `str, optional` | DuckDB SQL SELECT statement that reads from the `eventstream` table alias and returns rows in the eventstream schema. Each returned row is added as a new synthetic event. |
| `churn` | `dict, optional` | Creates a churn event after a period of inactivity. |

### Churn keys

- `inactivity_days` (int or float, required) — gap in days after which a churn event is inserted.
- `active_events` (list of str, optional) — only these events count as activity; defaults to all events.

---

# Add Segment

Source: https://retentioneering.com/docs/data-processors/add-segment

Add a new categorical segment column to the eventstream.

Exactly one of `rules`, `func`, `sql`, `funnel_events`, or `time_range`
must be provided — unless `name` is already listed in
`schema.custom_cols`, in which case passing none of them promotes that
existing column to a segment in place, without recomputing its values.

## Usage

```python
# ordered CASE-WHEN rules over an existing column
stream.add_segment("region", rules=[
    ["country", "=", "US", "domestic"],
    ["country", "in", "('GB', 'DE', 'FR')", "europe"],
    ["other"],
])

# the deepest funnel step each path completed in order
stream.add_segment("funnel", funnel_events=["add_to_cart", "shipping_details", "purchase"])

# "inside" / "outside" a time window
stream.add_segment("incident", time_range=("2024-03-10", "2024-03-17"))

# a DuckDB SELECT returning one label per row
stream.add_segment("device", sql="SELECT CASE WHEN platform = 'mobile' THEN 'mobile' ELSE 'web' END FROM eventstream")

# promote a column that is already in the eventstream, keeping its values
stream.add_segment("returned")
```

## How it works

A segment column stores one label **per event row**, which is what lets a
segment be either static (the same label along a whole path) or dynamic
(changing as the path goes on). See [Segments](https://retentioneering.com/docs/segments) for what that
distinction buys you.

### rules — CASE-WHEN over existing columns

Conditions are tried in order; the last entry is the fallback label.

Before — a `country` column that came along with the source data:

| user_id | event | country |
|---|---|---|
| u1 | home | US |
| u1 | cart | US |
| u2 | home | DE |
| u2 | cart | DE |

```python
stream.add_segment("region", rules=[
    ["country", "=", "US", "domestic"],
    ["international"],
])
```

After:

| user_id | event | country | region |
|---|---|---|---|
| u1 | home | US | domestic |
| u1 | cart | US | domestic |
| u2 | home | DE | international |
| u2 | cart | DE | international |

Write string values **unquoted** — `"US"`, not `"'US'"` — since they get quoted
for you. The one exception is `op="in"`, whose value is passed through to SQL
as-is and so must be a complete tuple:
`["country", "in", "('GB', 'DE', 'FR')", "europe"]`.

### funnel_events — how deep did this path get

Labels each path with the deepest funnel step it completed **in order**. This is
the mode that turns a funnel drop-off into a comparable group.

Before — four paths against an `add_to_cart → checkout → purchase` funnel:

- u1: `home → add_to_cart → checkout → purchase`
- u2: `home → add_to_cart → checkout`
- u3: `home → checkout → purchase` (skipped `add_to_cart`)
- u4: `home`

```python
stream.add_segment("funnel", funnel_events=["add_to_cart", "checkout", "purchase"])
```

After — one label per path, written onto every one of its events:

| path | funnel |
|---|---|
| u1 | `purchase` |
| u2 | `checkout` |
| u3 | `out_of_funnel` |
| u4 | `out_of_funnel` |

Note u3: it reached both `checkout` and `purchase`, but never completed
`add_to_cart` first, so the strictly ordered funnel gives it no credit at all.
Paths that never complete even the first step are labelled `out_of_funnel`.

### time_range — inside vs outside a window

```python
stream.add_segment("incident", time_range=("2024-03-10", "2024-03-17"))
```

Every event is labelled `inside` or `outside` by its own timestamp, so a user
active both during and after an incident contributes to both groups. That is
what makes this the right tool for "how did behaviour change during X?" — see
[Segments](https://retentioneering.com/docs/segments#inside-vs-outside-an-anomalous-period).

### func and sql — anything else

`func` receives the raw DataFrame and returns one label per row, in row order;
`sql` is a DuckDB `SELECT` over the `eventstream` alias returning a single
column, also one value per row in order.

### Promoting an existing column

If the column is already in the eventstream — it rode along from the source data
as a custom column — call `add_segment` with just the name and no mode argument.
Its values are kept as they are and the column becomes a segment.

```python
stream.add_segment("returned")
```

## Parameters

| Parameter | Type | Description |
|---|---|---|
| `name` | `str` | Name of the new segment column. |
| `rules` | `list, optional` | CASE-WHEN rules. A list of conditions plus a final else entry: Example: `[["country", "=", "US", "domestic"], ["international"]]`. |
| `func` | `callable, optional` | A function that accepts the raw pandas DataFrame and returns a collection of segment labels with the same length and order as the eventstream rows. |
| `sql` | `str, optional` | DuckDB SQL SELECT statement that reads from the `eventstream` table alias and returns exactly one column — the segment label for each row. Row count and order must match the eventstream. Example: `"SELECT CASE WHEN platform = 'mobile' THEN 'mobile' ELSE 'web' END FROM eventstream"`. |
| `funnel_events` | `list of str, optional` | Ordered list of at least 2 event names defining a strict, ordered ("closed") funnel. A path is assigned `funnel_events[k]` only if it contains every event `funnel_events[0]` through `funnel_events[k]` *and* their last occurrences appear in that same order — reaching a later step without having completed the earlier ones in order does not count towards it. A path is assigned the highest such `k`; if it never completes even `funnel_events[0]`, it is labeled `out_of_funnel`. Segment values (in ascending funnel order): `out_of_funnel`, then each event name from `funnel_events[0]` to `funnel_events[-1]`. |
| `time_range` | `tuple or list, optional` | `(start, end)` — two timestamps (string or `pd.Timestamp`) bounding an inclusive interval over `schema.timestamp_col`. Each event is labeled `inside` if its timestamp falls within `[start, end]`, otherwise `outside`. |
| `path_col` | `str, optional` | Path ID column override for `funnel_events` mode; defaults to `schema.path_col`. |

### Values rules

- Each condition entry is `[column, op, value, label]` — translates to `WHEN <column> <op> <value> THEN <label>` in SQL. A string `value` is quoted for you, so write `"US"`, not `"'US'"` — the exception is `op="in"`, whose value is inserted raw and must therefore be a complete SQL tuple: `"('GB', 'DE', 'FR')"`.
- The last entry is `[else_label]` — the ELSE branch label.

---

# Add Clusters

Source: https://retentioneering.com/docs/data-processors/add-clusters

Cluster paths using ML and add a new segment column with `cluster_0`, `cluster_1`,
etc. cluster labels.

Per-path metrics are computed from `features`, optionally scaled, then passed to
the chosen clustering algorithm. The resulting cluster label is broadcast to every
row of the corresponding path.

## Usage

```python
stream.add_clusters(
    name="cluster",
    features=[
        {"metric": "length"},
        {"metric": "event_count", "metric_args": {"event": "purchase"}},
    ],
    method="kmeans",
    n_clusters=4,
    scaler="minmax",
)
```

## How it works

`add_clusters` is the non-interactive half of
[Cluster Analysis](https://retentioneering.com/docs/widgets/cluster-analysis): same features, same scaler,
same algorithms, no UI. It computes the `features` metrics per path, scales
them, clusters the result, and writes the label back onto every row of the
corresponding path as a new [segment](https://retentioneering.com/docs/segments) column — after which the
clusters behave like any other segment, in diff mode, Segment Overview, or the
`in_segment` metric.

Two things are worth knowing before you call it.

**`n_clusters` is required for k-means, and that is not an oversight.** The
widget searches a range and picks a winner by silhouette score, because it can
show you the result and let you disagree. A processor writing a column into your
data has no such conversation, so it asks for the exact number rather than
guessing one. The intended flow is to settle the question in the widget first —
"Save Clusters" emits the matching call, and headlessly
`cluster_analysis_data()["best_params"]` carries the same value:

```python
features = [{"metric": "length"}, {"metric": "active_days"}]

result = stream.cluster_analysis_data(features=features)
stream = stream.add_clusters("behavior", features=features, **result["best_params"])
```

**Clustering is not deterministic across feature sets.** Labels are positional
(`cluster_0`, `cluster_1`, …) and carry no meaning of their own — adding a
feature or changing the scaler can renumber every group. Rename them to
something you can read once the split is settled, with
[`rename_segment_levels`](https://retentioneering.com/docs/data-processors/rename-segment-levels):

```python
stream = stream.rename_segment_levels("behavior", {"cluster_0": "browsers", "cluster_1": "buyers"})
```

See [Path Metrics](https://retentioneering.com/docs/path-metrics) for what you can cluster on, and
[Cluster Analysis](https://retentioneering.com/docs/widgets/cluster-analysis#how-it-works) for why the
choice of `features` *is* the analysis.
## Parameters

| Parameter | Type | Description |
|---|---|---|
| `name` | `str` | Name of the new segment column to add. |
| `features` | `list of dict` | Metric configurations used as clustering features. Each dict has a `"metric"` key (str) and an optional `"metric_args"` key (dict). Available metrics: `"length"`, `"duration"`, `"event_count"`, `"has_event"`, `"event_count_bulk"`, `"has_event_bulk"`, `"has_all_events"`, `"has_any_event"`, `"time_between"`, `"first_event_time"`, `"active_days"`, `"matches_pattern"`, `"in_segment"`. See the [Path Metrics](https://retentioneering.com/docs/path-metrics) documentation page for the full metric reference. |
| `method` | `str, default "kmeans"` | Clustering algorithm. One of `"kmeans"` or `"hdbscan"`. |
| `scaler` | `str or None, default "minmax"` | Feature scaler applied before clustering. One of `"minmax"`, `"std"`, or `None`. (`"standard"` is accepted as a legacy alias of `"std"`.) |
| `n_clusters` | `int, optional` | Number of clusters (required for `"kmeans"`). |
| `min_cluster_size` | `int, optional` | Minimum cluster size (used by `"hdbscan"`). |
| `cluster_selection_epsilon` | `float, optional` | HDBSCAN cluster-selection epsilon. |
| `nmf_components` | `int, optional` | When set, reduces features to this many NMF components before clustering. |
| `path_col` | `str, optional` | Path ID column override; defaults to `schema.path_col`. |
| `event_col` | `str, optional` | Event column override; defaults to `schema.event_col`. |

---

# Add Start End Events

Source: https://retentioneering.com/docs/data-processors/add-start-end-events

Prepend a `path_start` and append a `path_end` synthetic event to each path.

Idempotent: a path that already starts or ends with these events is left
unchanged on that side.

You normally don't need to call this directly — `transition_graph`,
`step_matrix`, and `step_sankey` insert `path_start`/`path_end` themselves,
each using its own `path_col`. Calling this upfront bakes in one
specific path definition and can produce misleading boundaries if a
widget is later given a different `path_col`.

## Usage

```python
stream.add_start_end_events()
```

## How it works

`path_start` and `path_end` are the boundary markers that make "where do paths
begin?" and "where do users leave?" answerable as ordinary event questions — a
[Step Matrix](https://retentioneering.com/docs/widgets/step-matrix) anchored at `path_end`, or a
`cart → path_end` edge on the [Transition Graph](https://retentioneering.com/docs/widgets/transition-graph)
that reads as "abandoned here".

Before:

- u1: `home → cart`
- u2: `home`

```python
stream.add_start_end_events()
```

After — each marker copies the timestamp of the event it brackets, so a
single-event path gets both at the same moment:

| user_id | event | timestamp |
|---|---|---|
| u1 | `path_start` | 10:00 |
| u1 | home | 10:00 |
| u1 | cart | 10:01 |
| u1 | `path_end` | 10:01 |
| u2 | `path_start` | 10:00 |
| u2 | home | 10:00 |
| u2 | `path_end` | 10:00 |

You rarely need to call this yourself: `transition_graph`, `step_matrix` and
`step_sankey` add the markers on the fly, each using its own `path_col`. Calling
it upfront freezes one definition of a path into the data, which then quietly
mislabels the boundaries if a widget is later given a different `path_col` — for
example after [`split_sessions`](https://retentioneering.com/docs/data-processors/split-sessions), where
per-user markers would sit in the middle of most sessions.

## Parameters

| Parameter | Type | Description |
|---|---|---|
| `path_col` | `str, optional` | Path ID column override; defaults to `schema.path_col`. |

---

# Collapse Events

Source: https://retentioneering.com/docs/data-processors/collapse-events

Merge consecutive or grouped events into a single representative event.

Exactly one of `consecutive`, `event_groups`, `group_col`, or
`session_col` must be provided.

## Usage

```python
# Collapse any run of the same event
stream.collapse_events(consecutive=True)

# Collapse only repeated page_view events
stream.collapse_events(consecutive=["product_view"])

# Merge checkout steps into a single "checkout" event
stream.collapse_events(event_groups=[{"events": ["checkout_start", "checkout_step", "checkout_confirm"], "name": "checkout"}])
```

## How it works

Toy paths showing exactly what happens to the eventstream for each collapsing mode.

### consecutive

Collapses each run of the same event into one row, keeping the run's first
timestamp.

Before: `page_view → page_view → page_view → click → click → product_view`

```python
stream.collapse_events(consecutive=True)
```

After — every run collapses, regardless of which event it is:

`page_view → click → product_view`

```python
stream.collapse_events(consecutive=["page_view"])
```

After — only `page_view` runs collapse; the two `click` events are left alone
since `click` isn't in the list:

`page_view → click → click → product_view`

### event_groups

Available keys:

- `events` (str or list of str) — collapse any run of these events, wherever it occurs in the path, into one group.
- `separator` (str or list of str) — collapse every event up to and including the next separator event into one group.
- `start_event` + `end_event` (str or list of str) — collapse every event between a `start_event` and the next `end_event`, inclusive of both, into one group.
- `name` (str) — label for the merged event, required unless `cases` are given.
- `cases` (list of dict, optional) — conditional labels evaluated against the group's own events, falling back to `name` for groups no case matched.

`event_groups` merges a *set of related events* into one representative
event, named with a help of the `name` or `cases` additional arguments.
Which events belong to a group is decided by exactly one
of 3 boundary modes per group dict: `events`, `separator`, or `start_event` +
`end_event`.

#### Mode 1: events

Any run of the listed events collapses into one group, wherever it occurs —
the events don't need to be adjacent to *each other* in the parameter, just
adjacent in the path. Events not in the list pass through unchanged and end
a run.

Before: `home → payment_details → shipping_details → review_page → purchase`

```python
stream.collapse_events(event_groups=[
    {"events": ["payment_details", "shipping_details", "review_page"], "name": "checkout"},
])
```

After: `home → checkout → purchase`

#### Mode 2: separator

A session is every event up to and including the next `separator` event
(inclusive of the separator itself). Trailing events after the *last*
separator in a path aren't part of any session and pass through unchanged.

Before: `search → search_submit → browse → search → search_submit → purchase`

```python
stream.collapse_events(event_groups=[
    {"separator": "search_submit", "name": "search_session"},
])
```

After — two separate `search_session` groups (each ending at its own
`search_submit`); the trailing `purchase` has no separator after it, so it's
left uncollapsed:

`search_session → search_session → purchase`

#### Mode 3: start_event + end_event

A session is every event between a `start_event` and the next `end_event`,
inclusive of both.

Before: `home → checkout_start → payment_details → checkout_complete → confirmation_page`

```python
stream.collapse_events(event_groups=[
    {"start_event": "checkout_start", "end_event": "checkout_complete", "name": "checkout"},
])
```

After: `home → checkout → confirmation_page`

#### cases

`cases` names a group differently depending on what happened *inside it* —
each case is a [Filter Paths](https://retentioneering.com/docs/data-processors/filter-paths)-style
condition, evaluated against only the group's own events. The group-level
`name` is the fallback for groups no case matched. `cases` is orthogonal to
the boundary mode — it can be combined with any of `events`, `separator`, or
`start_event`/`end_event` above; the example below just picks
`start_event`/`end_event` for concreteness.

Before — two independent checkout sessions (same boundary mode as
`start_event`/`end_event` above), one that reached `purchase` and one that
didn't:

- user 1: `checkout_start → payment_details → purchase → checkout_end`
- user 2: `checkout_start → payment_details → checkout_end`

```python
stream.collapse_events(event_groups=[{
    "start_event": "checkout_start",
    "end_event": "checkout_end",
    "cases": [
        {
            "condition": {"op": "=", "metric": "has_event", "value": True,
                          "metric_args": {"event": "purchase"}},
            "name": "successful_checkout",
        },
    ],
    "name": "abandoned_checkout",  # fallback for groups the case didn't match
}])
```

After — the label depends on whether `purchase` occurred inside each group:

- user 1: `successful_checkout`
- user 2: `abandoned_checkout`

#### multiple groups

`event_groups` is a *list* — each group dict is applied in order, and every
group after the first sees the *output* of the ones before it, not the
original path. A later group can therefore match against an event name
created by an earlier group.

Before: `payment_details → shipping_details → review_page → purchase`

```python
stream.collapse_events(event_groups=[
    {"events": ["payment_details", "shipping_details"], "name": "checkout_details"},
    {"events": ["checkout_details", "review_page"], "name": "checkout"},
])
```

After — the first group merges `payment_details`/`shipping_details` into
`checkout_details`; the second group then merges that `checkout_details`
event together with `review_page` (which it can only match because the first
group already ran) into a single `checkout`:

`checkout → purchase`

### group_col

Collapses each run of consecutive rows sharing the same value of `group_col`
into one event, named after that value.

Before: `home → catalog → search → add_to_cart → purchase`, with
`screen_type` = `browse, browse, browse, action, action`

```python
stream.collapse_events(group_col="screen_type")
```

After — three `browse` rows collapse into one `browse` event, and the two
`action` rows collapse into one `action` event:

`browse → action`

### session_col + session_type_col

Collapses every session (identified by `session_col`) into a single event
named after that session's `session_type_col` value.

Before: `home → catalog → add_to_cart → purchase`, with `session_id` =
`s1, s1, s2, s2` and `session_kind` = `browsing_session, browsing_session,
buying_session, buying_session`

```python
stream.collapse_events(session_col="session_id", session_type_col="session_kind")
```

After — each session becomes a single event named after its type, at the
session's first timestamp:

`browsing_session → buying_session`

### agg

By default, every non-event column takes its **first** value within the
merged rows (ordered by timestamp). `agg` overrides this per column with one
of `"first"`, `"last"`, `"min"`, `"max"`, `"mean"`, `"mode"`, `"any"`.

Before — a `price` column alongside three consecutive `add_to_cart` events:

| user_id | event | timestamp | price |
|---|---|---|---|
| u1 | add_to_cart | 00:00:00 | 10 |
| u1 | add_to_cart | 00:00:05 | 25 |
| u1 | add_to_cart | 00:00:09 | 15 |
| u1 | purchase | 00:00:20 | 50 |

```python
stream.collapse_events(consecutive=True)  # no agg: price takes the first value
```

| user_id | event | timestamp | price |
|---|---|---|---|
| u1 | add_to_cart | 00:00:00 | 10 |
| u1 | purchase | 00:00:20 | 50 |

```python
stream.collapse_events(consecutive=True, agg={"price": "max"})
```

| user_id | event | timestamp | price |
|---|---|---|---|
| u1 | add_to_cart | 00:00:00 | 25 |
| u1 | purchase | 00:00:20 | 50 |

```python
stream.collapse_events(consecutive=True, agg={"price": "mean"})
```

| user_id | event | timestamp | price |
|---|---|---|---|
| u1 | add_to_cart | 00:00:00 | 16.67 |
| u1 | purchase | 00:00:20 | 50 |

## Parameters

| Parameter | Type | Description |
|---|---|---|
| `consecutive` | `bool or list of str, optional` | Collapse consecutive repeats of the same event into one. Pass `True` to collapse all events; pass a list of event names to collapse only those specific events. |
| `event_groups` | `list of dict, optional` | Merge a chain of events into a single representative event. See [event_groups](https://retentioneering.com/docs/data-processors/collapse-events#event_groups) below for worked examples. |
| `group_col` | `str, optional` | Group consecutive rows by this column's value: each run of rows sharing the same value is collapsed into one event named after that value. Example: a `session_type` column with values `browse, browse, search` collapses the path into `browse -> search` events. |
| `session_col` | `str, optional` | Collapse events within each session defined by this column. Requires `session_type_col` as well. |
| `session_type_col` | `str, optional` | Column that distinguishes session event types (used with `session_col`). |
| `agg` | `dict, optional` | Aggregation rules for non-event columns when rows are merged, as a `{column: agg_func}` dict. `agg_func` is one of `"first"` (default), `"last"`, `"min"`, `"max"`, `"mean"`, `"mode"`, `"any"`. See [agg](https://retentioneering.com/docs/data-processors/collapse-events#agg) below. Example: `{"price": "max"}`. |
| `path_col` | `str, optional` | Path ID column override; defaults to `schema.path_col`. |
| `event_col` | `str, optional` | Event column override; defaults to `schema.event_col`. |

---

# To Daily States

Source: https://retentioneering.com/docs/data-processors/to-daily-states

Convert the eventstream into daily lifecycle-state events.

Each path is expanded to one row per calendar day from its first event
to `max_dormant_days` days after its last event. Every row is labelled
with one of six engagement states.

Active days:
    `new`          — first-ever active day for this path
    `current`      — active within the past 7 days
    `reactivated`  — active 8-30 days ago, not in the last 7
    `resurrected`  — last active more than 30 days ago

Inactive days:
    `at_risk_wau`  — was active within the past 7 days
    `at_risk_mau`  — was active 8-30 days ago
    `dormant`      — was last active more than 30 days ago

## Usage

```python
stream.to_daily_states()
stream.to_daily_states(active_events=["purchase", "add_to_cart"], max_dormant_days=60)
```

## How it works

This processor changes what a "step" in a path means. Normally the next event in
a path is the next thing the user did, whenever that was. After
`to_daily_states`, every path has exactly one event per calendar day — the
user's engagement state on that day — so consecutive steps are consecutive days
and the gaps become visible as events in their own right.

Before — one user, active on two days:

| user_id | event | timestamp |
|---|---|---|
| u1 | login | 2024-01-01 |
| u1 | login | 2024-01-03 |

```python
stream.to_daily_states()
```

After — one row per day, labelled with that day's state:

| user_id | event | timestamp |
|---|---|---|
| u1 | `new` | 2024-01-01 |
| u1 | `at_risk_wau` | 2024-01-02 |
| u1 | `current` | 2024-01-03 |

The first-ever active day is `new`; 2 January has no activity but the user was
active within the last 7 days, so it is `at_risk_wau`; on 3 January the user is
active again and recently seen, so `current`.

Rows keep being generated for `max_dormant_days` after the path's last event, so
you can see a path decay through `at_risk_wau` → `at_risk_mau` → `dormant`
without it silently disappearing from the data. That tail is capped at the last
day present in the dataset, so paths active near the end of your observation
window get fewer trailing rows than older ones — worth remembering before
comparing state distributions across cohorts.

Pass `active_events` to decide what counts as activity: with
`active_events=["purchase"]`, a user who browses daily but never buys still
decays to `dormant`.

Because the output is an ordinary eventstream whose events are states, every
tool works on it unchanged — a [Transition Graph](https://retentioneering.com/docs/widgets/transition-graph)
becomes a state-transition diagram, and a
[Step Sankey](https://retentioneering.com/docs/widgets/step-sankey) becomes a retention flow.

## Parameters

| Parameter | Type | Description |
|---|---|---|
| `active_events` | `list of str, optional` | Events that count as "activity". If omitted, any event counts. |
| `max_dormant_days` | `int, default 30` | Days after a path's last event to continue generating state rows. Capped at the last day present in the dataset, so paths active near the end of the observation window get fewer trailing rows. |
| `agg` | `dict, optional` | Per-column aggregation overrides (e.g. `{"revenue": "sum"}`). |
| `path_col` | `str, optional` | Override the path ID column. |
| `event_col` | `str, optional` | Override the event column. |

---

# Drop Segment

Source: https://retentioneering.com/docs/data-processors/drop-segment

Remove a segment column from the eventstream.

## Usage

```python
stream.drop_segment("cluster")
```

## Parameters

| Parameter | Type | Description |
|---|---|---|
| `name` | `str` | Name of the segment column to remove. Must exist in `schema.segment_cols`. |

---

# Drop Events

Source: https://retentioneering.com/docs/data-processors/drop-events

Remove events from the eventstream by name.

Raises an error if any listed event does not exist.

## Usage

```python
stream.drop_events(["debug_event", "system_ping"])
```

## Parameters

| Parameter | Type | Description |
|---|---|---|
| `names` | `list of str` | Event names to remove entirely. |

---

# Edit Events

Source: https://retentioneering.com/docs/data-processors/edit-events

Rename and/or delete events in a single operation.

A convenience combination of `rename_events` and `drop_events` — useful when
one pass over the unique event list should both clean up names and remove
noise. At least one of `rename` or `delete` must be provided.

## Usage

```python
stream.edit_events(rename={"old_checkout": "checkout"}, delete=["system_ping"])
stream.edit_events(delete=["debug_event", "internal_event"])
```

## How it works

Before: `btn_clk → debug → purchase`

```python
stream.edit_events(rename={"btn_clk": "button_click"}, delete=["debug"])
```

After — renaming and deleting happen in the same pass, so you only walk the
unique event list once:

`button_click → purchase`

Renaming an event to a name that already exists merges the two. Events absent
from `rename` are left alone; use [`drop_events`](https://retentioneering.com/docs/data-processors/drop-events)
or [`rename_events`](https://retentioneering.com/docs/data-processors/rename-events) when you only need one
of the two operations.

## Parameters

| Parameter | Type | Description |
|---|---|---|
| `rename` | `dict, optional` | Mapping of `{old_name: new_name}`. Events whose current name is a key are renamed to the corresponding value. |
| `delete` | `list of str, optional` | Event names to remove from the eventstream entirely. |

---

# Rename Events

Source: https://retentioneering.com/docs/data-processors/rename-events

Rename events using a mapping dict.

Events not present in `mapping` are left unchanged. To also delete events in
the same step, use `edit_events` instead.

## Usage

```python
stream.rename_events({"old_checkout": "checkout", "cart_add": "add_to_cart"})
```

## Parameters

| Parameter | Type | Description |
|---|---|---|
| `mapping` | `dict` | Mapping of `{old_name: new_name}`. |

---

# Rename Segment Levels

Source: https://retentioneering.com/docs/data-processors/rename-segment-levels

Rename levels of a segment column using a mapping dict.

Levels not present in `mapping` are left unchanged. Useful for cleaning up
raw segment data, or for giving a clustering result (e.g. from `add_clusters`)
human-readable names. Renaming a level to match another existing level merges
the two.

## Usage

```python
stream.add_clusters(
    name="cluster", features=[{"metric": "length"}], n_clusters=3
).rename_segment_levels(
    "cluster", {"cluster_0": "buyers", "cluster_1": "browsers"}
)
```

## Parameters

| Parameter | Type | Description |
|---|---|---|
| `segment_col` | `str` | Name of the segment column. Must be listed in `schema.segment_cols`. |
| `mapping` | `dict` | Mapping of `{old_level: new_level}`. Keys must be levels already present in `segment_col` (see `get_segment_levels`). |

---

# Sample Paths

Source: https://retentioneering.com/docs/data-processors/sample-paths

Randomly sample paths (and all their events).

Exactly one of `n` or `frac` must be provided (mirrors
`pandas.DataFrame.sample`).

## Usage

```python
stream.sample_paths(n=1000)
stream.sample_paths(frac=0.1, random_state=42)  # 10 % of paths
```

## Parameters

| Parameter | Type | Description |
|---|---|---|
| `n` | `int, optional` | Number of paths to keep. |
| `frac` | `float, optional` | Fraction of total paths to keep, in the range `(0.0, 1.0]`. Passing `1.0` returns the eventstream unchanged. |
| `random_state` | `int, optional` | Seed for the random number generator; pass an integer for reproducible results. |
| `path_col` | `str, optional` | Path ID column override; defaults to `schema.path_col`. |

---

# Split Sessions

Source: https://retentioneering.com/docs/data-processors/split-sessions

Split each path into sub-sessions and add session ID and index columns.

At least one boundary criterion must be provided: `separator`,
`start_event` + `end_event`, or `timeout`. `separator` and
`start_event`/`end_event` are mutually exclusive; `timeout` may be combined
with either.

## Usage

```python
stream.split_sessions(timeout="30m")
stream.split_sessions(separator="app_open")
stream.split_sessions(start_event="session_start", end_event="session_end")
stream.split_sessions(separator="app_open", timeout="1h")
```

## How it works

Toy paths showing exactly what happens to the eventstream for each
session-boundary mode. Exactly one of `separator`, `start_event` + `end_event`,
or `timeout` must be given; `timeout` may additionally be combined with either
of the other two to also break a session on an inactivity gap.

### timeout

Every event belongs to a session; a new session starts whenever the gap to
the previous event exceeds `timeout`. Unlike the other two modes, no rows are
ever dropped from the output.

Before: `A` at `00:00:00`, `B` at `00:00:30`, `C` at `01:00:00`

```python
stream.split_sessions(timeout="1m")
```

After — `A` and `B` are within a minute of each other and share a session; the
hour-long gap before `C` starts a new one:

`session_id` = `user_1_1, user_1_1, user_1_2`

`timeout` can combine with `separator` or `start_event` + `end_event` to also
break a session early on an inactivity gap, without changing which events get
dropped.

### separator

A session is every event up to the next `separator` event, exclusive on both
ends — the separator marks the **start** of the following session and is
dropped from the output. Events before the first separator aren't part of any
session and pass through unchanged, with `session_id`/`session_index` left
empty.

Before: `sep → event_1 → event_2 → sep → event_3`

```python
stream.split_sessions(separator="sep")
```

After — two sessions; the `sep` events themselves are removed:

`event_1 → event_2 → event_3`, with `session_id` = `user_1_1, user_1_1, user_1_2`

### start_event + end_event

A session is every event strictly between a `start_event` and the next
`end_event` — both boundary events are dropped from the output.

Before: `start → event_1 → event_2 → end`

```python
stream.split_sessions(start_event="start", end_event="end")
```

After:

`event_1 → event_2`, with `session_id` = `user_1_1, user_1_1`

## Parameters

| Parameter | Type | Description |
|---|---|---|
| `session_col` | `str, default "session_id"` | Name of the new column that holds the unique session identifier. |
| `session_index_col` | `str, default "session_index"` | Name of the new column that holds the 1-based session index within each path. |
| `separator` | `str or list of str, optional` | Event name(s) that mark a session boundary. The separator event starts a new session; the separator row itself is dropped from the output. |
| `start_event` | `str or list of str, optional` | Event name(s) that mark the start of a session. Must be provided together with `end_event`. |
| `end_event` | `str or list of str, optional` | Event name(s) that mark the end of a session. Must be provided together with `start_event`. |
| `timeout` | `str or pandas.Timedelta, optional` | Inactivity gap after which a new session starts, as a pandas-style duration string with an explicit unit — e.g. `"30m"`, `"1h"`, `"1800s"` — or a `pandas.Timedelta`. Bare numbers are rejected to avoid unit ambiguity. |
| `path_col` | `str, optional` | Path ID column override; defaults to `schema.path_col`. |
| `event_col` | `str, optional` | Event column override; defaults to `schema.event_col`. |

---

# Truncate Paths

Source: https://retentioneering.com/docs/data-processors/truncate-paths

Trim each path to the window between two anchor events (inclusive).

For each path, the first occurrence of `start_event` and the first occurrence
of `end_event` that comes after it are found. Events outside this window are
dropped. Paths that do not contain both anchors in the correct order are
removed entirely.

Use `start_event="path_start"` / `end_event="path_end"` to refer to the actual
first and last events of the path.

## Usage

```python
stream.truncate_paths(start_event="registration", end_event="purchase")
stream.truncate_paths(start_event="registration", end_event="path_end")
```

## How it works

For each path, `truncate_paths` finds the first `start_event` and the first
`end_event` **after** it, and keeps only the window between them, both anchors
included. A path missing either anchor — or containing them in the wrong order
— is dropped entirely, so the output holds only paths that actually made the
journey you asked about.

Before — three paths, only one of which registers and then buys:

- u1: `home → registration → browse → purchase → logout`
- u2: `home → browse → purchase` (never registered)
- u3: `home → registration → browse` (never purchased)

```python
stream.truncate_paths(start_event="registration", end_event="purchase")
```

After — u1 is trimmed to the window; u2 and u3 are gone:

- u1: `registration → browse → purchase`

That "dropped entirely" behaviour is the point: the result is a clean
population of converting paths, ready for a
[Transition Graph](https://retentioneering.com/docs/widgets/transition-graph) or
[Step Sankey](https://retentioneering.com/docs/widgets/step-sankey) that answers "how do users who *did*
convert get there?"

Use `path_start` / `path_end` to anchor on the real ends of a path — for
example, everything a user did before their first purchase:

```python
stream.truncate_paths(start_event="path_start", end_event="purchase")
```

## Parameters

| Parameter | Type | Description |
|---|---|---|
| `start_event` | `str` | Name of the event that marks the start of the window. |
| `end_event` | `str` | Name of the event that marks the end of the window. |
| `path_col` | `str, optional` | Path ID column override; defaults to `schema.path_col`. |
| `event_col` | `str, optional` | Event column override; defaults to `schema.event_col`. |

---

# URLs to Events

Source: https://retentioneering.com/docs/data-processors/urls-to-events

Turn a raw URL column into structured event names using a URL path tree.

Each URL is matched against the `nodes` tree. A node with
`aggregate_children` set becomes an aggregation point: the node's own URL
keeps its path as the event name, while every deeper page collapses to the
single label `<path>/<slug>`. The original URL column is replaced in-place.

## Usage

```python
stream.urls_to_events(
    column="page",
    nodes=[
        {"path": "/catalog", "aggregate_children": True},
        {"path": "/checkout", "aggregate_children": True, "name": "checkout"},
    ],
)
```

## How it works

Web logs usually carry a URL per pageview, and raw URLs make terrible event
names: every product page is its own event, so no two paths look alike and
every [representation of them](https://retentioneering.com/docs/path-analysis) comes out shredded. `urls_to_events` turns URLs into a handful of readable event names by
declaring which parts of the site matter.

Each URL is first normalized — host, query string and locale prefix are stripped
by default — and then matched against the `nodes` tree.

Before, a raw `page` column:

| page |
|---|
| `https://shop.com/en/catalog` |
| `https://shop.com/en/catalog/phones?sort=price` |
| `https://shop.com/catalog/phones/iphone-15` |
| `https://shop.com/checkout/payment` |
| `https://shop.com/admin/logs` |

```python
stream.urls_to_events(
    column="page",
    nodes=[
        {"path": "/catalog", "aggregate_children": True},
        {"path": "/checkout"},
        {"path": "/admin", "exclude": True},
    ],
)
```

After:

| page |
|---|
| `catalog` |
| `catalog/sub-page` |
| `catalog/sub-page` |
| `checkout/payment` |
| *(row dropped)* |

Three things happened:

- `/catalog` itself, being the aggregation node's own URL, keeps its path.
- Everything **below** `/catalog` collapsed into the single event
  `catalog/sub-page`. The slug is a fixed placeholder — it is *not* the URL's own
  segment, so `/catalog/phones` and `/catalog/laptops/mac` land in the same
  event. That is the whole point: one event for "some page inside the catalog".
- `/admin` was excluded, so that row is gone from the eventstream.
- `/checkout` has no `aggregate_children`, so its pages keep their full paths.

### Naming the collapsed pages

`name` sets the slug used for the pages collapsed into a node — it does not
rename the node's own URL:

```python
nodes=[{"path": "/catalog", "aggregate_children": True, "name": "product"}]
# /catalog            -> catalog
# /catalog/phones     -> catalog/product
```

To pull one branch back out of the aggregation, declare it as its own node.
Giving it a `name` gives that branch its own slug; giving it
`aggregate_children` makes it a nested aggregation point:

```python
nodes=[
    {"path": "/catalog", "aggregate_children": True},
    {"path": "/catalog/phones", "name": "phones"},
]
# /catalog/phones/iphone-15 -> catalog/phones
# /catalog/laptops/mac      -> catalog/sub-page
```

Set `keep_full_paths=True` to ignore every `aggregate_children` and keep the
normalized path of each URL — useful for a first look at what is actually in the
column before deciding where to cut. The `host_col`, `query_col`, `locale_col`
and `slug_col` parameters save the stripped-off parts into columns of their own
instead of discarding them.

## Parameters

| Parameter | Type | Description |
|---|---|---|
| `column` | `str` | Name of the column that contains raw URL strings. |
| `nodes` | `list of dict` | URL path tree. Each node dict must have a `"path"` key (str) and may include: |
| `strip_host` | `bool, default True` | Remove the scheme and hostname, keeping only the pathname. |
| `strip_query` | `bool, default True` | Remove the query string and URL fragment. |
| `strip_locale` | `bool, default True` | Remove a leading 2-letter BCP-47 locale segment (e.g. `"en"`, `"fr-ca"`). |
| `keep_full_paths` | `bool, default False` | When `True`, `aggregate_children` nodes are ignored and every URL keeps its normalized path. |
| `host_col` | `str, optional` | If provided, save the extracted hostname into this new column. |
| `query_col` | `str, optional` | If provided, save the extracted query string into this new column. |
| `locale_col` | `str, optional` | If provided, save the detected locale prefix into this new column. |
| `slug_col` | `str, optional` | If provided, save the sub-page slug into this new column. |