<!--
SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0
-->

# Global Router

A hierarchical routing service that sits between the Dynamo frontend and local routers in different pool namespaces. The global router supports both disaggregated and aggregated serving with flexible pool selection based on request characteristics.

## Overview

The Global Router supports two modes:

- **Disagg mode** (default): Registers as both prefill and decode worker. Routes prefill requests based on (ISL, TTFT) and decode requests based on (request token count, ITL) to separate pool types. The decode strategy retains the `context_length_*` configuration names.
- **Agg mode**: Registers as a single generate worker. Routes all requests by (TTFT target, ITL target), with an optional ISL dimension, to unified pools that handle both prefill and decode.

Both modes support priority-based pool overrides from agent hints and optional priority retry to faster pools.

## Supported Backends

- **vLLM** - Uses synchronous prefill path (frontend waits for prefill to complete)
- **Mocker** - Uses same synchronous path as vLLM
- **SGLang** - Uses the bootstrap path (async KV transfer)

**Not currently supported or validated:**
- **TensorRT-LLM** - The core frontend supports a non-bootstrap handoff path, but Global Router has no dedicated TensorRT-LLM end-to-end integration.

## Architecture

### Disagg Mode

```
Frontend
    |
    v
Global Router (registers as both prefill + decode)
    |
    +---> Prefill Pool 0 (namespace: prefill_pool_0)
    |         |
    |         +---> Local Router ---> Prefill Worker 0
    |                           +---> Prefill Worker 1
    |
    +---> Prefill Pool ...
    |
    +---> Decode Pool 0 (namespace: decode_pool_0)
    |         |
    |         +---> Local Router ---> Decode Worker 0
    |                           +---> Decode Worker 1
    |
    +---> Decode Pool ...
```

### Agg Mode

```
Frontend
    |
    v
Global Router (registers as Chat + Completions)
    |
    +---> Agg Pool 0 (namespace: agg_pool_0)
    |         |
    |         +---> Local Router ---> Worker 0 (prefill + decode)
    |                           +---> Worker 1 (prefill + decode)
    |
    +---> Agg Pool 1 (namespace: agg_pool_1)
    |         |
    |         +---> Local Router ---> Worker 0 (prefill + decode)
    |                           +---> Worker 1 (prefill + decode)
    |
    +---> Agg Pool ...
```

## Usage

```bash
python -m dynamo.global_router \
  --config path/to/global_router_config.json \
  --model-name Qwen/Qwen3-0.6B \
  --namespace dynamo
```

### Arguments

All options can be set via CLI flags or environment variables. CLI flags take precedence over environment variables.

| Argument | Required (CLI or env) | Env var | Default | Description |
|----------|----------------------|---------|---------|-------------|
| `--config` | Yes | `DYN_GLOBAL_ROUTER_CONFIG` | - | Path to JSON configuration file |
| `--model-name` | Yes | `DYN_GLOBAL_ROUTER_MODEL_NAME` | - | Model name for registration (must match workers) |
| `--namespace` | No | `DYN_NAMESPACE` | "dynamo" | Namespace for global router |
| `--component-name` | No | `DYN_GLOBAL_ROUTER_COMPONENT_NAME` | "global_router" | Component name |
| `--default-ttft-target-ms` | No | `DYN_GLOBAL_ROUTER_DEFAULT_TTFT_TARGET_MS` | None | Default TTFT target (ms) for prefill pool selection |
| `--default-itl-target-ms` | No | `DYN_GLOBAL_ROUTER_DEFAULT_ITL_TARGET_MS` | None | Default ITL target (ms) for pool selection |

## Configuration

The configuration file format depends on the mode. The `mode` field determines which mode is used; if omitted, it defaults to `"disagg"`.

### Disagg Mode Configuration

```jsonc
{
    "mode": "disagg",                     // Optional, defaults to "disagg"
    "enable_priority_retry": false,        // Optional, retry failed requests on faster pools
    "num_prefill_pools": <int>,
    "num_decode_pools": <int>,
    "prefill_pool_dynamo_namespaces": [],
    "decode_pool_dynamo_namespaces": [],
    "prefill_pool_priorities": [],         // Optional, lower integer = faster pool
    "decode_pool_priorities": [],          // Optional, lower integer = faster pool

    "prefill_pool_selection_strategy": {
        "isl_min": <int>,
        "isl_max": <int>,
        "isl_resolution": <int>,
        "ttft_min_ms": <float>,
        "ttft_max_ms": <float>,
        "ttft_resolution": <int>,
        "prefill_pool_mapping": [[]],     // 2D array [isl_resolution][ttft_resolution] -> pool index
        "priority_overrides": []          // Optional
    },

    "decode_pool_selection_strategy": {
        "context_length_min": <int>,
        "context_length_max": <int>,
        "context_length_resolution": <int>,
        "itl_min_ms": <float>,
        "itl_max_ms": <float>,
        "itl_resolution": <int>,
        "decode_pool_mapping": [[]],      // 2D array [context_length_resolution][itl_resolution] -> pool index
        "priority_overrides": []          // Optional
    }
}
```

### Agg Mode Configuration

```jsonc
{
    "mode": "agg",
    "enable_priority_retry": false,        // Optional, retry failed requests on faster pools
    "num_agg_pools": <int>,
    "agg_pool_dynamo_namespaces": [],
    "agg_pool_priorities": [],             // Optional, lower integer = faster pool

    "agg_pool_selection_strategy": {
        "ttft_min_ms": <float>,              // Minimum TTFT target (ms)
        "ttft_max_ms": <float>,              // Maximum TTFT target (ms)
        "ttft_resolution": <int>,         // Number of grid rows for TTFT dimension
        "itl_min_ms": <float>,              // Minimum ITL target (ms)
        "itl_max_ms": <float>,              // Maximum ITL target (ms)
        "itl_resolution": <int>,          // Number of grid columns for ITL dimension
        "isl_min": <int>,                  // Optional: minimum input sequence length
        "isl_max": <int>,                  // Optional: maximum input sequence length
        "isl_resolution": <int>,           // Optional: number of ISL grid rows
        "agg_pool_mapping": [[]],         // 2D [ttft][itl], or 3D [isl][ttft][itl] when all isl_* fields are set
        "priority_overrides": []          // Optional
    }
}
```

### Aggregate Pool Selection

In aggregated mode, the same pool handles both prefill and decode. By default, the router uses TTFT × ITL because both SLA targets matter for a single routing decision:

- **TTFT target** captures the user's prefill latency requirement. ISL is implicitly accounted for — a user sending a large prompt with a tight TTFT target is saying "I need a fast pool."
- **ITL target** captures the user's decode latency requirement. With chunked prefill, ITL reflects the combined prefill+decode contention. Without chunked prefill, ITL reflects pure decode performance.

This creates natural pool separation:
- Tight TTFT + tight ITL -> premium interactive pool
- Relaxed TTFT + tight ITL -> decode-optimized pool
- Tight TTFT + relaxed ITL -> prefill-optimized pool
- Relaxed TTFT + relaxed ITL -> batch/throughput pool

For workload classes whose prompt length matters independently of the SLA targets, configure all three `isl_*` fields. This adds ISL as the leading mapping dimension, yielding `agg_pool_mapping[isl_idx][ttft_idx][itl_idx]`. For example, this routes 4K prompts to pool 0 and 16K prompts to pool 1 while keeping one SLA bucket:

```json
{
  "ttft_min_ms": 1,
  "ttft_max_ms": 10000,
  "ttft_resolution": 1,
  "itl_min_ms": 1,
  "itl_max_ms": 100,
  "itl_resolution": 1,
  "isl_min": 0,
  "isl_max": 24576,
  "isl_resolution": 2,
  "agg_pool_mapping": [[[0]], [[1]]]
}
```

The `isl_*` fields must be configured together. Existing aggregate configurations without them keep the 2D TTFT × ITL mapping.

### Pool Selection

The default pool selection uses a 2D grid lookup. Each dimension is divided into buckets based on the resolution.

**Prefill Pool Selection** (disagg mode, based on ISL and TTFT target):

1. Compute `isl_step = (isl_max - isl_min) / isl_resolution`
2. Compute `ttft_step_ms = (ttft_max_ms - ttft_min_ms) / ttft_resolution`
3. For a request with input sequence length `ISL` and target TTFT:
   - `isl_idx = clamp((ISL - isl_min) / isl_step, 0, isl_resolution - 1)`
   - `ttft_idx = clamp((ttft_target_ms - ttft_min_ms) / ttft_step_ms, 0, ttft_resolution - 1)`
4. Lookup pool: `pool_index = prefill_pool_mapping[isl_idx][ttft_idx]`

**Decode Pool Selection** (disagg mode, based on request token count and ITL target):

Valid `PreprocessedRequest` payloads require `token_ids`. The handler passes the
length of that list into the strategy's `context_length` dimension and does not
add subsequently generated tokens. Its `request.get("token_ids", [])` fallback
only protects malformed direct calls; a missing field routes those calls with a
request token count of `0`. The bucket calculation otherwise uses
`context_length_*` and `itl_target` with `decode_pool_mapping`.

**Agg Pool Selection** (agg mode, based on TTFT and ITL targets, with optional ISL):

Without `isl_*`, the grid uses `ttft_target` and `itl_target` with `agg_pool_mapping[ttft_idx][itl_idx]`. With all three `isl_*` fields, it first computes `isl_idx` using the same clamped bucket logic as prefill selection, then uses `agg_pool_mapping[isl_idx][ttft_idx][itl_idx]`.

### Priority-Based Pool Override

All strategies support optional `priority_overrides` rules. When a request carries a priority value (from `nvext.agent_hints.priority`), the global router evaluates the override rules after the grid lookup. The first rule whose `[min_priority, max_priority]` range contains the request priority wins, and the request is routed to that rule's `target_pool` instead of the grid result. If no rule matches (or no priority is present), the grid result is used as normal.

This is useful for straggler mitigation in RL workloads: the RL framework can tag slow requests with a high priority, and the global router redirects them to a dedicated min-latency pool.

```jsonc
"priority_overrides": [
    {
        "min_priority": 10,     // inclusive lower bound
        "max_priority": 100,    // inclusive upper bound
        "target_pool": 1        // pool index to route to
    }
]
```

Priority is set by the client via the NVIDIA OpenAI extension:

```json
{
    "messages": [...],
    "nvext": {
        "agent_hints": {
            "priority": 50
        }
    }
}
```

### Priority Retry

Set `enable_priority_retry` to `true` to retry a request on faster pools when forwarding to the selected pool fails before any response has been streamed. Pool speed is configured with `prefill_pool_priorities`, `decode_pool_priorities`, or `agg_pool_priorities` depending on the mode. Lower integer values mean faster pools.

If a priority list is omitted, the global router uses pool order as the default priority: pool `0` has priority `0`, pool `1` has priority `1`, and so on. With default priorities, pool `0` is the fastest pool.

Retry order starts with the selected pool and then walks faster pools from slow to fast. For example:

```jsonc
{
    "enable_priority_retry": true,
    "prefill_pool_dynamo_namespaces": ["prefill-fast", "prefill-mid", "prefill-slow"],
    "prefill_pool_priorities": [0, 1, 2]
}
```

If the grid selects `prefill-slow` (pool `2`) and that request fails, the global router retries pool `1` and then pool `0`. If the grid selects the fastest pool, there is no faster pool to retry.

In disaggregated mode, prefill and decode retry are independent. A prefill failure retries through `prefill_pool_priorities`; a decode failure retries through `decode_pool_priorities`. If a failed decode attempt already caused the prefill engine to retire or drop the request's KV cache, the retry is still allowed; the backend should handle any resulting cache miss or failure on the new attempt. Current Dynamo backends do not support this yet.

### Passing SLA Targets

Clients can pass TTFT and ITL targets via `nvext.router` in the request:

```json
{
    "messages": [...],
    "nvext": {
        "router": {
            "ttft_target": 100,
            "itl_target": 20
        }
    }
}
```

The preprocessor forwards `nvext.router` to the backend as the typed `router` field on `PreprocessedRequest`. If not provided, the middle of the configured range is used as default. For disagg mode, `ttft_target` drives prefill pool selection and `itl_target` drives decode pool selection. For agg mode, both `ttft_target` and `itl_target` drive pool selection.

## Request Flow

### Disagg Mode

1. Frontend receives request and sends to Global Router (registered as prefill)
2. Global Router selects prefill pool based on (ISL, TTFT_target, priority)
3. Request is forwarded to local router in the selected prefill pool namespace
4. If forwarding fails before streaming a response and priority retry is enabled, Global Router retries faster prefill pools
5. Local router forwards to a prefill worker
6. Prefill response returns with `disaggregated_params`
7. Frontend sends decode request to Global Router (registered as decode)
8. Global Router selects decode pool based on (request token count, ITL_target, priority)
9. Request is forwarded to local router in the selected decode pool namespace
10. If forwarding fails before streaming tokens and priority retry is enabled, Global Router retries faster decode pools
11. Tokens stream back through the chain

### Agg Mode

1. Frontend receives request and sends to Global Router (registered as Chat + Completions)
2. Global Router selects agg pool based on (ISL, TTFT_target, ITL_target, priority) when ISL routing is configured; otherwise it uses (TTFT_target, ITL_target, priority)
3. Request is forwarded to local router in the selected agg pool namespace
4. If forwarding fails before streaming tokens and priority retry is enabled, Global Router retries faster agg pools
5. Local router forwards to a worker that handles both prefill and decode
6. Tokens stream back through the chain

## Example

See `examples/global_planner/` for a complete example with:
- Global router configuration
- Local router setup for each pool
- Mocker workers for testing
