<!-- # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0 -->

# Standalone Router

A backend-agnostic standalone KV-aware router service for Dynamo deployments. For details on how KV-aware routing works, see [Routing Concepts](/docs/components/router/router-concepts.md).

## Overview

The standalone router provides configurable KV-aware routing for any set of workers in a Dynamo deployment. It can be used for disaggregated serving (e.g., routing to prefill workers), multi-tier architectures, or any scenario requiring intelligent KV cache-aware routing decisions.

This component is **fully configurable** and works with any Dynamo backend (vLLM, TensorRT-LLM, SGLang, etc.) and any worker endpoint.

## Usage

### Command Line

```bash
python -m dynamo.router \
    --endpoint dynamo.prefill.generate \
    --router-block-size 64 \
    --no-router-track-active-blocks
```

### Arguments

**Required:**
- `--endpoint`: Full endpoint path for workers in the format `namespace.component.endpoint` (e.g., `dynamo.prefill.generate`)

**Router Configuration:**
Most KV tuning options use the `--router-*` prefix, but shared options such as
`--load-aware`, `--serve-indexer`, `--use-remote-indexer`, and `--shared-cache-*` do
not. Standalone-only options include `--endpoint` and `--router-block-size`. Legacy
names such as `--block-size` and `--kv-events` are still accepted but deprecated.
Run `python -m dynamo.router --help` for the standalone command surface. The
[Frontend Configuration Reference](/docs/components/frontend/configuration.md#router)
is the canonical reference for shared embedded-router flags and environment variables;
see [Configuration and Tuning](/docs/components/router/router-configuration.md) for
behavioral guidance.

## Architecture

The standalone router exposes three endpoints via the Dynamo runtime:

1. **`generate`**: Routes requests to the best worker and streams back generation results (KV-aware routing).
2. **`best_worker_id`**: Given token IDs, returns the best worker ID for the request without routing; useful for debugging or custom routing logic.
3. **`get_overlap_scores`**: Given token IDs, returns per-worker/per-DP-rank matched block counts for device, host-pinned, disk, and configured shared-cache tiers without routing.

Clients call the `generate` endpoint to stream completions, call `best_worker_id` to decide which worker to use and then contact that worker directly, or call `get_overlap_scores` when an external scheduler wants the raw tiered overlap signal.

## Example: Manual Disaggregated Serving (Alternative Setup)

> [!Note]
> **This is an alternative advanced setup.** The recommended approach for disaggregated serving is to use the frontend's automatic prefill routing, which activates when you register workers with `WorkerType.Prefill`. See [Disaggregated Serving](/docs/components/router/router-disaggregated-serving.md) for the default setup.
>
> Use this manual setup if you need explicit control over prefill routing configuration or want to manage prefill and decode routers separately.

For an integrated frontend disaggregated example, see [`examples/backends/vllm/launch/disagg_router.sh`](/examples/backends/vllm/launch/disagg_router.sh). For explicit multi-router composition, see the [Global Router README](/components/src/dynamo/global_router/README.md).

```bash
# Start frontend router for decode workers
python -m dynamo.frontend \
    --router-mode kv \
    --http-port 8000 \
    --router-kv-overlap-score-credit 0  # Pure load balancing for decode

# Start standalone router for prefill workers
python -m dynamo.router \
    --endpoint dynamo.prefill.generate \
    --router-block-size 64 \
    --no-router-track-active-blocks

# Start decode workers
python -m dynamo.vllm --model MODEL_NAME --block-size 64 &

# Start prefill workers
python -m dynamo.vllm --model MODEL_NAME --block-size 64 --disaggregation-mode prefill &
```

For event-driven prefix-cache state, add the backend-specific KV event publishing flags to the workers that the router indexes. Use `--no-router-kv-events` on the router only when approximate cache-state prediction is acceptable.

>[!Note]
> **Why `--no-router-track-active-blocks` for prefill routing?**
> Active block tracking is used for load balancing across decode (generation) phases. For prefill-only routing, decode load is not relevant, so disabling this reduces overhead and simplifies the router state.
>
> **When should I use `--no-router-track-prefill-tokens`?**
> Use it on decode-only routers that should ignore already-completed prompt work. This keeps `active_prefill_tokens`, queue pressure, and load estimates focused on decode-side work after a prefill-to-decode handoff.
>
> **Why `--router-block-size` should be set for standalone routers:**
> Standalone routers default to block size `128`, but they do not infer block size from the ModelDeploymentCard (MDC) during worker registration. Set the value explicitly so routing decisions match the backend worker block size.

## Configuration Best Practices

>[!Note]
> **Block Size Matching:**
> The block size must match across:
> - Standalone router (`--router-block-size`)
> - All worker instances (backend-specific, e.g. `--block-size` for vLLM)
>
> **Endpoint Matching:**
> The `--endpoint` argument must match where your target workers register. For example:
> - vLLM prefill workers: `dynamo.prefill.generate`
> - vLLM decode workers: `dynamo.backend.generate`
> - Custom workers: `<your_namespace>.<your_component>.<your_endpoint>`

## Integration with Backends

To integrate the standalone router with a backend:

1. Workers should register at the endpoint specified by the `--endpoint` argument
2. Clients call the `router.generate` endpoint to stream completions (router selects the best worker), or call `router.best_worker_id` to get the best worker ID and then send requests to that worker
3. Router state is updated automatically as requests are routed; no separate "free" call is required

See [`components/src/dynamo/vllm/handlers.py`](../vllm/handlers.py) for a reference implementation (search for `prefill_router_client`).

## See Also

- [Router Guide](/docs/components/router/router-guide.md) - Deployment modes and quick start
- [Configuration and Tuning](/docs/components/router/router-configuration.md) - CLI flags, transport modes, and metrics
- [Disaggregated Serving](/docs/components/router/router-disaggregated-serving.md) - Prefill and decode routing setups
- [Router Design](/docs/design-docs/router-design.md) - Architecture details and event transport modes
- [Frontend Router](../frontend/README.md) - Main HTTP frontend with integrated routing
- [Router Benchmarking](/benchmarks/router/README.md) - Performance testing and tuning
