# Embedding Models

This page provides detailed technical specifications for the embedding model family supported by {{ncm_short_name}}. For information about supported features and capabilities, refer to [Tested Models](index.md).

## Llama Nemotron Embedding 1B v2

| Property | Value |
|----------|--------|
| Creator | NVIDIA |
| Architecture | Transformer encoder (fine-tuned Llama 3.2 1B, 16 layers) |
| Description | Optimized for **multilingual and cross-lingual** text question-answering retrieval with support for long documents (up to 8192 tokens) and dynamic embedding size (Matryoshka Embeddings). Reduces data storage footprint by 35x through dynamic embedding sizing. Ready for commercial use. |
| Max Sequence Length | 8192 |
| Embedding Dimensions | 2048 (configurable: 384, 512, 768, 1024, or 2048) |
| Parameters | 1 billion |
| Supported Languages | English, Arabic, Bengali, Chinese, Czech, Danish, Dutch, Finnish, French, German, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Norwegian, Persian, Polish, Portuguese, Russian, Spanish, Swedish, Thai, Turkish (26 languages) |
| Training Data | Semi-supervised pre-training on 12M samples and fine-tuning on 1M samples from public QA datasets with commercial licenses |
| License | [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/), [Llama 3.2 Community License](https://www.llama.com/llama3_2/license/) |
| Default Name | nvidia/llama-nemotron-embed-1b-v2 |
| HuggingFace | [nvidia/llama-nemotron-embed-1b-v2](https://huggingface.co/nvidia/llama-nemotron-embed-1b-v2) |
| NIM | [nvidia/llama-nemotron-embed-1b-v2](https://catalog.ngc.nvidia.com/orgs/nim/teams/nvidia/containers/llama-nemotron-embed-1b-v2) |

### Model Entity Configuration

Create a Model Entity for this embedding model:

| Configuration | Value |
|---------------|--------|
| Workspace | default |
| Name | llama-nemotron-embed-1b-v2 |
| Base Model | nvidia/llama-nemotron-embed-1b-v2 |
| Number of Parameters | 1,000,000,000 |
| Precision | bf16-mixed |

### Training Options

- **LoRA (merged)**: 1x 80GB GPU, tensor parallel size 1
- **Full SFT**: 1x 80GB GPU, tensor parallel size 1

!!! note
    Embedding models only support **merged LoRA** (`peft` with `merge=True`). Unmerged LoRA adapters are not supported because the embedding NIM requires ONNX format, which cannot represent standalone adapters.

### Resource Requirements

- **Minimum GPU Memory**: 80GB
- **Recommended GPU**: A100
- **Training Time**: Varies based on dataset size and epochs

### Hyperparameter and Data Recommendations

This fine-tuning recipe supports full fine-tuning, updating all 1 billion parameters, and requires careful hyperparameter and data selection to prevent overfitting.

The following table provides conservative hyperparameter defaults specifically optimized to prevent overfitting for embedding models:

| Parameter | API Field Name | Type | Description | Recommended Value |
| --- | --- | --- | --- | --- |
| Learning Rate | `learning_rate` | `number` | Step size for updating model parameters. Lower values help prevent overfitting in embedding models. | `5e-6` |
| Weight Decay | `weight_decay` | `number` | Regularization parameter to prevent overfitting by penalizing large weights. | `0.01` |
| Number of Epochs | `epochs` | `integer` | Number of complete passes through the training dataset. Limited to prevent overfitting. | `1` |
| Training Data Size | N/A | N/A | Number of training examples to prevent overfitting while maintaining model performance. | `5,000-10,000` examples |

NVIDIA recommends evaluating fine-tuned embedding models against the baseline to detect overfitting and potential performance degradation.

### Deployment Configuration

- **Full SFT and LoRA (merged)**:
 - NIM Image: `nvcr.io/nim/nvidia/llama-nemotron-embed-1b-v2:1.13.0`
 - GPU Count: 1x 80GB

### Deployment and Inference

This model supports inference deployment through NVIDIA Inference Microservices (NIM). After customization, access your model through the **Inference Gateway**:

1. **Deploy the model**: Create a ModelDeploymentConfig and ModelDeployment to deploy your fine-tuned model. See [about](../../run-inference/about.md) for details.
2. **Access through Inference Gateway**: The Inference Gateway provides unified access to all deployed models via three routing patterns:

 - **Model Entity routing**: `/v2/workspaces/{workspace}/inference/gateway/model/{name}/-/v1/embeddings`
 - **Provider routing**: `/v2/workspaces/{workspace}/inference/gateway/provider/{deployment}/-/v1/embeddings`

```python
import os
from nemo_platform import NeMoPlatform

client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)

# Get pre-configured OpenAI client for inference
oai_client = client.models.get_openai_client()
```

!!! note
    The embedding model requires NIM container images that support embedding inference. When the deployment reaches `READY` state, a ModelProvider is automatically created for routing inference requests.

### Example Usage

After fine-tuning and deployment, you can use the model for embedding tasks:

```python
import os
from nemo_platform import NeMoPlatform

client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)

MODEL_NAME = "llama-nemotron-embed-1b-v2"
DEPLOYMENT_NAME = "my-embedding-deployment"

# Call embeddings via Inference Gateway
response = client.inference.gateway.provider.post(
    "v1/embeddings",
    name=DEPLOYMENT_NAME,
    workspace="default",
    body={
        "model": MODEL_NAME,
        "input": ["What is the capital of France?"],
        "input_type": "query",
    },
)

embedding = response["data"][0]["embedding"]
print(f"Embedding dimension: {len(embedding)}")
```

For detailed fine-tuning instructions, refer to the [Embedding Customization tutorial](../tutorials/embedding-customization-job.ipynb).

For more information about formatting training datasets for the embedding model, refer to [Dataset Format Requirements](data-format.md).
