---
title: "Import and Fine-Tune Private Hugging Face Models"
description: ""
---

Use this tutorial to learn how to import a private Hugging Face model into NeMo Customizer, fine-tune it with LoRA, and deploy it for inference.

## Prerequisites

<Markdown src="/snippets/_snippets/tutorials/prereqs.mdx" />

### Tutorial-Specific Prerequisites

- Completed the [Quickstart](/documentation/get-started) to install and deploy NeMo Platform locally.
- Installed the Python SDK and any tutorial packages you need in your environment.
- A Hugging Face token with access to the private or gated model repository.
- A Hugging Face model with a compatible architecture. This tutorial uses `google/gemma-2-2b-it` as an example, but success depends on architectural compatibility.
- Sufficient GPU memory for the model and LoRA training job.

<Note>

This tutorial uses the current NeMo Platform SDK resources: `client.files.filesets`, `client.models`, `client.customization.automodel.jobs`, and `client.inference.gateway`.

</Note>

### Known Issues

<Warning>

**Conv1D Model Architecture Limitation**: Models that use `Conv1D` layers are not compatible with NeMo Customizer AutoModel LoRA.

**Error signature**: `AttributeError: 'Conv1D' object has no attribute 'config'`

**Affected models include:**

- `microsoft/DialoGPT-*` series
- `openai-gpt` models
- Some older `gpt2` variants
- Other models with Conv1D-based architectures

**Root cause**: These models use Conv1D layers that lack the linear layers expected by NeMo's LoRA transformation utilities.

**Solution**: Use modern transformer architectures instead:

- Llama models (3.1, 3.2, 3.3 series)
- Nemotron models
- Phi models
- Gemma models (used in this tutorial)

For a complete list of tested models, see the [Model Catalog](/documentation/customizer-reference/models/model-catalog).

</Warning>

## Quick Start

### 1. Initialize the SDK

The SDK needs your NeMo Platform server URL. By default, this tutorial uses `http://localhost:8080`.

```sh
export NMP_BASE_URL=<YOUR_NMP_BASE_URL>
export HF_TOKEN=<YOUR_HUGGINGFACE_TOKEN>
```

```python
import json
import os
import time
import uuid
from pathlib import Path

from IPython.display import clear_output
from nemo_automodel_plugin.schema import (
    AutomodelJobInput,
    BatchSpec,
    DatasetSpec,
    OptimizerSpec,
    OutputRequest,
    ParallelismSpec,
    ScheduleSpec,
    TrainingSpec,
)
from nemo_platform import ConflictError, NeMoPlatform
from nemo_platform.types.files import HuggingfaceStorageConfigParam
from nemo_platform.types.secrets import PlatformSecretResponse


def max_wait_time_checker(seconds: int, job_name: str = ""):
    start_time = time.time()

    def check():
        if time.time() - start_time > seconds:
            raise TimeoutError(f"{job_name} took longer than {seconds} seconds")

    return check


NMP_BASE_URL = os.environ.get("NMP_BASE_URL", "http://localhost:8080")
client = NeMoPlatform(base_url=NMP_BASE_URL, workspace="default")
```

### 2. Store the Hugging Face Token

Private and gated Hugging Face repositories require a token. Store it as a NeMo Platform secret and reference that secret from the Hugging Face fileset.

```python
def create_or_get_secret(name: str, value: str | None, label: str) -> PlatformSecretResponse:
    if not value:
        raise ValueError(f"{label} is not set")
    try:
        secret = client.secrets.create(
            name=name,
            workspace="default",
            value=value,
        )
        print(f"Created secret: {name}")
        return secret
    except ConflictError:
        print(f"Secret '{name}' already exists, continuing...")
        return client.secrets.retrieve(name=name, workspace="default")


hf_secret = create_or_get_secret("hf-token", os.getenv("HF_TOKEN"), "HF_TOKEN")
```

### 3. Create a Model FileSet and Model Entity

Create a Hugging Face-backed fileset for the private model, then register a Model Entity that points to that fileset. Model files are downloaded by the platform when training or deployment needs them.

```python
HF_REPO_ID = "google/gemma-2-2b-it"
MODEL_NAME = "gemma-2-2b-it"
model_storage = HuggingfaceStorageConfigParam(
    type="huggingface",
    repo_id=HF_REPO_ID,
    repo_type="model",
    token_secret=hf_secret.name,
)

try:
    base_model_fs = client.files.filesets.create(
        workspace="default",
        name=MODEL_NAME,
        description=f"Private Hugging Face model {HF_REPO_ID}",
        storage=model_storage,
        cache=True,
    )
    print(f"Created model fileset: {base_model_fs.name}")
except ConflictError:
    print(f"Model fileset '{MODEL_NAME}' already exists, refreshing Hugging Face settings...")
    client.files.filesets.delete(workspace="default", name=MODEL_NAME)
    base_model_fs = client.files.filesets.create(
        workspace="default",
        name=MODEL_NAME,
        description=f"Private Hugging Face model {HF_REPO_ID}",
        storage=model_storage,
        cache=True,
    )

try:
    base_model = client.models.create(
        workspace="default",
        name=MODEL_NAME,
        fileset=f"default/{MODEL_NAME}",
        trust_remote_code=False,
    )
    print(f"Created Model Entity: {base_model.name}")
except ConflictError:
    print(f"Model Entity '{MODEL_NAME}' already exists, updating fileset if needed...")
    base_model = client.models.update(
        workspace="default",
        name=MODEL_NAME,
        fileset=f"default/{MODEL_NAME}",
        trust_remote_code=False,
    )

time_check = max_wait_time_checker(600, "ModelSpec")
while not base_model.spec:
    time_check()
    time.sleep(10)
    base_model = client.models.retrieve(workspace="default", name=MODEL_NAME)

if base_model.spec:
    base_model.spec.linear_layers = None
    base_model = client.models.update(
        workspace="default",
        name=MODEL_NAME,
        spec=base_model.spec,
        verbose=True,
    )
print(base_model.model_dump_json(indent=2))
```

### 4. Prepare and Upload Training Data

Create chat-format JSONL files and upload them to a fileset. The file service, not a dataset-specific SDK resource, is the current entry point for training data.

```python
DATASET_NAME = f"{MODEL_NAME}-training-data"
DATASET_PATH = Path(DATASET_NAME).absolute()
DATASET_PATH.mkdir(parents=True, exist_ok=True)

training_data = [
    {
        "messages": [
            {"role": "user", "content": "Hello, how are you?"},
            {"role": "assistant", "content": "I'm doing well. How can I help you today?"},
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "What is machine learning?"},
            {
                "role": "assistant",
                "content": "Machine learning is a way for computers to learn patterns from data.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "Can you help me with Python?"},
            {"role": "assistant", "content": "Yes. Share the code or problem you want to work on."},
        ]
    },
]

validation_data = [
    {
        "messages": [
            {"role": "user", "content": "What is artificial intelligence?"},
            {
                "role": "assistant",
                "content": "Artificial intelligence is software that performs tasks associated with human intelligence.",
            },
        ]
    },
    {
        "messages": [
            {"role": "user", "content": "Explain renewable energy."},
            {
                "role": "assistant",
                "content": "Renewable energy comes from naturally replenished sources such as sunlight, wind, and water.",
            },
        ]
    },
]

with open(DATASET_PATH / "training.jsonl", "w", encoding="utf-8") as f:
    for item in training_data:
        f.write(json.dumps(item) + "\n")

with open(DATASET_PATH / "validation.jsonl", "w", encoding="utf-8") as f:
    for item in validation_data:
        f.write(json.dumps(item) + "\n")

try:
    client.files.filesets.create(
        workspace="default",
        name=DATASET_NAME,
        description="Private Hugging Face model LoRA training data",
        cache=True,
    )
    print(f"Created dataset fileset: {DATASET_NAME}")
except ConflictError:
    print(f"Dataset fileset '{DATASET_NAME}' already exists, continuing...")

client.files.fsspec.put(
    lpath=DATASET_PATH,
    rpath=f"default/{DATASET_NAME}/",
    recursive=True,
)
print(client.files.list(fileset=DATASET_NAME, workspace="default"))
```

### 5. Start a LoRA Customization Job

Submit a LoRA job to the Automodel backend. Reference the model and dataset filesets using `workspace/name`.

```python
job_suffix = uuid.uuid4().hex[:4]
JOB_NAME = f"{MODEL_NAME}-lora-job-{job_suffix}"
OUTPUT_NAME = f"{MODEL_NAME}-lora-{job_suffix}"

spec = AutomodelJobInput(
    model=f"default/{base_model.name}",
    dataset=DatasetSpec(
        training=f"default/{DATASET_NAME}",
        validation=f"default/{DATASET_NAME}",
    ),
    training=TrainingSpec(
        training_type="sft",
        finetuning_type="lora",
        max_seq_length=2048,
    ),
    schedule=ScheduleSpec(epochs=1),
    batch=BatchSpec(global_batch_size=64, micro_batch_size=1),
    optimizer=OptimizerSpec(learning_rate=5e-5),
    parallelism=ParallelismSpec(
        num_gpus_per_node=1,
        num_nodes=1,
        tensor_parallel_size=1,
        pipeline_parallel_size=1,
        context_parallel_size=1,
        expert_parallel_size=1,
    ),
    output=OutputRequest(name=OUTPUT_NAME),
)

job = client.customization.automodel.jobs.create(
    spec=spec,
    workspace="default",
    name=JOB_NAME,
)

print(f"Submitted job: {job.job.name}")
print(f"Output adapter: {OUTPUT_NAME}")
```

### 6. Track Training Progress

Poll job status until training reaches a terminal state.

```python
time_check = max_wait_time_checker(3600, "Customization Job")

while True:
    time_check()
    status = client.jobs.get_status(name=job.job.name, workspace="default")
    clear_output(wait=True)
    print(f"Job status: {status.status}")

    step = max_steps = None
    training_phase = None
    for job_step in status.steps or []:
        if job_step.name == "training":
            for task in job_step.tasks or []:
                details = task.status_details if isinstance(task.status_details, dict) else {}
                step = details.get("step")
                max_steps = details.get("max_steps")
                training_phase = details.get("phase")
                break
            break

    if isinstance(step, (int, float)) and isinstance(max_steps, (int, float)) and max_steps:
        print(f"Training: step {step}/{max_steps} ({100 * step / max_steps:.1f}%)")
    if isinstance(training_phase, str):
        print(f"Phase: {training_phase}")

    if status.status in ("completed", "failed", "cancelled", "error"):
        print(f"\nJob finished: {status.status}")
        break

    time.sleep(10)

if status.status != "completed":
    raise RuntimeError(f"Training job finished with status: {status.status}")
```

### 7. Deploy the Base Model With LoRA Enabled

Create a deployment for the base model with LoRA support enabled. The LoRA adapter from training is served by the same deployment.

```python
deploy_suffix = uuid.uuid4().hex[:4]
DEPLOYMENT_CONFIG_NAME = f"private-hf-lora-cfg-{deploy_suffix}"
DEPLOYMENT_NAME = f"private-hf-lora-{deploy_suffix}"

deployment_config = client.inference.deployment_configs.create(
    workspace="default",
    name=DEPLOYMENT_CONFIG_NAME,
    engine="vllm",
    model_spec={
        "model_namespace": "default",
        "model_name": MODEL_NAME,
        "lora_enabled": True,
    },
    executor_config={
        "gpu": 1,
        "image_name": "vllm/vllm-openai",
        "image_tag": "v0.22.1",
        "additional_args": ["--max-lora-rank", "32"],
    },
)

deployment = client.inference.deployments.create(
    workspace="default",
    name=DEPLOYMENT_NAME,
    config=deployment_config.name,
)

print(f"Deployment name: {deployment.name}")
print(f"Deployment status: {deployment.status}")
```

```python
time_check = max_wait_time_checker(1800, "Deployment")
while True:
    time_check()
    time.sleep(15)
    deployment_status = client.inference.deployments.retrieve(
        name=deployment.name,
        workspace="default",
    )
    clear_output(wait=True)
    print(f"Deployment: {deployment.name}")
    print(f"Status: {deployment_status.status}")

    if deployment_status.status in ("RUNNING", "READY"):
        if not client.models.wait_for_gateway(deployment.name, workspace="default", timeout=60):
            raise RuntimeError("Inference gateway did not become ready")
        break
    if deployment_status.status in ("FAILED", "ERROR", "TERMINATED", "LOST"):
        raise RuntimeError(f"Deployment failed with status: {deployment_status.status}")
```

### 8. Test the Deployed Model

Call the inference gateway through the SDK. Use the base model name to test the original model and the output adapter name to test the LoRA-adapted model.

<Tabs>

<Tab title="Python SDK">

```python
messages = [
    {"role": "user", "content": "Can you summarize what LoRA fine-tuning does?"}
]


def chat(model_id: str):
    return client.inference.gateway.provider.post(
        "v1/chat/completions",
        name=deployment.name,
        workspace="default",
        body={
            "model": model_id,
            "messages": messages,
            "temperature": 0.2,
            "max_tokens": 128,
        },
    )


BASE_INFERENCE_MODEL_NAME = f"default/{MODEL_NAME}"
INFERENCE_MODEL_NAME = f"default--{OUTPUT_NAME}"
base_response = chat(BASE_INFERENCE_MODEL_NAME)
lora_response = chat(INFERENCE_MODEL_NAME)

print("Base model response:")
print(base_response["choices"][0]["message"]["content"])
print("\nLoRA-adapted model response:")
print(lora_response["choices"][0]["message"]["content"])
```

</Tab>

<Tab title="cURL">

```bash
export OUTPUT_NAME="<output adapter name from the Python job block>"
export INFERENCE_MODEL_NAME="default--${OUTPUT_NAME}"

curl -s "${NMP_BASE_URL}/apis/inference-gateway/v2/workspaces/default/openai/-/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "'${INFERENCE_MODEL_NAME}'",
    "messages": [
      {"role": "user", "content": "Can you summarize what LoRA fine-tuning does?"}
    ],
    "temperature": 0.2,
    "max_tokens": 128
  }' | jq
```

</Tab>

</Tabs>

## Next Steps

Learn how to [check customization job metrics](/documentation/customizer-reference/tutorials/metrics) to monitor training progress and performance for your fine-tuned model.
