<a id="nemo-ms-deploy-infrastructure-core-jobs"></a>
# Manage Jobs

This section describes how to configure jobs in {{platform_name}}. The Jobs service is responsible for scheduling batch jobs, collecting telemetry and managing job results.

## Execution Profiles

You can configure jobs to run on different hardware by specifying an execution profile. This allows you to define different compute resources and job execution backends for different job types and use cases.

An execution profile is defined by the following attributes:

- A profile name (`profile`)
- A compute provider (e.g. `cpu` or `gpu`)
- An [execution backend](#execution-backends) (e.g. `docker`, `kubernetes_job`, `volcano_job`)

By default, the {{platform_name}} defines a default CPU and default GPU provider to launch CPU and GPU bound jobs.

You can configure multiple execution profiles to suit the shape of your compute environment. For example, if you have a compute environment with heterogeneous infrastructure (e.g., two types of GPU hardware such as A100 and H200), you can define a list of execution profiles as follows:

```yaml
jobs:
 executors:
 # Allow any CPU-bound job to run anywhere
 - provider: cpu
 profile: default
 backend: kubernetes_job
 config: {...}
 
 # Run on A100 hardware
 - provider: gpu
 profile: a100-pool
 backend: kubernetes_job
 config:
 # use the appropriate pod scheduling for a100-pool
 node_selector:
 node-pool-name: a100-pool

 # Run on H200 hardware
 - provider: gpu
 profile: h200-pool
 backend: kubernetes_job
 config:
 # use the appropriate pod scheduling for h200-pool
 node_selector:
 node-pool-name: h200-pool
```

For full configuration details, see the [platform configuration reference](config-reference.md).

### Default Execution Profiles

The {{platform_name}} defines a default execution profile for each execution backend depending on the platform's control plane. The default execution profile is used when no specific execution profile is specified for a job.

- Default CPU Execution Profile (`cpu`): The default execution profile for CPU based jobs.
- Default GPU Execution Profile (`gpu`): The default execution profile for GPU based jobs.

You may configure the default execution profiles by updating the `executor_defaults` section of the `jobs` section of the [platform configuration](config-reference.md). The structure of the `executor_defaults` matches the configuration of the execution backend configuration.

```yaml
jobs:
 executor_defaults:
 # Override the default Docker execution profile configuration
 docker:
 storage:
 volume_name: nemo-jobs-storage

 # Override the default Kubernetes execution profile configuration
 kubernetes_job:
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 effect: NoSchedule
 node_selector:
 kubernetes.io/arch: amd64

 # Override the default Volcano execution profile configuration
 volcano_job:
 storage:
 pvc_name: nemo-jobs-storage
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 effect: NoSchedule
 node_selector:
 kubernetes.io/arch: amd64
```

## Execution Backends

Execution backends are the containerized job execution systems that {{platform_name}} jobs are scheduled to run on. Each execution backend is responsible for launching and managing the job containers, and the Jobs service communicates with the execution backend to schedule and manage jobs.

{{platform_name}} currently supports the following execution backends:

- Docker (`docker`)
- Kubernetes Jobs (`kubernetes_job`)
- Volcano Jobs (`volcano_job`)

### Docker

The {{platform_name}} supports Docker as an execution backend for CPU and GPU based jobs. When the shared GPU pool is configured (see below), Docker will use only the configured GPU devices and will not over-schedule jobs if there are not currently enough GPUs available.

**Note:** The {{platform_name}} supports Docker-based job execution by default when running the {{platform_name}} in quickstart mode, which requires no configuration.

!!! tip "If you are running GPU jobs with Docker, see [GPU Configuration](../get-started/setup.md) for information on configuring the shared GPU pool. This configuration is shared between the jobs and models services to prevent GPU resource conflicts."

```yaml
jobs:
 executors:
 # Define the default CPU provider
 - provider: cpu
 profile: default
 backend: docker
 config:
 storage:
 volume_name: nemo-platform_jobs_storage

 # Define the default GPU provider
 - provider: gpu
 profile: default
 backend: docker
 config:
 storage:
 volume_name: nemo-microservices_jobs_storage
```

### Kubernetes Jobs

The {{platform_name}} supports Kubernetes Jobs as an execution backend for CPU and GPU based jobs.

```yaml
jobs:
 executors:
 # Define the default CPU provider
 - profile: default
 backend: kubernetes_job
 provider: cpu
 config:
 # Storage is the kubernetes_job storage configuration
 storage:
 # Define the name of a persistent volume claim that will be used by launched jobs
 pvc_name: nemo-core-jobs-storage

 # Define the default GPU provider
 - profile: default
 backend: kubernetes_job
 provider: gpu
 config:
 storage:
 pvc_name: nemo-core-jobs-storage
 # You can configure custom labels and annotations on Kubernetes Jobs and their pods. This may be useful within environments that require adding integration with service meshes or similar cluster-level integrations.
 job_metadata:
 labels:
 my-custom-label: "value"
 annotations:
 example.com/annotation: "value"
 pod_metadata:
 labels:
 sidecar.istio.io/inject: "false"
 annotations:
 example.com/annotation: "value"
 # You can configure typical Kubernetes pod scheduling behavior via node selectors, tolerations, and node/pod affinities.
 node_selector:
 kubernetes.io/arch: amd64
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 effect: NoSchedule
 affinity:
 nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 nodeSelectorTerms:
 - matchExpressions:
 - key: highmem
 operator: In
 values:
 - "true"
```

#### KAI Scheduler

[KAI Scheduler](https://github.com/NVIDIA/KAI-Scheduler) is a Kubernetes scheduler for AI/ML workloads. You can direct Kubernetes Jobs to use KAI Scheduler by setting `scheduler_name` in the execution profile config, and by adding the required queue labels to the job and pod metadata.

```yaml
jobs:
 executors:
 - profile: default
 backend: kubernetes_job
 provider: gpu
 config:
 # Direct pods to the KAI scheduler
 scheduler_name: kai-scheduler
 # Required labels for KAI Scheduler queue assignment
 job_metadata:
 labels:
 kai.scheduler/queue-name: default
 pod_metadata:
 labels:
 kai.scheduler/queue-name: default
 storage:
 pvc_name: nemo-core-jobs-storage
```

See the [KAI Scheduler documentation](https://github.com/kai-scheduler/KAI-Scheduler/tree/main/docs) for more information on how to configure jobs with KAI Scheduler.

### Volcano Jobs

The {{platform_name}} supports Volcano Jobs as an execution backend for launching distributed GPU jobs.

Volcano Jobs are configured using the same configuration as Kubernetes Jobs, but with the following additional configuration:

- `queue`: The Volcano queue to submit the job to.
- `scheduler_name`: The Volcano scheduler to use for the job.
- `plugins`: The Volcano plugins to use for the job.
- `max_retry`: The maximum number of retries for the job.
- `enable_multi_node_networking`: Enable multi-node networking injection. Sets annotations to trigger Kyverno policy mutations. This is only available if the platform is configured to use multi-node networking (see [Multi-Node Networking](helm/multinode-networking.md)).

```yaml
jobs:
 executors:
 - profile: default
 backend: volcano_job
 provider: gpu_distributed
 config:
 queue: default
 scheduler_name: volcano
 plugins:
 pytorch: ["--master=leader", "--worker=worker", "--port=23456"]
 max_retry: 0
 # Additional configuration for the Volcano Job, shared with Kubernetes Jobs
 ...
```
