[[_TOC_]]

# Gitlab CI/CD pipeline

This document explains the basics of the Gitlab CI/CD pipeline for Kit and Rendering.


## Introduction

Kit SDK uses a [dynamic child pipeline](https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html#dynamic-child-pipelines).
This is needed to read the MR description and schedule the appropriate stages and jobs based on dev choices.
It is also essential for kit and rendering to coexist since they both contain their own pipeline and toolchain and run independently as 2 projects merged into one, see the [Kit and Rendering](#kit-and-rendering) section below for more details.

CI files structure:
- [.gitlab-ci.yml](/.gitlab-ci.yml) -> contains a few checks and jobs to generate the full pipeline
- [.gitlab-ci-base.yml](/.gitlab-ci-base.yml) -> base yml file, will be modified into .gitlab-ci-generated.yml
- [.gitlab-ci-common.yml](/.gitlab-ci-common.yml) -> common include, variables, hidden jobs, etc
- [.gitlab-ci-generated.yml](/.gitlab-ci-generated.yml) -> generated yml file that contains the full pipeline

Kit CI files are located here:
- [stage-kit-build.yml](/kit/tools/ci/gitlab/stage-kit-build.yml) -> build jobs for kit
- [stage-kit-test-unit.yml](/kit/tools/ci/gitlab/stage-kit-test-unit.yml) -> tests are spread in multiple files
- ... and more located in [/kit/tools/ci/gitlab/](/kit/tools/ci/gitlab/)

Rendering CI files are located here:
- [stage-rtx-build.yml](/rendering/tools/ci/gitlab/stage-rtx-build.yml) -> build jobs for rendering
- [stage-rtx-test.yml](/rendering/tools/ci/gitlab/stage-rtx-test.yml) -> test jobs for rendering
- ... and more located in [/rendering/tools/ci/gitlab/](/rendering/tools/ci/gitlab/)

Other yml files are included, either locally or externally, see include sections from the yml files.


## Kit and rendering

Before going into the details of the CI/CD generation it's important to explain the kit and rendering relationship. Historically kit and rendering were 2 separate projects but ended up being merged together after a while (probably because of constant breakages on each sides). This is why the root of kit has two main folders, [kit](/kit/) and [rendering](/rendering/). They both kept most of their toolchain and dependencies, making the repo more complicated than a typical project.

The rendering subtree runs only when changes to rendering/* files are found. In that case we run the full pipeline, meaning it will build rendering, test it and publish it. After rendering publishes then the Kit pipeline starts by checking that rendering was published successfully, if so it pulls it and starts building / testing / deploying Kit. The script responsible for checking the rendering package (also known as `rtx_plugins`) is [rtx_plugins_lookup.py](/kit/tools/buildscripts/rtx_plugins_lookup.py).

There is one special case that happens sometimes. Consider 2 commits pushed to default branch one after the other. If the first one contains rendering changes it will have to rebuild the `rtx_plugins` but this takes time. If the 2nd commit only modified Kit files it will do a lookup and see that `rtx_plugins` is missing, in this case there is a special path taken by the CI/CD pipeline to only build / test / publish `rtx_plugins` without running the full rendering test suite. In `repo_generate_ci` this is handled by having 2 different triggers, one called `trigger_rendering` that is set to true when rendering changes are detected and another one called `trigger_rtx_plugins` that is set to true when rendering changes are not detected but the `rtx_plugins` are missing and need to be built. Read more below for the trigger system.


## CI/CD Generation

To generate the child pipeline (the one building/testing/deploying Kit and Rendering) we use a tool called `repo_generate_ci`.
It's registered in [repo_tools.toml](kit/tools/repoman/internal/repo_tools.toml), its implementation is in [generate_ci.py](/kit/tools/repoman/internal/generate_ci.py) and its settings are in [repo.toml](/kit/repo.toml).


The file [.gitlab-ci.yml](/.gitlab-ci.yml) is the entry point of the CI/CD for Gitlab. It contains very few jobs because most of the Kit pipeline logic is within the child pipeline. To generate the pipeline there is a job called `generate-pipeline` which will call `repo.sh generate_ci`, this will run the python script [generate_ci.py](/kit/tools/repoman/internal/generate_ci.py) that will take care of that work.

```yml
generate-pipeline:
  artifacts:
    paths:
      - .gitlab-ci-generated.yml
  script:
    - ./kit/repo.sh generate_ci
```

The [generate_ci.py](/kit/tools/repoman/internal/generate_ci.py) script will do a few things:
* Read the MR (merge request) description (checkbox system)
* Read the list of modified files
* Read an existing yml file called [.gitlab-ci-base.yml](/.gitlab-ci-base.yml) and transform it into [.gitlab-ci-generated.yml](/.gitlab-ci-generated.yml)
* Apply rules and patterns defined under `[repo_generate_ci]` in [repo.toml](/kit/repo.toml)
* Call `omni.repo.kit_tools_internal.kit_gitlab` located [here](https://gitlab-master.nvidia.com/omniverse/repo/repo_kit_tools_internal/-/blob/main/omni/repo/kit_tools_internal/kit_gitlab.py) and apply some additional changes.


Here's a few details on each of these behaviors.

### MR description and checkbox system

Kit MR default template contains a section with optional checkboxes, for example:

```
# Kit
# > - [x] Build
# > - [x] Unit testing
# > - [ ] Integration testing
```

We have a regex called `mr_checkboxes_regex` that will read all these options and transform them into an environment variable according to rules defined in [repo.toml](/kit/repo.toml), for example :

```toml
trigger_build = { name="Build", env="_MR_TRIGGER_BUILD", checked=true }
trigger_unit_test = { name="Unit testing", env="_MR_TRIGGER_UNIT_TEST", checked=true }
trigger_integration_test = { name="Integration testing", env="_MR_TRIGGER_INTEGRATION_TEST", checked=false }
```

If we take the key `trigger_unit_test` as an example, it will look for a checkbox named `Unit Testing` and will set the environment variable `_MR_TRIGGER_UNIT_TEST` with the value of the checkbox (true or false). The `checked` value represents the initial state of the checkbox (default value).

The [generate_ci.py](/kit/tools/repoman/internal/generate_ci.py) script will then set those ENV vars so that all pipeline jobs can use them, mainly for defining Gitlab CI rules (see https://docs.gitlab.com/ee/ci/yaml/#rules).

#### To add a new checkbox:

1. Add a new line under `[repo_generate_ci.checkboxes]` in [repo.toml](/kit/repo.toml), example :
    ```toml
    skip_all_tests = { name="Skip All Tests", env="MR_SKIP_ALL_TESTS", checked=false }
    ```
2. In that example the key is `skip_all_tests`, `name` represents the string that will be detected in the merge request and `env` is the name of the env variable that will be set.
3. Add a new variable in [.gitlab-ci-base.yml](/.gitlab-ci-base.yml) like this :
    ```yml
    MR_SKIP_ALL_TESTS:
        value: "%%MR_SKIP_ALL_TESTS"
        description: "Will be Set to true if the checkbox '[x] Skip All Tests' is selected in the MR description, false otherwise"
    ```
4. Use that value in a rule somewhere - or any other usage (could be used in a script too).
    ```yml
    rules:
      - if: $MR_SKIP_ALL_TESTS == "true"
    ```
5. Note that the environment value set is a string, either `"true"` or `"false"` (lowercase).


### List of modified files

`repo_generate_ci` will read the list of modified files and apply rules based on file patterns, for example :

```toml
[repo_generate_ci.file_patterns]
trigger_rendering = { match_rule="any", patterns=["rendering/*", "runtime/*", "__overflow__"] }
```

Here the key `trigger_rendering` is set to match if `any` file under either `rendering/` or `runtime/` is changed. If a modified file is found under those folders, the pipeline will build / test / deploy the rendering subtree, otherwise it will skip it and only build Kit.

### Modifying .gitlab-ci-base.yml

`repo_generate_ci` will modify a file called [.gitlab-ci-base.yml](/.gitlab-ci-base.yml) and transform it into [.gitlab-ci-generated.yml](/.gitlab-ci-generated.yml). If you open [.gitlab-ci-base.yml](/.gitlab-ci-base.yml) you'll notice lots of variables starting with `%%`, those are the values that will be modified during the pipeline generation, for example :

```yml
  _MR_TRIGGER_BUILD:
    value: "%%_MR_TRIGGER_BUILD"
    description: "Will be Set to true if the checkbox '[x] Build' is selected in the MR description, false otherwise"
```

Here the actual value of the checkbox `[x] Build` will be set into `_MR_TRIGGER_BUILD` once the generation is done and saved into a new file called [.gitlab-ci-generated.yml](/.gitlab-ci-generated.yml). This file will be the entrypoint yml file of our child pipeline.

This system has the benefit of being explicit and easy to understand since we only slightly modify an existing file. It also allows to do things that would not be possible otherwise, for example dynamically change the `needs` of a given job, this is what happens for `.kit-rtx-plugins-lookup-needs`.

### Applying rules

`repo_generate_ci` will apply rules based on the Gitlab pipeline source. For example a commit can be in a push (post merge) or a merge request or a scheduled pipeline, etc. This allow the flexibility of running different jobs given the context, typically merge requests have more optional jobs than post merge. Such rules are called `[repo_generate_ci.push_rules]` or `[repo_generate_ci.schedule_rules]` in [repo.toml](/kit/repo.toml).

### repo_kit_tools

`repo_generate_ci` calls an external tool `omni.repo.kit_tools_internal.kit_gitlab` located [here](https://gitlab-master.nvidia.com/omniverse/repo/repo_kit_tools_internal/-/blob/main/omni/repo/kit_tools_internal/kit_gitlab.py). For historical reason it's located outside the Kit repo but only used by Kit so it could move into Kit eventually. It currently apply tags to jobs and process `[prefixes]` rules from [repo.toml](/kit/repo.toml).


## Runners

### Horde

There are a set of linux horde VMs with L40 GPU that are used by the following jobs:

* core-api-docker-compose-test

Current runners are:

* aljohns-sensor-rtx-dev-1 (10.63.180.169)
* aljohns-sensor-rtx-dev-2 (10.63.181.189)

To use these runners:

```
my-job:
  tags:
    - sensor-rtx-dev-horde
```

#### Setting up a new runner

First create the horde instance at https://horde.nvidia.com/:
* os = Linux
* OS version = Ubuntu 22.04
* Cpu fam = x86_64
* GPU = L40
* Purpose = Service (IMPORTANT)
* Set instance name to `sensor-rtx-dev-N` (where N is incrementing)

Then follow instructions at https://confluence.nvidia.com/pages/viewpage.action?pageId=2006217907#GitlabCI/CDRunner-4.3SettingupaLinuxHORDEinstanceas'shell'runnertype

During setup on the `New project runner` gitlab page, make sure to:
* set tag to `sensor-rtx-dev-horde`
* set job timeout to 3600 seconds

Then [setup docker for rtx rendering](../../../../tests/coreapi/README.md#rendering-setup-for-docker).

Here is a bash snippet you can run as sudo to perform a number of the steps above at once:
```
# create gitlab-runner user
useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
rm -f /home/gitlab-runner/.bash_logout
usermod -aG docker gitlab-runner

# setup docker for rtx
apt-get update
apt-get install -y nvidia-container-toolkit
nvidia-ctk --version  # should be 1.16 or higher
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker

# install and start gitlab runner
curl -L --output /usr/local/bin/gitlab-runner "https://s3.dualstack.us-east-1.amazonaws.com/gitlab-runner-downloads/latest/binaries/gitlab-runner-linux-amd64"
chmod +x /usr/local/bin/gitlab-runner
gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
gitlab-runner start
```

Then (as gitlab-runner user) install poetry:
```
pip3 install poetry --upgrade
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
. ~/.bashrc
poetry --version  # check it
```

Then, update `/etc/gitlab-runner/config.toml` and add:
```
[[runners]]
  pre_build_script = "bash ${CI_PROJECT_DIR}/.horde-pre_build.sh || /bin/true"
```

Then (as root):
```
systemctl daemon-reload
systemctl restart gitlab-runner
```

Note that after you install the pre-build hook, you can see when jobs have run on that host (when
you're on the runner):

```
]$ cat /home/gitlab-runner/runs.txt
Fri Aug  2 04:04:16 UTC 2024 --- 17137233 --- core-api-docker-compose-test --- omniverse/kit
Fri Aug  2 04:59:00 UTC 2024 --- 17138215 --- core-api-docker-compose-test --- omniverse/kit
```


## Additional details

### Linux only pipeline

The pipeline to run on linux only is less intuitive and deserves its own section.
There is a rule called `.rules-linux-only` that is set on most jobs, here's its definition :

```yml
# Special rule to run CI/CD on linux only with a MR checkbox
# It uses the job_name since the env CI_RUNNER_EXECUTABLE_ARCH is only available at the job level, not at the pipeline level
# For jobs that don't have `linux-x86_64` in their name set the env `ADD_TO_LINUX_ONLY` instead.
.rules-linux-only:
  rules:
    - if: $_MR_LINUX_ONLY == "true" && $CI_JOB_NAME !~ /.*linux-x86_64.*/ && $ADD_TO_LINUX_ONLY == null
      when: never
```

The rule will prevent all jobs that don't have linux-x86_64 in their name from running.

The point to highlight is that it can be a little brittle when devs add new jobs, especially external jobs.
If the job uses needs and depends on both `kit-build-release-linux-x86_64` and `kit-build-release-windows-x86_64` then it's very important to set the windows artifacts as optional.
Failure to do so will break the pipeline when someone clicks on the `[x] Linux Only checkbox`. Here's an example that shows how it has to be done:

```yml
job-example:
  needs:
    - job: kit-build-release-linux-x86_64
      optional: false
      artifacts: true
    - job: kit-build-release-windows-x86_64
      optional: true
      artifacts: true
  rules:
    - !reference [.rules-linux-only, rules]
```

Notice the `optional` difference between windows and linux. This is because the linux only build will not have a  `kit-build-release-windows-x86_64` job (or any windows job). It's very important to follow this construct otherwise the linux only build will break with an invalid yml error.
