# Camera Intrinsic Calibration Tools

This directory contains a comprehensive suite of tools for camera intrinsic calibration, simulation, and validation using Omniverse Kit sensor simulation. The tools have been refactored around a central library architecture with support for both standard pinhole and fisheye camera models.

## Overview

The tooling provides an end-to-end workflow for camera calibration:
1. **Data Capture**: Generate synthetic calibration images through realistic ISP pipeline simulation
2. **Calibration**: Extract camera intrinsic parameters using OpenCV algorithms
3. **Simulation Validation**: Verify calibration accuracy by comparing real vs simulated images in Omniverse

## Architecture

### Tool Library
- **`kit/source/python/sensors/tools/cam/calibration/calibration_run.py`** - Standalone OpenCV calibration
- **`kit/source/python/sensors/tools/cam/calibration/calibration_sim_runner.py`** - General script for simulation inside a Kit instance
- **`kit/source/python/sensors/tools/cam/calibration/calibration_sim_validation.py`** - Calibration validation via simulation inside a Kit instance
- **`kit/source/python/sensors/tools/cam/calibration/calibration.py`** - Core library containing calibration algorithms and simulation tools

### Experiments
- **`kit/source/python/sensors/experiments/cam/cam_calibration_exps.py`** - General calibration experiment classes
- **`kit/source/python/sensors/experiments/cam/cam_calibration_real2sim_exp.py`** - Real2sim experiment for CI/CD
- **`kit/source/python/sensors/experiments/cam/cam_raw_isp_capture_exp.py`** - Raw ISP capture experiment

### KPIs
- **`kit/source/python/sensors/kpis/cam/cam_calibration_kpis.py`** - General calibration KPI classes
- **`kit/source/python/sensors/kpis/cam/cam_calibration_real2sim_kpi.py`** Real2sim KPI for CI/CD

### Other
- **`kit/source/python/sensors/tools/cam/calibration/camera_raw_isp_capture.py`** - ISP pipeline batch capture tool
- **`kit/source/python/sensors/tools/cam/calibration/digital_twin_manipulator.py`** - Sensor pose & light manipulator widget (can be launched via `digital_twin_manipulator_exp.py`)
- **`kit/source/python/sensors/tools/cam/calibration/digital_twin_test_sequence.py`** - Digital twin motion sequence simulation (can be launched via `digital_twin_test_sequence_exp.py`)

## Quick Start

### Local Google Drive Mount

Add these env vars to your bashrc or source them when you want to mount the drive:
```bash
export OMNI_SENSORS_VALIDATION_GDRIVE_TOKEN='{"token":"ya29.a0AeDClZD6lfLTuQcWYEeabJGBckMNsTDOTQIhSn7ZxRAE1fHH5AxiCwqqFOsu5uyB8uOe_0eVgukLKOT3hPa0XE6frWNHNZkFfJ5CSptTbx6g20aF4f4UqKjQsaPZ3qOj-4KSWVBZvz5M1VyjPxf-fValTyD_zivScyMwqvYSaCgYKARYSARISFQHGX2Mi7C5tawI3bh1x-P7ql0yn1A0175","token_type":"Bearer","refresh_token":"1//03-JJoQR__wJnCgYIARAAGAMSNwF-L9Ir3KHPVIJZbbmGMyqvy9x4yMvwmRK95a4KxETp0lgIQEc0c5RT8oV_WdfA8DoM4ICtdwQ","expiry":"2024-11-18T16:28:49.970577127+01:00"}'
export OMNI_SENSORS_VALIDATION_GDRIVE_UPLOAD_CLIENT_TOKEN='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'
export OMNI_SENSORS_VALIDATION_GDRIVE_ENABLE_CACHE=1
```

Mount share:
```bash
python3 kit/source/python/sensors/pytest/test_support/data_source.py
```

If you need a new token, fetch it:
```bash
rclone config reconnect sensors-validation-gdrive: --config ./_testoutput/mounts/sensors-validation-gdrive_rclone.conf
```

### Recommended: Using run.sh for Dependency Management

To ensure correct dependencies are installed and available, use the `run.sh` script from the validation tests directory instead of running Python directly.

If a local Python environment is supposed to be used, install evolve PyPi:

API only (enough to run these tools): 
```bash
pip install --index-url https://urm.nvidia.com/artifactory/api/pypi/ct-omniverse-pypi/simple evolve[api]
```

To install all features, use `evolve[all]` instead, or choose specific features as per the evolve README.

### Basic Calibration

Make sure environment variable `SENSORS_REPO` is defined and contains the absolute path to the sensors repository.

```bash
cd kit/source/python/sensors/tests/validation
./run.sh ../../tools/cam/calibration/calibration_run.py \
    --input_images_dir ${SENSORS_REPO}/kit/source/python/sensors/tools/sensor_lib/cam/calibration_data/real \
    --rows 15 \
    --cols 21 \
    --square_size 5.9 \
    --is_fisheye
```

### Simulation Validation

```bash
cd kit/source/python/sensors/tests/validation
./run.sh ../../tools/cam/calibration/calibration_sim_validation.py \
    --pattern_size 21 15 \
    --square_size 5.9 \
    --is_fisheye \
    --output ./_results
```
Note: Add `--input_images_dir ./calibration_images` to define a set of custom input images.

The following diagram illustrates the execution flow for calibration_sim_validation.py:

```
calibration_sim_validation [argparse]
            |
            v
┌─────────────────────────────────┐
│  OpenCVCalibrationSimRunnerExp  │  [params]
│                                 │
│  SensorSimulation()             │
└─────────────────────────────────┘
            |
            | run
            v
calibration_sim_runner [argparse]
            |
            v
┌────────────────────────┐
│  OpenCVCalibrationExp  │  [params]
│                        │
│  CameraSimulation()    │
└────────────────────────┘
            |
            | run
            v
    sim, capture, etc.
```

### ISP Pipeline Capture

```bash
cd kit/source/python/sensors/tests/validation
./run.sh --experiment-pattern CamRawIspCaptureExp --kpi-pattern none
```

### CI/CD Experiments & KPIs

Pytest:
```bash
./kit/repo.sh test --suite sensors -e "-m unstable" -e "-k test_cam_intrinsic_calibration"
```

Experiment with predefined arguments:
```bash
./run.sh --experiment-pattern CamCalibrationReal2SimExp --kpi-pattern CamCalibrationReal2SimKpi
```

### Digital Twin Manipulator

Implemented as a script that creates a Kit widget, the manipulator allows the user to look at the digital twin scene and browse the different poses available for a specific sensor.
When a new sensor pose is set, the manipulator also turns the corresponding target light on.
The widget also allows the user to set a Perspective camera and to visualize all the poses at once.

The script tha controls the widget is `digital_twin_manipulator.py`.
At the top of the script, the motion data (pose) source and camera model can be selected by modifying
```bash
MOTION_DATA_FUNCTION = "Camera_Yi_poses"  # Motion data sets available in ``sensor_lib`` and ``digital_twin_setup.py``
CAMERA_MODEL = "Camera_Yi"  # Camera assets available in ``sensor_lib``
```

To run the manipulator in a simple way that takes care of the dependencies, run the following experiment:
```bash
cd kit/source/python/sensors/tests/validation
./run.sh --experiment-pattern DigitalTwinManipulatorExp --kpi-pattern none
```

### Digital Twin Test Sequence

This tool simulates camera tester sequences in a digital twin environment for camera intrinsic calibration.
It creates a virtual camera testing scenario with controlled motion sequences, lighting conditions, and 
calibration targets.

The script provides an automated camera calibration sequence simulation that:
- Loads the camera tester scene from Omniverse
- Creates a custom perspective camera
- Executes predefined motion sequences (but does not capture images or do calibration)
- Controls the sequence of target lights

| Motion Data Sources | Description | Use Case |
|--------|-------------|----------|
| `tester_log` | Real tester log data (default) | SIGGRAPH demo |
| `Camera_Valeo_Fov120_cal_rational_poses` | Predefined calibration target positions for Valeo camera | Standard calibration workflows |
| `Camera_Yi_poses` | Predefined calibration target positions for Yi camera | Standard calibration workflows |
| `Camera_Yi_poses_debug` | Predefined *reference* calibration target positions for Yi camera | Standard calibration workflows |

| Command Line Arguments | Type | Default | Description |
|----------|------|---------|-------------|
| `--motion_data` | str | `tester_log` | Motion data source |

To run the general case:
```bash
cd kit/source/python/sensors/tests/validation
./run.sh ../../tools/cam/calibration/digital_twin_test_sequence.py --motion_data tester_log
```

To run it as an experiment with predefined arguments:
```bash
cd kit/source/python/sensors/tests/validation
./run.sh --experiment-pattern DigitalTwinTestSequenceExp --kpi-pattern none
```

## Command Line Arguments

### Common Parameters

| Argument | Type | Default | Description |
|----------|------|---------|-------------|
| `--input_images_dir` | str | - | Directory containing calibration images |
| `--output` | str | `kit/_testoutput/opencv_calibration_simulation_validation` | Output directory |
| `--pattern_size` | int int | `21 15` | Checkerboard pattern size (cols, rows) |
| `--square_size` | float | `5.9` | Physical size of checkerboard squares (mm) |
| `--is_fisheye` | flag | - | Use fisheye camera model (flag, no value needed) |
| `--nit_level` | float | `1000` | Target brightness level (nits) |
| `--pixel_pitch_mm` | float | `0.003` | Pixel physical size (mm) |
| `--nodal_offset_z` | float | `0.5` | Z offset to nodal point (cm) |

### Simulation Parameters

| Argument | Type | Default | Description |
|----------|------|---------|-------------|
| `--use_simulated_images` | bool | `false` | Use simulated images for calibration |
| `--dump_metrics_json` | bool | `true` | Export metrics to JSON |
| `--enable_visualization` | bool | `true` | Generate comparison visualizations |
| `--usd_file_path` | str | CameraTester scene | USD stage file path |
| `--kit_args` | str | `""` | Additional Kit application flags |

## Camera Models

#### Fisheye Model (Default)
- **Distortion Parameters**: 4 coefficients (k1, k2, k3, k4)
- **Projection**: Equidistant fisheye model

#### Pinhole Model
- **Distortion Parameters**: 5+ coefficients (k1, k2, p1, p2, k3, ...)
- **Projection**: Standard perspective projection

## Output Files

### Calibration Results
- **`calibration_params.json`** - Camera parameters, poses, and metrics
- **`lens_distortion_simulator_output.json`** - Simulation output data

### Validation Visualizations
- **`comparisons/`** - Overlay images showing real vs simulated corners
- **`simulated/`** - Raw simulated images for manual inspection

### File Naming Convention
```
overlay_[image_name].png          # Comparison visualization
simulated_[image_name].png        # Simulated image
```

### ISP Pipeline Stages (Legacy Capture)

| Stage | Description | Output Format |
|-------|-------------|---------------|
| **HDR** | Raw sensor data | 16-bit float |
| **Color Correction** | Color matrix + white balance | 16-bit float |
| **CFA** | Bayer pattern encoding | 32-bit RGGB |
| **Noise** | Shot noise + dark current | 32-bit |
| **Companding** | Tone curve application | 16-bit |
| **ISP** | ISP model processing | YUV422 |
| **RGB** | Final processed output | 8-bit RGB |
