# Sensor Validation Framework Documentation

This framework is used to run sensor tests, that is, run *experiments* and validate the results via key performance indicators (*KPI*s).
Experiments, as defined in validation and Evolve terminology, are simulations that contain sensors and typically generate sensor data.
KPIs, on the other hand, are metrics that are computed from said sensor data.

Experiments can also be leveraged to run premerge validation tests.

## Add New Experiment
1. Create experiment Python file in `sensors/experiments/cam/colorchecker/`
2. Basic structure (`cam_colorchecker_exp1.py` example):
```python
from evolve.api import ExperimentAPI

class CamColorcheckerExp1(ExperimentAPI):  # Must inherit from ExperimentAPI
    ...
```

## Add New KPI
1. Create KPI Python file in `sensors/kpis/` (e.g. `camera_colorchecker_kpi.py`)
2. Basic structure (must inherit from base KPI class):
```python
from evolve.api import KPI

class CameraColorcheckerKpi(KPI):  # Must inherit from BaseKPI
    ...
```

## Test Configuration
Add to test array in `tests.json`:
```json
[
    {
        "experiments": ["CamColorcheckerExp1"],
        "kpis": ["CameraColorcheckerKpi"]
    },
    {
        "experiments": ["ExpABC,ExpDEF"],
        "kpis": ["KPIXY, KPIZA"]
    }
]
```
Multiple experiments and KPIs can be added to the same test.

## Regenerate Poetry Dependencies
When the dependencies in the `pyproject.toml` files in the various test folders have been modified, run the following command to regenerate the dependencies:
```bash
cd kit
source/python/sensors/tests/validation/run.sh lock
```

## Running Tests
Use `run.sh` as follows:
```bash
# Run all tests defined in tests.json
./run.sh

# Run specific tests from tests.json with matching pattern
./run.sh --experiment-pattern Cam.* --kpi-pattern Camera.*

```

## Run Premerge Tests
```bash
cd kit
./repo ci ext-tests-sensors
```

## Clean the peotry env

```bash
cd kit
source/python/sensors/tests/validation/run.sh clean
```
