Skip to content

testing.md

Latest commit

 

History

History
252 lines (200 loc) · 16 KB

testing.md

File metadata and controls

252 lines (200 loc) · 16 KB
Raw

Testing the Alp SDK

How to verify everything works, from a fresh clone, without touching hardware. Coverage map per <alp/...> header below.

What this doc is. An index of which tests exist for which surfaces and how to run them locally.

What this doc is not. The verification ledger. Per-feature pass/fail status (⏳ untested / 🟡 partial / ✅ verified / ❌ failing) lives in docs/test-plan.md. Real-hardware HIL contract lives in docs/ci/HW-IN-LOOP.md. Per-area ZTEST counts + identified coverage gaps live in docs/test-coverage-audit.md.

Quick start (from a fresh clone)

git clone https://github.com/alplabai/alp-sdk
cd alp-sdk

# One-time setup: Zephyr workspace + Python deps + apt hints.
bash scripts/bootstrap.sh

# Make Zephyr reachable for builds.  For native_sim/native/64 the
# host gcc is used; the Zephyr SDK is only needed for cross-compiled
# real-silicon scenarios (E1M-AEN / V2N-M33 boards).
export ZEPHYR_BASE="$PWD/../zephyr"
export ZEPHYR_TOOLCHAIN_VARIANT=host   # use `zephyr` only when ZEPHYR_SDK_INSTALL_DIR is set
                                        # for the *.aen / *.v2n-m33 scenarios

# Run every locally-runnable test (no HIL).
bash scripts/test-all.sh

A green run proves:

  • Every <alp/...> public function compiles + links across the Yocto and baremetal plain-CMake backends.
  • Every Zephyr-side wrapper passes its ZTEST failure-path suite under native_sim/native/64.
  • Every Yocto-side wrapper passes its ctest failure-path suite.
  • Every cryptographic primitive in <alp/security.h> passes its KAT + AEAD round-trip on OpenSSL.
  • Every shipped example app builds + runs under twister.
  • clang-format is clean on the diff.
  • board.yaml metadata validates against its schema.
  • Doxygen builds with zero warnings.

It does not prove:

  • Any peripheral works against real silicon. That's HIL territory (see docs/test-plan.md; ⏳ / 🟡 rows flip to ✅ only when a person attaches real-bench evidence per docs/ci/HW-IN-LOOP.md -- there is no automated runner or CI workflow that does this).
  • Performance or memory budgets. See tests/bench/ for those.

Layers of verification

┌────────────────────────────────────────────────────────────────┐
│ HIL: real silicon, real broker, real sensor                    │
│   Explicit bench run, no CI workflow (docs/ci/HW-IN-LOOP.md).  │
│   Flips test-plan.md rows from 🟡 → ✅.                         │
├────────────────────────────────────────────────────────────────┤
│ CI: GitHub-hosted runners, no hardware                         │
│   pr-twister + pr-plain-cmake + pr-static-analysis +           │
│   pr-generated-files + pr-metadata-validate + pr-doxygen +     │
│   coverity (weekly)                                            │
│   VS Code extension CI runs in alplabai/alp-sdk-vscode.        │
│   Necessary but not sufficient to tag a release.               │
├────────────────────────────────────────────────────────────────┤
│ Local: developer workstation, no hardware                      │
│   scripts/test-all.sh wraps every locally-runnable stage.      │
│   Same coverage as CI; runs in ~3 min on a warm cache.         │
└────────────────────────────────────────────────────────────────┘

Coverage map per public header

Header Local test(s) Bench evidence (flips ✅ per HW-IN-LOOP.md)
<alp/peripheral.h> — I²C tests/yocto/peripheral_i2c.c + tests/unit/i2c_registry/ AEN bench + Yocto bench
<alp/peripheral.h> — SPI tests/yocto/peripheral_spi.c + tests/unit/spi_registry/ AEN bench + Yocto bench
<alp/peripheral.h> — UART tests/yocto/peripheral_uart.c + tests/unit/uart_registry/ + examples/peripheral-io/uart-echo/ AEN bench + Yocto bench
<alp/peripheral.h> — UART RX ringbuf tests/unit/uart_registry/ (rx_ringbuf scenario) + examples/peripheral-io/uart-rx-ringbuf/ AEN bench
<alp/peripheral.h> — GPIO tests/yocto/peripheral_gpio.c + tests/unit/gpio_registry/ + examples/peripheral-io/gpio-button-led/ AEN bench + Yocto bench
Portable-class lifecycle contract — every alp_<class>_open/close/capabilities (14 classes + I²C/SPI target modes, alp_init/alp_deinit, UART RX ringbuf, dispatcher ops-vtable NULL-slot guard) tests/zephyr/conformance/ — the data-driven conformance gate a new backend must pass; on real-SoM builds expectations derive from alp_has() (caps↔backend parity) and WDT arming is opt-in (CONFIG_TEST_ALP_CONFORMANCE_WDT_ARM); see the matrix in its src/main.c + docs/porting-new-som.md §11 AEN bench (qualified boards in platform_allow build via --build-only today; CI runs native_sim)
<alp/pwm.h> tests/unit/pwm_registry/ + examples/peripheral-io/pwm-led-fade/ AEN bench
<alp/adc.h> tests/unit/adc_registry/ + examples/peripheral-io/adc-voltmeter/ AEN bench
<alp/dac.h> tests/zephyr/conformance/ only (no tests/unit/dac_registry/) + examples/peripheral-io/dac-waveform/ + examples/aen/aen-dac-regcheck/ AEN bench
<alp/counter.h> tests/unit/counter_registry/ + tests/unit/qenc_registry/ + examples/power-timing/counter-alarm/ + examples/peripheral-io/qenc-readout/ AEN bench
<alp/i2s.h> tests/unit/i2s_registry/ + examples/audio/i2s-tone/ AEN bench
<alp/can.h> tests/unit/can_registry/ + examples/peripheral-io/can-loopback/ AEN bench
<alp/rtc.h> tests/unit/rtc_registry/ + examples/power-timing/rtc-clock/ AEN bench
<alp/wdt.h> tests/unit/wdt_registry/ + examples/power-timing/wdt-feed/ AEN bench
<alp/i3c.h> tests/zephyr/conformance/ only (no tests/unit/i3c_registry/) + examples/aen/aen-i3c-regcheck/ — no portable examples/peripheral-io/ sibling yet, and the regcheck is not observable under a Flow C RAM-run (#935) AEN bench
<alp/audio.h> (Zephyr) tests/zephyr/audio/ + examples/audio/audio-loopback/ AEN bench
<alp/audio.h> (Yocto, ALSA) tests/yocto/audio_alsa.c — 11 failure paths Yocto bench
<alp/iot.h> — MQTT cleartext tests/yocto/iot_mqtt.c (parse + open) + tests/zephyr/iot/ Yocto bench + AEN bench
<alp/iot.h> — MQTT TLS tests/yocto/iot_mqtt.c (5 TLS tests: default / pinned-CA / missing-CA / insecure / default-port-8883) Yocto bench
<alp/security.h> (Zephyr) tests/zephyr/security/ AEN bench
<alp/security.h> (Yocto, OpenSSL) tests/yocto/security_openssl.c — 16 tests: SHA-256 NIST "abc" KAT, SHA-384/512 length, AEAD round-trip, tag-mismatch, key-length / NULL refusals, TRNG fill meta-alp-sdk image build (flips ✅)
<alp/ble.h> tests/zephyr/ble/ AEN bench
<alp/mproc.h> — shmem/mbox/hwsem tests/zephyr/mproc/ (smoke scenario) AEN bench
<alp/mproc.h> — IPC framing tests/zephyr/mproc/ (nanopb_framing scenario, 9 ZTESTs) AEN bench
<alp/inference.h> tests/zephyr/inference/ + tests/yocto/inference_dispatcher.c AEN bench + Yocto bench
<alp/usb.h> tests/zephyr/usb/ AEN bench
<alp/hw_info.h> tests/zephyr/hw_info/ AEN bench + production-test bench
<alp/display.h> / <alp/gui.h> / <alp/camera.h> / <alp/storage.h> compile-only via tests/smoke.c + headers-include test (real impls pending)
Chip drivers (chips/*/) tests/zephyr/chips/ with fakes for lsm6dso, bme280, ssd1306 per-chip on AEN bench
<alp/soc_caps.h> generation pr-generated-files.yml (drift gate) n/a (generator-deterministic)
ABI snapshot scripts/abi_snapshot.py + docs/abi/v0.17-snapshot.json (drift gate) n/a
board.yaml schema + loader pr-metadata-validate.yml smoke + tests/scripts/test_project_*.py n/a

How to run a single layer

Yocto / plain-CMake only

cmake -B build/yocto -S . -DALP_OS=yocto -DALP_BUILD_TESTS=ON
cmake --build build/yocto --parallel
ctest --test-dir build/yocto --output-on-failure

Optional native libs unlock the Yocto-side wrappers (otherwise they stay at the NOSUPPORT stubs and the matching tests skip):

sudo apt-get install -y libmosquitto-dev libasound2-dev libssl-dev pkg-config

Zephyr / twister only

export ZEPHYR_BASE="$PWD/../zephyr"
# Examples load a pre-generated per-core Kconfig fragment (#866):
python3 scripts/gen_example_alp_conf.py
python3 "$ZEPHYR_BASE/scripts/twister" \
    --testsuite-root tests/zephyr \
    --testsuite-root examples \
    -p native_sim/native/64 \
    --inline-logs --no-detailed-test-id

Baremetal (compile-only)

cmake -B build/baremetal -S . -DALP_OS=baremetal
cmake --build build/baremetal --parallel
# No tests at this layer yet -- the v0.1 baremetal backend is stubs.
# Real-impl tests land alongside the bare-metal AEN port.

Just the new feature scenarios

# LwRB UART RX ringbuf (feature-on compile + ZTEST run)
twister --testsuite-root tests/zephyr -p native_sim/native/64 \
    -T tests/unit/uart_registry -s alp_sdk.unit.uart_registry

# nanopb mproc IPC framing
twister --testsuite-root tests/zephyr -p native_sim/native/64 \
    -T tests/zephyr/mproc -s alp_sdk.mproc.nanopb_framing

Static analysis

# One-time: pin local clang-format to v22 (the version CI uses).
bash scripts/setup-clang-format.sh

# Diff-only clang-format (matches CI's pr-static-analysis behaviour).
git diff -U0 "$(git merge-base origin/dev HEAD)" -- '*.c' '*.h' ':!zephyr/**' ':!vendors/**' ':!tests/scripts/fixtures/rzv2n_svd/**' | clang-format-diff.py -p1

Skipping the pin step is the single most common cause of green-locally / red-in-CI on the pr-static-analysis job -- clang-format reflows a handful of constructs differently between major versions. See docs/contribution.md for the divergence list.

CI ↔ local correspondence

Every CI workflow has a local counterpart that runs the same coverage:

CI workflow Local equivalent
pr-plain-cmake.yml bash scripts/test-all.sh --yocto-only
pr-twister.yml bash scripts/test-all.sh --zephyr-only
pr-static-analysis.yml bash scripts/test-all.sh (clang-format-diff + shellcheck stages)
pr-generated-files.yml python3 scripts/gen_soc_caps.py --check
pr-metadata-validate.yml python3 scripts/validate_metadata.py + alp_project.py
public/private classifier python3 scripts/check_public_private.py
pr-doxygen.yml bash scripts/test-all.sh (doxygen stage; committed docs/doxygen/Doxyfile)
(extension CI lives in alplabai/alp-sdk-vscode) cd ../alp-sdk-vscode && npm test
coverity.yml none (Coverity Scan only)

The plain run of scripts/test-all.sh covers everything except Coverity and pr-generated-files (those are narrow drift checks and run automatically in CI).

Per-feature verification policy

Hardware-bound features ship ⏳ untested and stay there until HIL evidence lands. Software-bound features (parsers, framing, KATs) can flip to ✅ on a deterministic test alone -- see the security backend row in docs/test-plan.md for an example.

The full status legend is at the top of docs/test-plan.md.

Adding tests for new features

The convention every new feature follows:

  1. Public surface change. Add doxygen-commented declarations to the matching <alp/...> header.
  2. Failure-path tests. Append ZTESTs (Zephyr) or test_* functions (Yocto / plain-CMake) covering NULL args, INVAL inputs, NOSUPPORT contracts.
  3. Test-plan row. Append a row in docs/test-plan.md with default status ⏳ untested (or 🟡 partial if failure paths are covered and the happy path runs on the local runner).
  4. Coverage map row. Add an entry to the table above mapping the new function to its test(s).
  5. CHANGELOG entry. Document the feature under [Unreleased].

The matching pull request template is at .github/PULL_REQUEST_TEMPLATE.md. The CONTRIBUTING.md walks through the same checklist.

See also