Test Results
A PASS without recorded evidence is rejected by
validate.py. Strategy: test-strategy.md| ID | Test | Type | Req | Task | Status | Actual result |
|---|---|---|---|---|---|---|
| TEST-001 | Hardware baseline regression: GNSS, LBS, data bearer, camera, SD, SMS, voice all still report success after a firmware change. | hardware_regression | REQ-007 | TASK-002 | NOT_RUN | — |
| TEST-002 | The project gate is proven to fail on bad state. Covers both validate.py (14 seeded violations) and secrets_scan.py (4 cases). | unit | — | TASK-003 | PASS | validate.py: correctly failed a fixture seeded with 8 deliberate violations, naming each rule, and caught 5 derived ones (missing completed_at, verified_at, invalid priority, orphan refs, PASS-without-evidence). secrets_ evidence: Captured runs. validate fixture: 13 errors, exit 1. Scanner: exit 1 on tracked leaks (2/4), exit 0 on placeholder template and on gitignored secrets.h. Recorded in LESSONS_LEARNED.md as L-08. |
| TEST-003 | No broker credential exists in any git-tracked file. | security | REQ-021 | TASK-004 | PASS | zero matches across working tree and full git history (66 committable files) evidence: python3 scripts/project-tracker/secrets_scan.py --history -> PASS |
| TEST-103 | Modem reaches network registration and a live data bearer within 60 s of the PWRKEY power-on sequence. | hardware_regression | REQ-003 | TASK-103 | PASS | At baseline measured 2026-09-29: +CEREG: 0,1, +CGPADDR: 1,100.90.93.199, +CSQ: 26,99 evidence: lbsmatrix serial capture 2026-09-29, markers MARK-BEARER-UP through MARK-CLBS-WITH-CTX8 |
| TEST-301 | GNSS cold start produces a real fix with quality 1 and sats_used > 0. | hardware_regression | REQ-007 | TASK-301 | PASS | Fix at LOCK t+130s sats=4 hdop=7.3; 18.480816, 73.898640, ALT 550.7m (2026-09-30 17:46 local). Prior 2026-09-29: 18.480776, 73.897998 in 69 s, 17 GPS + 10 GLONASS, C/N0 up to 73. REGRESSION 2026-09-30 18:37-18:49 local: evidence: gnss_hold serial captures 2026-09-29 (69s) and 2026-09-30 17:46 (/tmp/gnss-hold5.log, LOCK t+130s sats=4); GGA quality 1 confirmed. 0-sat regression same day 18:37-18:49: /tmp/gnss-hold6.log, /tmp/gnss-hold7.log, /tmp/gnss-hold8.log, /tmp/gnss-hold8b.log (mode:DIO clock div:1, GNSS-ENGINE-NO-READY, NMEA streaming, satsUsed=0 fix=no to t+500s); also /tmp/position7.log and /tmp/position10.log (searching satsInView=0 hdop=99.99). |
| TEST-302 | Cellular LBS returns a position that agrees with the independent GNSS fix. | hardware_regression | REQ-008 | TASK-303 | FAIL | 09-29: +CLBS: 0,18.481703,73.897415,550 (PASS, corroborated by GNSS 18.480776, 73.897998). 09-30: +CLBS: 10 in all tested modes (mode1-cid1, mode9-cid1, mode2-cid1, default-CID, mode4) with true cold modem power-cycle an evidence: lbsmatrix serial capture 2026-09-29: MARK-TRY-mode1-cid1 VERDICT => SUCCESS (+CLBS: 0). lbsmatrix 2026-09-30 (/tmp/lbsmatrix-now.log): all modes return +CLBS: 10. lbs_debug 09-30 with cold modem power-cycle (/tmp/lbs-cold.log): PWRKEY OFF/ON, byte-for-byte 09-29 flow -> +CLBS: 10 for mode1/9/2. Socket probe (/tmp/lbs-debug8/9.log): CIPOPEN to lbs-simcom.com:3002 OK, CIPOPEN to mqtt.easyiothings.com:80 OK, but AT+CLBS returns 10 even with NETOPEN active. See /tmp/lbs-cold.log, /tmp/lbsmatrix-now.log, /tmp/lbs-debug9.log. |
| TEST-403 | A telemetry payload published by the device is received by an independent MQTT subscriber on the target cluster. | end_to_end | REQ-012 | TASK-405 | PASS | MQTT over PPP verified end to end on the physical unit (firmware A7670M7_V1.11.1). CONNACK rc=0 sp=00, MARK-MQTT-CONNECTED, PUBLISH 71 bytes to nomadlink/one/gps, RESULT MQTT_PUBLISH PASS. An independent mosquitto_sub on evidence: mqtt_ppp serial capture 2026-09-30: 'connect select() -> ready after 3201 ms', 'TCP to broker CONNECTED', 'CONNACK rc=0 sp=00', 'MARK-MQTT-CONNECTED', 'RESULT MQTT_PUBLISH PASS'; subscriber log: nomadlink/one/gps {\"src\":\"mqtt_ppp\",\"uptime_s\":32,\"phase\":\"RUNNING\"} |
| TEST-104 | NAT throughput through the SoftAP meets the PRD NFR of 10-15 Mbps. | performance | REQ-004 | TASK-104 | NOT_RUN | — |
| TEST-105 | WAN failover from Wi-Fi STA to LTE occurs within 3 s of upstream packet loss. | performance | REQ-004 | TASK-104 | NOT_RUN | — |
| TEST-201 | OV2640 captures live frames with non-zero luma variance. | hardware_regression | REQ-005 | TASK-201 | PASS | MID 0x7fa2, PID 0x0026, VER 0x0042. Luma mean 35.0, stddev 15.4, block stddev 9.2. Non-zero variance is what proves a live sensor; a failed init also yields a flat fill. evidence: cam_diag serial capture |
| TEST-501 | Outbound SMS is received by the recipient, with delivery report enabled. | hardware_regression | REQ-014 | TASK-501 | PASS | +CMGS: 2 and +CMGS: 3, both confirmed received by the user. Note AT+CMGL="SENT" is NOT supported (CMS ERROR: Operation not allowed), so the sent box cannot be read back. evidence: sms_test serial capture; user confirmed receipt of both messages |
| TEST-601 | Outgoing voice call establishes and reaches active state. | hardware_regression | REQ-016 | TASK-601 | PASS | +VOICE CALL: BEGIN and +CLCC: 1,0,2 (active). User reported hearing audio. NOTE: the far-end audio path has not been separately proven and ringback must not be confused with full-duplex voice. See RISK-003. evidence: voice_sms serial capture |
| TEST-804 | SD card mounts and completes a write/read/delete round trip under the product firmware framework. | hardware_regression | REQ-023 | TASK-805 | FAIL | PASSES under Arduino-ESP32 3.3.11 (3,984,588,800-byte FAT32 card). FAILS under ESP-IDF with err=0x107 on every card command. The framework discrepancy is unresolved. See BUG-001 and RISK-006. evidence: sd_diag capture (Arduino pass); nomad-sentinel/TASKS.md 1.7 (IDF fail) |
| TEST-020 | PPP establishes and assigns an IP on the Airtel SIM. | integration | REQ-004 | TASK-020 | PASS | PASS. All four steps completed on the Airtel SIM.
STEP 1 PASS - data bearer up. AT+CGATT=1 -> OK. AT+CGDCONT=1,"IP","airtelgprs.com" -> OK. AT+CGACT=1,1 -> OK. AT+CGPADDR=1 -> +CGPADDR: 1,100.76.89.65 (a carrier-allocat evidence: Serial console capture on /dev/cu.usbmodem58750034621, firmware/baseline/ppp_client, Arduino ESP32 core 3.3.11. Verbatim excerpts, first run: > ATD*99# < CONNECT received - handing UART to PPP > TX frame 1 (45 bytes): 7E FF 7D 23 C0 21 7D 21 7D 21 7D 20 7D 34 7D 22 7D 26 7D 20 7D 20 7D 20 7D 20 7D 25 7D 26 CA 73 8C 9E 7D 27 7D 22 7D 28 7D 22 36 4B 7E > PPP phase -> ESTABLISH | NETWORK | RUNNING > PPP-IP local=100.76.140.112 gateway=10.64.64.64 up=1 mtu=1500 > RESULT PPP_NEGOTIATION PASS (phase RUNNING, 217 bytes out, 235 bytes in) > [link] up phase=RUNNING out=217 in=235 (repeated, stable, no reset) Cold-boot repeat run, ESP32 reset via DTR/RTS with the USB cable untouched: > PPP phase -> NETWORK | RUNNING > PPP-IP local=100.77.131.255 gateway=10.64.64.64 up=1 mtu=1500 > RESULT PPP_NEGOTIATION PASS (phase RUNNING, 230 bytes out, 273 bytes in) |
| TEST-021 | I2S audio hardware is present and a mic-to-DAC loopback carries signal. | hardware_regression | REQ-017 | TASK-021 | NOT_RUN | — |
| TEST-101 | A client associates to the SoftAP and receives a DHCP lease. | integration | REQ-001 | TASK-101 | NOT_RUN | — |
| TEST-102 | Stored Wi-Fi STA credentials persist across a reboot and auto-reconnect. | integration | REQ-002 | TASK-105 | NOT_RUN | — |
| TEST-202 | Camera control page applies resolution, quality and FPS changes at runtime. | end_to_end | REQ-006 | TASK-203 | NOT_RUN | — |
| TEST-303 | Location page shows live position and the map marker moves. | end_to_end | REQ-009 | TASK-304 | NOT_RUN | — |
| TEST-401 | Broker credentials entered in the web UI are stored and used for a connection. | integration | REQ-010 | TASK-401 | NOT_RUN | — |
| TEST-402 | Telemetry start/stop and interval control publish rate. | integration | REQ-011 | TASK-403 | NOT_RUN | — |
| TEST-404 | An MQTT command received on the command topic is executed and acknowledged. | end_to_end | REQ-013 | TASK-408 | NOT_RUN | — |
| TEST-502 | SMS commands are parsed and answered, and unauthorised senders are ignored. | end_to_end | REQ-015 | TASK-503 | NOT_RUN | — |
| TEST-602 | A call can be placed from the web dialpad and the audio path carries voice. | end_to_end | REQ-017 | TASK-604 | NOT_RUN | — |
| TEST-701 | ESP-SR wake-word and command recognition latency is under the 800 ms NFR. | performance | REQ-018 | TASK-701 | NOT_RUN | — |
| TEST-702 | A voice command dials the mapped contact. | end_to_end | REQ-019 | TASK-703 | NOT_RUN | — |
| TEST-801 | Every portal page loads and renders its live data. | smoke | REQ-020 | TASK-802 | NOT_RUN | — |
| TEST-802 | Configuration persists across reboot with no secret in source. | integration | REQ-021 | TASK-402 | NOT_RUN | — |
| TEST-803 | Security controls hold: admin auth enforced, no secret exposure, TLS in use. | security | REQ-022 | TASK-804 | NOT_RUN | — |
| TEST-810 | Flash the vendored esp32_nat_router base from the GitHub Pages web installer and confirm the device broadcasts the NomadLink SSID and serves its console. | hardware_verification | REQ-001 | TASK-102 | PASS | AP broadcasts SSID 'NomadLink'. The owner additionally confirmed the device delivers working internet access: 'internet works using NomadLink'. This was observed after a WiFi uplink was configured via the base's own /set evidence: User confirmation in session, 2026-10-01: 'now NomadLink is working' and 'internet works using NomadLink'. Independent observer, not a self-asserted marker. Corroborating repository-side evidence: published page HTTP 200 containing 'Connect & Install' (20479 B); manifest 200; all four firmware parts 200 at the declared offsets (17664 / 3072 / 8192 / 1374544 B). PROVEN BY THIS TEST: AP boots and broadcasts 'NomadLink'; a client behind the AP reaches the internet, so SoftAP + DHCP lease + NAPT + forwarding + STA uplink all work; the delivered payload reaches the device. NOT MEASURED: forward throughput in Mbps, client count under load, or any 4G/GNSS/camera/SD capability - none is in this image. Also unmeasured: the base's /scan, /setup and /config pages were used, but not systematically tested against every field. |