Drop the HTTP fallback: directives arrive on <root>/<id>/alexaDirective and reports leave over MQTT

The library fetched every directive over HTTP (GET /Alex2ESP/<token>, the token taken from <root>/<id>/alexaDirective_e) and posted every report back over HTTP. The detour was built in 2024 because AsyncMqttClient hands a message larger than one TCP segment to onMessage in fragments. Alex2MQTT publishes the whole directive on <root>/<id>/alexaDirective, so the fragments are reassembled instead: by index/total, into one heap block of total + 1 bytes that is freed as soon as loop() has parsed it.

Removed: ESP8266HTTPClient with processHttpGet/processHttpPost and their two HTTPClient and two WiFiClient members; the 5-slot packet, topic and receive queues, combinedData, receivePayload and AlexaStatusMessage::outputString (17,264 bytes of .bss); the {MESSAGE_PAYLOAD_SPLIT} and {REPLACE_WITH_DATETIME} conventions; the subscription to the _e token topic; AlexaUtils.cpp (AlexaUtils::printMemoryInfo stays, in the header).

Receive: the bridge subscribes <root>/discover and <root>/+/alexaDirective. The MQTT callback only collects; loop() parses and dispatches, and answers discovery. A directive over ALEX2ESP_MAX_DIRECTIVE (2047) is refused. One directive is in flight: a second one that arrives before loop() ran is dropped. Both are logged. A byte-identical repeat of the waiting directive and a repeat of one of the last four messageIds (kept as 64-bit hashes, 32 bytes) are ignored, because the broker mirror delivers every message twice. The wildcard subscription also delivers the directives of other boards of the account; they are recognised by their topic before anything is allocated.

Publish: measureJson first. A message is refused when it is over ALEX2ESP_MAX_MESSAGE (2047), when its document overflowed, when there is no session, when the packet does not fit the largest free block, or when AsyncMqttClient returns 0. Every refusal is logged and returned as an AlexaSendResult; AlexaStatusMessage::send() keeps its bool. The limit now also applies to the discovery object of a device.

Time: timeOfSample is an ISO 8601 UTC instant from the clock of the board (gmtime_r + snprintf, no strftime). begin() calls configTime(0, 0, "pool.ntp.org", "time.nist.gov") and loop() holds the first connect until the clock is set or 5 s have passed. setTimeSource(false) leaves the clock to the sketch. AddContextProp() fills timeOfSample in when a hand-built property has none or carries the old placeholder.

Log: AlexaLog, with a level at run time (setLogLevel) and a ceiling at build time (ALEX2ESP_LOG_MAX), replaces the Serial prints and the Alex2ESP_DEBUG define; the new receive and publish paths need a line for every refusal. The log and the time stamp call vsnprintf/snprintf with a PROGMEM format and not the _P variants: newlib keeps those in one object with printf_P and sprintf_P, which links the FILE-based printf. Measured on basicLight: 5,776 bytes of flash.

For a sketch: the 1.x API is unchanged, the five examples compile unmodified and without warnings. loop() no longer blocks for two HTTP round trips per directive. send() publishes at once and returns false without a session, where 1.1.0 queued the report. The MQTT session opens from loop(), up to 5 s after begin(). getState() is CONNECTED once both subscriptions are acknowledged. The 1.2.0 section of readme.md lists every change; its transport section is rewritten.

Measured for d1_mini with empty credentials (PlatformIO 6.2.0, espressif8266 4.2.1, Arduino core 3.1.2), static RAM / flash in bytes, 1.1.0 -> this commit:
  basicLight           52,768 / 350,885 -> 34,116 / 336,757
  lightWithBrightness  52,880 / 354,729 -> 34,232 / 340,517
  lightWithColorTemp   53,028 / 355,389 -> 34,380 / 341,177
  tempSensor           52,676 / 349,441 -> 34,024 / 335,457
  blindControl         52,900 / 353,069 -> 34,256 / 338,925
basicLight with -DALEX2ESP_LOG_MAX=0: 34,108 / 333,289. SNTP and the time stamp are 1,848 bytes of the flash figure.

Tests: platformio.ini with [env:native] and test/test_bridge_logic, 32 host tests of the logic in src/AlexaBridgeLogic.cpp (reassembly at every fragment size, both limits, repeats, publish checks, topics, time stamps). They also pass under -fsanitize=address,undefined. .gitignore no longer hides /test.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
David 2026-09-28 14:31:57 +00:00
parent 21cc693e74
commit 8ea777806a
20 changed files with 1840 additions and 792 deletions

View file

@ -135,9 +135,16 @@ An `ActionMapping` takes an optional third argument, the directive payload as JS
## How it talks to Alex2MQTT
- **Discovery.** On `<root>/discover` the library answers with one discovery object per device, published straight to `<root>/discover_r` over MQTT (no HTTP round trip, no queue slot). The backend accepts one endpoint object per message and collects everything that arrives within 1 s for Alexa's discovery answer (up to 5 s for its proactive AddOrUpdate push), so all devices are published back to back the moment the request arrives. Each object sits on the heap (about 1 KB) until the broker acknowledges it; when the MQTT client refuses another one (free heap under 4 KB), the library prints `[Alex2ESP] discovery publish deferred at <endpointId>` and sends the rest from `loop()` as the queue drains, for up to 5 s after the request. `[Alex2ESP] discovery gave up: N device(s) not announced` means those devices missed this answer - on the backend's proactive discovery that can remove them from Alexa until the next one.
- **Directives.** A directive arrives as a short id on `<root>/<endpointId>/alexaDirective_e`; the library fetches the full directive over HTTP, fires `ReportState` or `Event` (and `DirectiveReceived`, if registered), and your status report is queued for an HTTP POST that the backend republishes on `<root>/<endpointId>/alexaResponce`, replacing `{REPLACE_WITH_DATETIME}` with the current time.
- **Limits.** The send queue holds 5 reports of up to 2047 bytes each. `send()` returns `false` and prints a `[Alex2ESP]` line on Serial when a report does not fit or the queue is full; nothing is ever sent truncated. Debug logging of the library's internals is compiled in by defining `Alex2ESP_DEBUG` in `AlexaUtils.cpp`; credentials are never printed.
Everything goes over MQTT (port 1883 of `alex2mqtt.stormysdream.club`); the library makes no HTTP requests.
- **Session.** `begin()` starts SNTP (`pool.ntp.org`, `time.nist.gov`) and returns; `loop()` opens the MQTT session once the clock is set, or after 5 s without an answer, and subscribes to `<root>/discover` and `<root>/+/alexaDirective`. `getState()` is `CONNECTED` when the broker has acknowledged both subscriptions. A sketch that sets the clock itself (its own `configTime()` with a time zone, an RTC) calls `alexClient.setTimeSource(false)` before `begin()`.
- **Discovery.** On `<root>/discover` the library answers with one discovery object per device on `<root>/discover_r`. The backend accepts one endpoint object per message and collects everything that arrives within 1 s for Alexa's discovery answer (up to 5 s for its proactive AddOrUpdate push), so all devices are published back to back from the next `loop()`. Each object sits on the heap (about 1 KB) until the MQTT client has sent it; when the client cannot take another one (free heap under 4 KB), the library prints `[Alex2ESP] discovery deferred at <endpointId>` and sends the rest from `loop()` as the queue drains, for up to 5 s after the request. `[Alex2ESP] error: discovery gave up: N device(s) not announced` means those devices missed this answer - on the backend's proactive discovery that can remove them from Alexa until the next one.
- **Directives.** The directive arrives as JSON on `<root>/<endpointId>/alexaDirective`. A directive larger than one TCP segment arrives in fragments, which are put together in one heap block that exists only until `loop()` has parsed it. `loop()` then fires `ReportState` or `Event` (and `DirectiveReceived`, if registered) with the directive: `directive["header"]`, `directive["endpoint"]`, `directive["payload"]`. One directive is handled at a time; a directive that arrives twice (a broker that mirrors its topics delivers every message twice) is handled once.
- **Reports.** `send()` publishes the report on `<root>/<endpointId>/alexaResponce` at once. The backend waits 7 s for it, so answer from the event handler. Every property carries the board's UTC time as `timeOfSample`.
- **Limits.** A directive and a report (or the discovery object of one device) may be 2047 bytes each; `-DALEX2ESP_MAX_DIRECTIVE=<bytes>` and `-DALEX2ESP_MAX_MESSAGE=<bytes>` in `build_flags` change that. `send()` returns `false` when the report was not sent: no session with the broker, the MQTT client or the heap cannot take it, or it is too large. Nothing is ever sent truncated, and nothing is dropped without a line on Serial.
- **Serial output.** Every line of the library starts with `[Alex2ESP]`, a problem with `[Alex2ESP] error:`. `alexClient.setLogLevel(AlexaLogLevel::ERROR)` leaves only the problems, `AlexaLogLevel::NONE` nothing; the default, `AlexaLogLevel::INFO`, adds the session, discovery and one line per directive (`[Alex2ESP] ESP-01 <- Alexa.PowerController.TurnOn`). `AlexaLogLevel::DEBUG` (sizes and free heap per message) has to be compiled in with `-DALEX2ESP_LOG_MAX=3`; `-DALEX2ESP_LOG_MAX=0` compiles every line out. Credentials and correlation tokens are never printed.
Boards that run 1.1.0 or older keep working: the backend still publishes the token on `<root>/<endpointId>/alexaDirective_e` and serves the HTTP routes they use.
---
@ -222,15 +229,38 @@ For instance, to manually report the state of a PowerController, you can use the
doc["namespace"] = "Alexa.PowerController";
doc["name"] = "powerState";
doc["value"] = "ON";
doc["timeOfSample"] = "{REPLACE_WITH_DATETIME}"; // Alex2MQTT server wil do the replace
doc["uncertaintyInMilliseconds"] = 0;
AddContextProp(doc.as<JsonObject>())
```
`AddContextProp` sets `timeOfSample` to the current time when the object has none. (Until 1.1.0 a sketch wrote `"{REPLACE_WITH_DATETIME}"` there for the server to replace; the library now replaces that itself.)
---
## Changelog
### 1.2.0
Behaviour changes:
- Directives arrive over MQTT. The library subscribes to `<root>/+/alexaDirective`, where Alex2MQTT has always published the whole directive, instead of fetching it over HTTP with the token from `<root>/<endpointId>/alexaDirective_e`. The HTTP detour dates from 2024, when a directive larger than one TCP segment reached the MQTT callback in pieces; the pieces are now put together by their offset and the total length, in one heap block that lives until `loop()` has parsed the directive. `loop()` no longer stalls for two HTTP round trips per directive.
- Reports leave over MQTT. `send()` publishes on `<root>/<endpointId>/alexaResponce` at once, where 1.1.0 queued the report for an HTTP POST from a later `loop()`. It returns `false` when there is no session with the broker, when the MQTT client or the heap cannot take the report, or when the report is over 2047 bytes (`ALEX2ESP_MAX_MESSAGE`); each case prints its reason. The 5-slot send queue is gone.
- `timeOfSample` is the board's own time in UTC, for example `2026-09-28T13:05:09Z`. 1.1.0 sent the placeholder `{REPLACE_WITH_DATETIME}`, which only the backend's HTTP route replaced. `begin()` starts SNTP (`pool.ntp.org`, `time.nist.gov`) and no longer connects itself: `loop()` opens the MQTT session once the clock is set, or after 5 s without an answer, so the session comes up a few seconds later than before. `alexClient.setTimeSource(false)` before `begin()` leaves the clock to the sketch. `AddContextProp()` fills `timeOfSample` in when the property has none or carries the old placeholder.
- A directive is parsed and handed to the sketch from `loop()`, one at a time. One that arrives while the previous one still waits for `loop()` is dropped, as is one over 2047 bytes (`ALEX2ESP_MAX_DIRECTIVE`); both print an error.
- A directive that arrives twice is handled once: the Alex2MQTT broker currently delivers every message twice through a mirror. The `messageId`s of the last four directives are remembered; the repeat prints `repeated directive ... ignored`.
- The subscription delivers the directives of every endpoint of the account. Those for endpoints of another board are recognised by their topic and neither buffered nor parsed.
- Discovery is answered from `loop()`, not inside the MQTT callback. A discovery object over 2047 bytes is refused with an error; the other devices are still announced.
- `getState()` stays `INITIALIZED` until the first connect, and becomes `CONNECTED` when the broker has acknowledged both subscriptions (1.1.0: the first of them). A refused subscription prints an error.
- Serial output goes through one log with levels. `alexClient.setLogLevel()` takes `AlexaLogLevel::NONE`, `ERROR`, `INFO` (the default) or `DEBUG`; `-DALEX2ESP_LOG_MAX=<0..3>` in `build_flags` sets the highest level that is compiled in (default 2, `INFO`). This replaces the `Alex2ESP_DEBUG` define inside `AlexaUtils.cpp`. 1.1.0 printed the memory figures for every MQTT message and the whole directive, correlation token included, for every directive; both are gone.
- `getDevice()` before `begin()` prints an error: the device would have no root topic. A second `begin()` is ignored with an error.
- New: `Alex2ESP::setLogLevel()`, `Alex2ESP::setTimeSource()`, `AlexaDevice::hasEndpointId()`, `AlexaLog`, `AlexaSendResult`.
- Removed: the queues and buffers of `AlexaUtils` (`enqueue`, `dequeue`, `dequeueVals`, `enqueueReceive`, `dequeueReceive`, `isQueueEmpty`, `isQueueFull`, `isReceiveQueueEmpty`, `isReceiveQueueFull`, `receivePayload`, `nextMessageId`) and its `log`/`logln`, which printed nothing unless the library was edited; `AlexaUtils::printMemoryInfo()` stays. `MAX_STATUS_REPORT_SIZE` (the limit is `ALEX2ESP_MAX_MESSAGE`). The library no longer includes `ESP8266HTTPClient`.
Memory: `examples/basicLight.cpp` for a D1 mini takes 34,116 bytes of static RAM (1.1.0: 52,768) and 336,757 bytes of flash (1.1.0: 350,885), as PlatformIO reports them (espressif8266 4.2.1, Arduino core 3.1.2). The static RAM was the five 2 KB queue slots, three more 2 KB buffers and the two HTTP clients. SNTP and the time stamp are 1.8 KB of the flash figure.
Tests: `pio test -e native` in the repository runs 32 host tests of the receive and publish logic (reassembly of fragments, the two size limits, repeated directives, topics, time stamps). No board is needed.
Boards that run 1.1.0 are not affected: the backend keeps the token topic and the HTTP routes.
### 1.1.0
Behaviour changes:
@ -253,7 +283,7 @@ Packaging: `library.json` and `library.properties` restored with the Forgejo URL
---
## Contributing
Feel free to submit pull requests or issues for feature requests and bug fixes.
Feel free to submit pull requests or issues for feature requests and bug fixes. `pio test -e native` in the repository root runs the host tests (no board needed); they have to pass.
---