Update image public.ecr.aws/emqx/emqx to v5.8.2 #858

Merged
jahanson merged 1 commit from renovate/public.ecr.aws-emqx-emqx-5.x into main 2024-11-16 07:20:27 -06:00
Collaborator

This PR contains the following updates:

Package Update Change
public.ecr.aws/emqx/emqx (source) patch 5.8.1 -> 5.8.2

Release Notes

emqx/emqx (public.ecr.aws/emqx/emqx)

v5.8.2: EMQX v5.8.2

Compare Source

5.8.2

Release Date: 2024-11-12

Enhancements
Core MQTT Functionalities
  • #​14059 Added a new configuration option for the retainer to cap message expiry intervals for retained messages. This enables garbage collection to remove messages sooner if storage is running low.

  • #​14072 Updated the virtual machine to use Unicode for its printable range. This improvement enhances the readability of certain binary data in messages. For instance, a binary previously displayed as <<116,101,115,116,228,184,173,230,150,135>> will now be formatted as <<"test中文"/utf8>>, providing clearer representation.

MQTT Durable Sessions
  • #​14130 Reduced CPU usage for idle durable sessions.

    Previously, idle durable sessions periodically woke up to refresh the list of DS streams. With this change, stream discovery is now event-based, significantly lowering CPU consumption during idle periods. Additionally, the update reduces the delay in notifying sessions of new streams, effectively eliminating the long-tail latency in end-to-end processing.

REST API
  • #​13889 Enhanced the performance of the /api/v5/monitor_current and /api/v5/metrics APIs.

    Previously, these APIs queried clustered nodes sequentially in a loop. Now, the queries are sent in parallel, reducing response time. The latency is now primarily dependent on the slowest node in the cluster.

    Additionally, a node parameter was added to the /api/v5/monitor_current API, allowing targeted queries to a single node instead of the entire cluster. For instance, using ?aggregate=false&node=emqx@node1.domain.name will return data exclusively for the specified node.

EMQX Clustering
  • #​13903 Added logs to inform the user when a replicant node cannot find a core node with the same release version as its own.
Security
  • #​13923 Added zone support in authentication, authorization, and mountpoint templates.

    Previously, to reference a client's zone in authentication or authorization rules, users needed to access it through client_attrs. Now, the ${zone} placeholder can be used directly in these templates, simplifying rule creation and enabling zone-specific configurations.

    For example, the following ACL rule uses ${zone} to dynamically apply permissions based on a client’s assigned zone: {allow, all, all, ["${zone}/${username}/#"]}.

  • #​14102 Added support for SSL private key passphrase from a secret file.

    EMQX can now read the passphrase from a secret file if password is configured as ...ssl_options.password = "file://{path-to-secret-file}".

Data Integration
  • #​14065 Added a new queuing_bytes metric for data integration. This metric shows the RAM and/or disk resources consumed by buffering for a specific action.
Observability
  • #​14096 Exposed emqx_conf_sync_txid as a Prometheus metric, allowing for monitoring the configuration file synchronization status of each node in the cluster.
MQTT over QUIC
  • #​13814 Connection Scope Keepalive for MQTT over QUIC Multi-Stream:

    Introduced a new feature to keep MQTT connections alive when data streams remain active, even if the control stream is idle.

    Previously, clients were required to send MQTT.PINGREQ on idle control streams to keep the connection alive. Now, a shared state tracks activity across all streams for each connection. This shared state is used to determine if the connection is still alive, reducing the risk of keepalive timeouts due to Head-of-Line (HOL) blocking.

  • #​14112 Added support ssl_options.hibernate_after in QUIC listener to reduce memory footprint of QUIC transport.

Bug Fixes
Core MQTT Functionality
  • #​13931 Updated the gen_rpc library to version 3.4.1, which includes a fix to prevent client socket initialization errors from escalating to the node level on the server side.

  • #​13969 Optimized the periodic cleanup of expired retained messages to ensure efficient resource usage, particularly in cases with a large volume of expired messages.

  • #​14068 Added the handle_frame_error/2 callback to all gateway implementation modules to handle message parsing errors.

  • #​14037 Improved the internal database bootstrap process to better tolerate temporary unavailability of peer nodes, particularly when a new node joins an existing cluster.

  • #​14116 Fixed an issue where the default configuration for the retainer was generated incorrectly after joining a cluster.

MQTT Durable Sessions
  • #​14042 Fix crash in the durable session after updates to subscription parameters (such as QoS, no_local, upgrade_qos, ...).

  • #​14052 Corrected memory usage reporting from cgroups when in use.

  • #​14055 Updated the /clients_v2 API to properly respect all filtering arguments when querying offline clients with durable sessions. Previously, only the username filter was applied, while other filtering arguments were ignored.

  • #​14151 Fixed handling of the conn_state filter in the /clients_v2 API for offline clients with durable sessions. Previously, these clients could be incorrectly selected with conn_state=connected.

  • #​14057 Resolved a compatibility issue that prevented the Messages DS database from starting due to a slightly different database configuration schema. This issue occurred when upgrading EMQX from version 5.7.x with session durability enabled.

REST API
  • #​14023 Fixed an issue with the GET /monitor HTTP API where returned values could appear higher than actual values, depending on the requested time window. For data points within a 1-hour window, this distortion is only visual on the Dashboard. However, for data points older than 1 hour, the data distortion is permanent.

    The affected metrics include:

    • disconnected_durable_sessions
    • subscriptions_durable
    • subscriptions
    • topics
    • connections
    • live_connections
EMQX Clustering
  • #​13996 Fixed an intermittent crash occurring when using emqx conf fix to resolve configuration discrepancies, particularly if a configuration key was missing on one of the nodes.
Security
  • #​13922 Updated the CRL (Certificate Revocation List) cache to use the full Distribution Point (DP) URL as the cache key. Previously, only the path part of the URL was used, causing conflicts when multiple DPs shared the same path.

  • #​13924 Fixed an issue where JWK keys could leak into debug logs upon JWT authentication failure.

Data Integration
  • #​13916 Fixed an issue where the parent metric failed was not incremented when a rule’s failed.no_result or failed.exception metrics were updated.

  • #​14001 Resolved a race condition where a resource (such as a connector, action, source, authentication, or authorization) could falsely report a connected, healthy channel after a brief disconnection. This issue could result in excessive action_not_found log entries when the race condition occurred.

  • #​13913 Fixed an issue with the actions and source HTTP APIs where a 500 status code would be returned if a timeout occurred while attempting to update or delete a resource.

  • #​14101 Resolved an issue where deleting a resource would fail if a source and an action were both created with the same name.

Observability
  • #​13909 Fixed log formatting for cases where the payload cannot be displayed as readable UTF-8 Unicode characters.

  • #​14061 Improved log information when emqx_cm:request_stepdown/3 fails.

    In scenarios where a client channel needs to terminate another channel with the same ClientID, a race condition may occur if the target channel has already been closed or terminated. In such cases, error logs and stack traces that provide no useful information will no longer be generated.

  • #​14070 Removed the connector's state from error and warning logs due to its potential length. For issue analysis, the connector's state can now be accessed through emqx_resource:list_instances_verbose/0. Below is an example of a log entry before this change:

    pid: <0.43914.0>, connector: connector:sqlserver:connector-05a2e105, reason: [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Argument data type varchar is invalid for argument 2 of dateadd function. SQLSTATE IS: 42000, state: {"resource_opts":{"start_timeout":5000,"start_after_created":true,"health_check_interval":15000},"pool_name":"connector:sqlserver:connector-05a2e105","installed_channels":{"action:sqlserver:action-4b033621:connector:sqlserver:connector-05a2e105":{"sql_templates":{"batch_insert_temp":{"send_message":{"batch_insert_tks":["{str,<<\" ( \">>}","{var,[<<\"messageId\">>]}","{str,<<\", \">>}","{var,[<<\"measurement\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_Fault_1\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_Fault_2\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_Fault_3\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_Fault_4\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_PV_1\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_PV_2\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_PV_3\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_PV_4\">>]}","{str,<<\", DATEADD(MS, \">>}","{var,[<<\"ms_shift\">>]}","{str,<<\", DATEADD(S, \">>}","{var,[<<\"s_shift\">>]}","{str,<<\", '19700101 00:00:00:000') ))\">>}"],"batch_insert_part":"insert into TransactionLog(MessageId, Measurement, Fault1, Fault2, Fault3, Fault4, Value1, Value2, Value3, Value4, DateStamp) \r\n"}}}}}},msg: invalid_request
    
  • #​14099 Removed an error-level log entry that was triggered when validation of UTF-8 strings in MQTT messages failed.

    Example of the removed log entry:

    {"time":"2024-10-11T06:05:07.610048+00:00","level":"error","msg":"supervisor: {esockd_connection_sup,0.53591191.0}, errorContext: connection_shutdown, reason: #{cause => invalid_topic,reason => malformed_utf8_string_length}, offender: [{pid,0.53591191.0},...]", ..., "error_logger":{"type":"supervisor_report","tag":"error_report"}}
    
  • #​14091 Implemented a fix to remove function_clause from log messages when users provide unsupported write syntax.

    Example of unsupported syntax:

    weather,location=us-midwest,season=summer temperature=82 ${timestamp}u 
    

    Before this fix, the error log would contain the function_clause error, as shown:

    pid: <0.558392.0>, info: {"stacktrace":["{emqx_bridge_influxdb_connector,parse_timestamp,[[1719350482910000000,<<\"u\">>]],[{file,\"emqx_bridge_influxdb_connector.erl\"},{line,692}]}", ...], ..., "error":"{error,function_clause}"}, tag: ERROR, msg: resource_exception
    

    This change improves log clarity by omitting function_clause in cases of syntax errors.


Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about these updates again.


  • If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [public.ecr.aws/emqx/emqx](https://www.emqx.io) ([source](https://github.com/emqx/emqx)) | patch | `5.8.1` -> `5.8.2` | --- ### Release Notes <details> <summary>emqx/emqx (public.ecr.aws/emqx/emqx)</summary> ### [`v5.8.2`](https://github.com/emqx/emqx/releases/tag/v5.8.2): EMQX v5.8.2 [Compare Source](https://github.com/emqx/emqx/compare/v5.8.1...v5.8.2) ##### 5.8.2 *Release Date: 2024-11-12* ##### Enhancements ##### Core MQTT Functionalities - [#&#8203;14059](https://github.com/emqx/emqx/pull/14059) Added a new configuration option for the retainer to cap message expiry intervals for retained messages. This enables garbage collection to remove messages sooner if storage is running low. - [#&#8203;14072](https://github.com/emqx/emqx/pull/14072) Updated the virtual machine to use Unicode for its printable range. This improvement enhances the readability of certain binary data in messages. For instance, a binary previously displayed as `<<116,101,115,116,228,184,173,230,150,135>>` will now be formatted as `<<"test中文"/utf8>>`, providing clearer representation. ##### MQTT Durable Sessions - [#&#8203;14130](https://github.com/emqx/emqx/pull/14130) Reduced CPU usage for idle durable sessions. Previously, idle durable sessions periodically woke up to refresh the list of DS streams. With this change, stream discovery is now event-based, significantly lowering CPU consumption during idle periods. Additionally, the update reduces the delay in notifying sessions of new streams, effectively eliminating the long-tail latency in end-to-end processing. ##### REST API - [#&#8203;13889](https://github.com/emqx/emqx/pull/13889) Enhanced the performance of the `/api/v5/monitor_current` and `/api/v5/metrics` APIs. Previously, these APIs queried clustered nodes sequentially in a loop. Now, the queries are sent in parallel, reducing response time. The latency is now primarily dependent on the slowest node in the cluster. Additionally, a `node` parameter was added to the `/api/v5/monitor_current` API, allowing targeted queries to a single node instead of the entire cluster. For instance, using `?aggregate=false&node=emqx@node1.domain.name` will return data exclusively for the specified node. ##### EMQX Clustering - [#&#8203;13903](https://github.com/emqx/emqx/pull/13903) Added logs to inform the user when a replicant node cannot find a core node with the same release version as its own. ##### Security - [#&#8203;13923](https://github.com/emqx/emqx/pull/13923) Added `zone` support in authentication, authorization, and mountpoint templates. Previously, to reference a client's `zone` in authentication or authorization rules, users needed to access it through `client_attrs`. Now, the `${zone}` placeholder can be used directly in these templates, simplifying rule creation and enabling zone-specific configurations. For example, the following ACL rule uses `${zone}` to dynamically apply permissions based on a client’s assigned zone: `{allow, all, all, ["${zone}/${username}/#"]}`. - [#&#8203;14102](https://github.com/emqx/emqx/pull/14102) Added support for SSL private key passphrase from a secret file. EMQX can now read the passphrase from a secret file if `password` is configured as `...ssl_options.password = "file://{path-to-secret-file}"`. ##### Data Integration - [#&#8203;14065](https://github.com/emqx/emqx/pull/14065) Added a new `queuing_bytes` metric for data integration. This metric shows the RAM and/or disk resources consumed by buffering for a specific action. ##### Observability - [#&#8203;14096](https://github.com/emqx/emqx/pull/14096) Exposed `emqx_conf_sync_txid` as a Prometheus metric, allowing for monitoring the configuration file synchronization status of each node in the cluster. ##### MQTT over QUIC - [#&#8203;13814](https://github.com/emqx/emqx/pull/13814) Connection Scope Keepalive for MQTT over QUIC Multi-Stream: Introduced a new feature to keep MQTT connections alive when data streams remain active, even if the control stream is idle. Previously, clients were required to send `MQTT.PINGREQ` on idle control streams to keep the connection alive. Now, a shared state tracks activity across all streams for each connection. This shared state is used to determine if the connection is still alive, reducing the risk of keepalive timeouts due to Head-of-Line (HOL) blocking. - [#&#8203;14112](https://github.com/emqx/emqx/pull/14112) Added support `ssl_options.hibernate_after` in QUIC listener to reduce memory footprint of QUIC transport. ##### Bug Fixes ##### Core MQTT Functionality - [#&#8203;13931](https://github.com/emqx/emqx/pull/13931) Updated the `gen_rpc` library to version 3.4.1, which includes a fix to prevent client socket initialization errors from escalating to the node level on the server side. - [#&#8203;13969](https://github.com/emqx/emqx/pull/13969) Optimized the periodic cleanup of expired retained messages to ensure efficient resource usage, particularly in cases with a large volume of expired messages. - [#&#8203;14068](https://github.com/emqx/emqx/pull/14068) Added the `handle_frame_error/2` callback to all gateway implementation modules to handle message parsing errors. - [#&#8203;14037](https://github.com/emqx/emqx/pull/14037) Improved the internal database bootstrap process to better tolerate temporary unavailability of peer nodes, particularly when a new node joins an existing cluster. - [#&#8203;14116](https://github.com/emqx/emqx/pull/14116) Fixed an issue where the default configuration for the retainer was generated incorrectly after joining a cluster. ##### MQTT Durable Sessions - [#&#8203;14042](https://github.com/emqx/emqx/pull/14042) Fix crash in the durable session after updates to subscription parameters (such as QoS, `no_local`, `upgrade_qos`, ...). - [#&#8203;14052](https://github.com/emqx/emqx/pull/14052) Corrected memory usage reporting from cgroups when in use. - [#&#8203;14055](https://github.com/emqx/emqx/pull/14055) Updated the `/clients_v2` API to properly respect all filtering arguments when querying offline clients with durable sessions. Previously, only the `username` filter was applied, while other filtering arguments were ignored. - [#&#8203;14151](https://github.com/emqx/emqx/pull/14151) Fixed handling of the `conn_state` filter in the `/clients_v2` API for offline clients with durable sessions. Previously, these clients could be incorrectly selected with `conn_state=connected`. - [#&#8203;14057](https://github.com/emqx/emqx/pull/14057) Resolved a compatibility issue that prevented the Messages DS database from starting due to a slightly different database configuration schema. This issue occurred when upgrading EMQX from version 5.7.x with session durability enabled. ##### REST API - [#&#8203;14023](https://github.com/emqx/emqx/pull/14023) Fixed an issue with the `GET /monitor` HTTP API where returned values could appear higher than actual values, depending on the requested time window. For data points within a 1-hour window, this distortion is only visual on the Dashboard. However, for data points older than 1 hour, the data distortion is permanent. The affected metrics include: - `disconnected_durable_sessions` - `subscriptions_durable` - `subscriptions` - `topics` - `connections` - `live_connections` ##### EMQX Clustering - [#&#8203;13996](https://github.com/emqx/emqx/pull/13996) Fixed an intermittent crash occurring when using `emqx conf fix` to resolve configuration discrepancies, particularly if a configuration key was missing on one of the nodes. ##### Security - [#&#8203;13922](https://github.com/emqx/emqx/pull/13922) Updated the CRL (Certificate Revocation List) cache to use the full Distribution Point (DP) URL as the cache key. Previously, only the path part of the URL was used, causing conflicts when multiple DPs shared the same path. - [#&#8203;13924](https://github.com/emqx/emqx/pull/13924) Fixed an issue where JWK keys could leak into debug logs upon JWT authentication failure. ##### Data Integration - [#&#8203;13916](https://github.com/emqx/emqx/pull/13916) Fixed an issue where the parent metric `failed` was not incremented when a rule’s `failed.no_result` or `failed.exception` metrics were updated. - [#&#8203;14001](https://github.com/emqx/emqx/pull/14001) Resolved a race condition where a resource (such as a connector, action, source, authentication, or authorization) could falsely report a connected, healthy channel after a brief disconnection. This issue could result in excessive `action_not_found` log entries when the race condition occurred. - [#&#8203;13913](https://github.com/emqx/emqx/pull/13913) Fixed an issue with the actions and source HTTP APIs where a 500 status code would be returned if a timeout occurred while attempting to update or delete a resource. - [#&#8203;14101](https://github.com/emqx/emqx/pull/14101) Resolved an issue where deleting a resource would fail if a source and an action were both created with the same name. ##### Observability - [#&#8203;13909](https://github.com/emqx/emqx/pull/13909) Fixed log formatting for cases where the payload cannot be displayed as readable UTF-8 Unicode characters. - [#&#8203;14061](https://github.com/emqx/emqx/pull/14061) Improved log information when `emqx_cm:request_stepdown/3` fails. In scenarios where a client channel needs to terminate another channel with the same ClientID, a race condition may occur if the target channel has already been closed or terminated. In such cases, error logs and stack traces that provide no useful information will no longer be generated. - [#&#8203;14070](https://github.com/emqx/emqx/pull/14070) Removed the connector's `state` from error and warning logs due to its potential length. For issue analysis, the connector's state can now be accessed through `emqx_resource:list_instances_verbose/0`. Below is an example of a log entry before this change: pid: <0.43914.0>, connector: connector:sqlserver:connector-05a2e105, reason: [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Argument data type varchar is invalid for argument 2 of dateadd function. SQLSTATE IS: 42000, state: {"resource_opts":{"start_timeout":5000,"start_after_created":true,"health_check_interval":15000},"pool_name":"connector:sqlserver:connector-05a2e105","installed_channels":{"action:sqlserver:action-4b033621:connector:sqlserver:connector-05a2e105":{"sql_templates":{"batch_insert_temp":{"send_message":{"batch_insert_tks":["{str,<<\" ( \">>}","{var,[<<\"messageId\">>]}","{str,<<\", \">>}","{var,[<<\"measurement\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_Fault_1\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_Fault_2\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_Fault_3\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_Fault_4\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_PV_1\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_PV_2\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_PV_3\">>]}","{str,<<\", \">>}","{var,[<<\"Analog_IN_PV_4\">>]}","{str,<<\", DATEADD(MS, \">>}","{var,[<<\"ms_shift\">>]}","{str,<<\", DATEADD(S, \">>}","{var,[<<\"s_shift\">>]}","{str,<<\", '19700101 00:00:00:000') ))\">>}"],"batch_insert_part":"insert into TransactionLog(MessageId, Measurement, Fault1, Fault2, Fault3, Fault4, Value1, Value2, Value3, Value4, DateStamp) \r\n"}}}}}},msg: invalid_request - [#&#8203;14099](https://github.com/emqx/emqx/pull/14099) Removed an error-level log entry that was triggered when validation of UTF-8 strings in MQTT messages failed. Example of the removed log entry: {"time":"2024-10-11T06:05:07.610048+00:00","level":"error","msg":"supervisor: {esockd_connection_sup,0.53591191.0}, errorContext: connection_shutdown, reason: #{cause => invalid_topic,reason => malformed_utf8_string_length}, offender: [{pid,0.53591191.0},...]", ..., "error_logger":{"type":"supervisor_report","tag":"error_report"}} - [#&#8203;14091](https://github.com/emqx/emqx/pull/14091) Implemented a fix to remove `function_clause` from log messages when users provide unsupported write syntax. Example of unsupported syntax: ```bash weather,location=us-midwest,season=summer temperature=82 ${timestamp}u ``` Before this fix, the error log would contain the `function_clause` error, as shown: pid: <0.558392.0>, info: {"stacktrace":["{emqx_bridge_influxdb_connector,parse_timestamp,[[1719350482910000000,<<\"u\">>]],[{file,\"emqx_bridge_influxdb_connector.erl\"},{line,692}]}", ...], ..., "error":"{error,function_clause}"}, tag: ERROR, msg: resource_exception This change improves log clarity by omitting `function_clause` in cases of syntax errors. </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xNy4xIiwidXBkYXRlZEluVmVyIjoiMzkuMTcuMSIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsicmVub3ZhdGUvY29udGFpbmVyIiwidHlwZS9wYXRjaCJdfQ==-->
smeagol-help added 1 commit 2024-11-15 20:03:44 -06:00
jahanson merged commit be3ae69a0a into main 2024-11-16 07:20:27 -06:00
jahanson deleted branch renovate/public.ecr.aws-emqx-emqx-5.x 2024-11-16 07:20:27 -06:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: jahanson/theshire#858
No description provided.