Description:
With openTelemetry=PREFERRED (the default) and an OpenTelemetry instance registered in GlobalOpenTelemetry, NativeMessageBuilder.buildComQuery() injects the trace context through:
propagateContext(queryAttributesBindings::setAttribute)
This adds every key that the configured TextMapPropagator emits to the statement's persistent query attributes. After sending, only getContextPropagationKey(), i.e. "traceparent", is removed. Every other key stays behind:
- "tracestate" from the W3C trace-context propagator;
- "baggage" from a W3C + baggage composite;
- vendor keys such as x-datadog-*.
On the next execution "traceparent" is missing again, so the context is injected again and the leftover keys are appended once more.
Result: on a PreparedStatement that is executed repeatedly (a loop, executeBatch() without rewriteBatchedStatements, or a cached statement), the attribute list grows by one set per execution. The bytes sent therefore grow quadratically.
Once the list passes 65,535 attributes, the server rejects the query with:
SQLException: Malformed communication packet. (error 1835, SQLState HY000)
Real-world case: a batch of ~100k single-row UPDATEs on one PreparedStatement, with the Datadog Java tracer's OpenTelemetry bridge (5 leaked keys per execution), sent ~21.8 GB in 4 minutes and failed after ~13,000 rows. A matching report with a packet capture: https://github.com/DataDog/dd-trace-java/issues/7660
Workaround: openTelemetry=DISABLED.
Related:
- Bug #116257: the same class of problem in Connector/NET. Trace attributes are re-added to a re-executed command, which crashes there.
- Bug #112058: also error 1835, but it does not apply here. The server supports CLIENT_QUERY_ATTRIBUTES, and the error only appears once the leaked attributes pass 65,535.
How to repeat:
Requirements: opentelemetry-api, opentelemetry-context and opentelemetry-common on the classpath (tested with 1.62.0), and a table created with CREATE TABLE t (id INT PRIMARY KEY, v INT); INSERT INTO t VALUES (1, 0);
// Any propagator that emits more than "traceparent", e.g. W3C trace context with a non-empty tracestate
GlobalOpenTelemetry.set(OpenTelemetry.propagating(ContextPropagators.create(new TextMapPropagator() {
public Collection<String> fields() { return List.of("traceparent", "tracestate"); }
public <C> void inject(Context ctx, C carrier, TextMapSetter<C> setter) {
setter.set(carrier, "traceparent", "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01");
setter.set(carrier, "tracestate", "vendor=opaque-value");
}
public <C> Context extract(Context ctx, C carrier, TextMapGetter<C> getter) { return ctx; }
})));
try (Connection c = DriverManager.getConnection(url, user, password);
PreparedStatement ps = c.prepareStatement("UPDATE t SET v = ? WHERE id = 1")) {
for (int i = 0; i < 5000; i++) { ps.setInt(1, i); ps.executeUpdate(); }
}
Compare SHOW SESSION STATUS LIKE 'Bytes_received' per 1,000 executions. The complete program is attached as QueryAttributeLeak.java.
Bytes received by the server per 1,000 executions, on Connector/J 26.7.0 against server 8.4.8:
| Executions | openTelemetry=PREFERRED (default) | openTelemetry=DISABLED |
|---|---|---|
| 1-1,000 | 16,698,170 | 45,938 |
| 1,001-2,000 | 49,823,668 | 45,938 |
| 2,001-3,000 | 82,948,668 | 45,938 |
| 3,001-4,000 | 116,073,668 | 45,938 |
| 4,001-5,000 | 149,198,668 | 45,938 |
Expected: a constant size per execution in both modes.
Actual: one extra "tracestate" attribute per execution, re-sent on every later execution. After 65,536 executions the query fails with error 1835.
Suggested fix:
Remove every key that propagateContext() injected, not only "traceparent". Or inject into a temporary list that is written to the packet but never stored in the statement's QueryAttributesBindings; the no-attributes branch of buildComQuery already uses a throwaway BindValue this way. Alternatively, inject only "traceparent": the callback parameter in OpenTelemetryHandler.propagateContext(BiConsumer<String, String> traceparentConsumer) suggests that was the intent.
Description: With openTelemetry=PREFERRED (the default) and an OpenTelemetry instance registered in GlobalOpenTelemetry, NativeMessageBuilder.buildComQuery() injects the trace context through: propagateContext(queryAttributesBindings::setAttribute) This adds every key that the configured TextMapPropagator emits to the statement's persistent query attributes. After sending, only getContextPropagationKey(), i.e. "traceparent", is removed. Every other key stays behind: - "tracestate" from the W3C trace-context propagator; - "baggage" from a W3C + baggage composite; - vendor keys such as x-datadog-*. On the next execution "traceparent" is missing again, so the context is injected again and the leftover keys are appended once more. Result: on a PreparedStatement that is executed repeatedly (a loop, executeBatch() without rewriteBatchedStatements, or a cached statement), the attribute list grows by one set per execution. The bytes sent therefore grow quadratically. Once the list passes 65,535 attributes, the server rejects the query with: SQLException: Malformed communication packet. (error 1835, SQLState HY000) Real-world case: a batch of ~100k single-row UPDATEs on one PreparedStatement, with the Datadog Java tracer's OpenTelemetry bridge (5 leaked keys per execution), sent ~21.8 GB in 4 minutes and failed after ~13,000 rows. A matching report with a packet capture: https://github.com/DataDog/dd-trace-java/issues/7660 Workaround: openTelemetry=DISABLED. Related: - Bug #116257: the same class of problem in Connector/NET. Trace attributes are re-added to a re-executed command, which crashes there. - Bug #112058: also error 1835, but it does not apply here. The server supports CLIENT_QUERY_ATTRIBUTES, and the error only appears once the leaked attributes pass 65,535. How to repeat: Requirements: opentelemetry-api, opentelemetry-context and opentelemetry-common on the classpath (tested with 1.62.0), and a table created with CREATE TABLE t (id INT PRIMARY KEY, v INT); INSERT INTO t VALUES (1, 0); // Any propagator that emits more than "traceparent", e.g. W3C trace context with a non-empty tracestate GlobalOpenTelemetry.set(OpenTelemetry.propagating(ContextPropagators.create(new TextMapPropagator() { public Collection<String> fields() { return List.of("traceparent", "tracestate"); } public <C> void inject(Context ctx, C carrier, TextMapSetter<C> setter) { setter.set(carrier, "traceparent", "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"); setter.set(carrier, "tracestate", "vendor=opaque-value"); } public <C> Context extract(Context ctx, C carrier, TextMapGetter<C> getter) { return ctx; } }))); try (Connection c = DriverManager.getConnection(url, user, password); PreparedStatement ps = c.prepareStatement("UPDATE t SET v = ? WHERE id = 1")) { for (int i = 0; i < 5000; i++) { ps.setInt(1, i); ps.executeUpdate(); } } Compare SHOW SESSION STATUS LIKE 'Bytes_received' per 1,000 executions. The complete program is attached as QueryAttributeLeak.java. Bytes received by the server per 1,000 executions, on Connector/J 26.7.0 against server 8.4.8: | Executions | openTelemetry=PREFERRED (default) | openTelemetry=DISABLED | |---|---|---| | 1-1,000 | 16,698,170 | 45,938 | | 1,001-2,000 | 49,823,668 | 45,938 | | 2,001-3,000 | 82,948,668 | 45,938 | | 3,001-4,000 | 116,073,668 | 45,938 | | 4,001-5,000 | 149,198,668 | 45,938 | Expected: a constant size per execution in both modes. Actual: one extra "tracestate" attribute per execution, re-sent on every later execution. After 65,536 executions the query fails with error 1835. Suggested fix: Remove every key that propagateContext() injected, not only "traceparent". Or inject into a temporary list that is written to the packet but never stored in the statement's QueryAttributesBindings; the no-attributes branch of buildComQuery already uses a throwaway BindValue this way. Alternatively, inject only "traceparent": the callback parameter in OpenTelemetryHandler.propagateContext(BiConsumer<String, String> traceparentConsumer) suggests that was the intent.