Bug #121552 OpenTelemetry context propagation leaks query attributes into reused statements, causing 1835 errors
Submitted: 19 Oct 15:13
Reporter: Vincent Botteman Email Updates:
Status: Open Impact on me:
None 
Category:Connector / J Severity:S3 (Non-critical)
Version:26.7.0 (also 9.5.0, 9.6.0, 9.7.0) OS:Any (reproduced on Windows 11 / Java 25; seen on Linux / Java 25)
Assigned to: CPU Architecture:Any

[19 Oct 15:13] Vincent Botteman
Description:
With openTelemetry=PREFERRED (the default) and an OpenTelemetry instance registered in GlobalOpenTelemetry, NativeMessageBuilder.buildComQuery() injects the trace context through:

    propagateContext(queryAttributesBindings::setAttribute)

This adds every key that the configured TextMapPropagator emits to the statement's persistent query attributes. After sending, only getContextPropagationKey(), i.e. "traceparent", is removed. Every other key stays behind:
- "tracestate" from the W3C trace-context propagator;
- "baggage" from a W3C + baggage composite;
- vendor keys such as x-datadog-*.

On the next execution "traceparent" is missing again, so the context is injected again and the leftover keys are appended once more.

Result: on a PreparedStatement that is executed repeatedly (a loop, executeBatch() without rewriteBatchedStatements, or a cached statement), the attribute list grows by one set per execution. The bytes sent therefore grow quadratically.

Once the list passes 65,535 attributes, the server rejects the query with:

    SQLException: Malformed communication packet. (error 1835, SQLState HY000)

Real-world case: a batch of ~100k single-row UPDATEs on one PreparedStatement, with the Datadog Java tracer's OpenTelemetry bridge (5 leaked keys per execution), sent ~21.8 GB in 4 minutes and failed after ~13,000 rows. A matching report with a packet capture: https://github.com/DataDog/dd-trace-java/issues/7660

Workaround: openTelemetry=DISABLED.

Related:
- Bug #116257: the same class of problem in Connector/NET. Trace attributes are re-added to a re-executed command, which crashes there.
- Bug #112058: also error 1835, but it does not apply here. The server supports CLIENT_QUERY_ATTRIBUTES, and the error only appears once the leaked attributes pass 65,535.

How to repeat:
Requirements: opentelemetry-api, opentelemetry-context and opentelemetry-common on the classpath (tested with 1.62.0), and a table created with CREATE TABLE t (id INT PRIMARY KEY, v INT); INSERT INTO t VALUES (1, 0);

    // Any propagator that emits more than "traceparent", e.g. W3C trace context with a non-empty tracestate
    GlobalOpenTelemetry.set(OpenTelemetry.propagating(ContextPropagators.create(new TextMapPropagator() {
        public Collection<String> fields() { return List.of("traceparent", "tracestate"); }
        public <C> void inject(Context ctx, C carrier, TextMapSetter<C> setter) {
            setter.set(carrier, "traceparent", "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01");
            setter.set(carrier, "tracestate", "vendor=opaque-value");
        }
        public <C> Context extract(Context ctx, C carrier, TextMapGetter<C> getter) { return ctx; }
    })));
    try (Connection c = DriverManager.getConnection(url, user, password);
         PreparedStatement ps = c.prepareStatement("UPDATE t SET v = ? WHERE id = 1")) {
        for (int i = 0; i < 5000; i++) { ps.setInt(1, i); ps.executeUpdate(); }
    }

Compare SHOW SESSION STATUS LIKE 'Bytes_received' per 1,000 executions. The complete program is attached as QueryAttributeLeak.java.

Bytes received by the server per 1,000 executions, on Connector/J 26.7.0 against server 8.4.8:

| Executions | openTelemetry=PREFERRED (default) | openTelemetry=DISABLED |
|---|---|---|
| 1-1,000 | 16,698,170 | 45,938 |
| 1,001-2,000 | 49,823,668 | 45,938 |
| 2,001-3,000 | 82,948,668 | 45,938 |
| 3,001-4,000 | 116,073,668 | 45,938 |
| 4,001-5,000 | 149,198,668 | 45,938 |

Expected: a constant size per execution in both modes.
Actual: one extra "tracestate" attribute per execution, re-sent on every later execution. After 65,536 executions the query fails with error 1835.

Suggested fix:
Remove every key that propagateContext() injected, not only "traceparent". Or inject into a temporary list that is written to the packet but never stored in the statement's QueryAttributesBindings; the no-attributes branch of buildComQuery already uses a throwaway BindValue this way. Alternatively, inject only "traceparent": the callback parameter in OpenTelemetryHandler.propagateContext(BiConsumer<String, String> traceparentConsumer) suggests that was the intent.