Bug #118528 Connector/J Memory Leak Problem

• Submitted: 26 Jun 8:43
• Modified: 8 Jul 19:55
• Reporter: ünal polat
• Status: Need Feedback
• Category: Connector / J
• Severity: S3 (Non-critical)
• Version: 9.1.0
• OS: Any
• Assigned to: MySQL Verification Team
• CPU Architecture: Any
• Tags: Connector/J, memory-leak

[26 Jun 8:43] ünal polat

Description:
Hi,
After upgrading Spring Boot 3.4.x, mysql version upgraded to 9.1.0 from 8.3.0.

In mysql 9.1.0, OpenTelemetry features comes in mysql connector/j library:
https://dev.mysql.com/doc/connector-j/en/connector-j-opentelemetry.html

We use OpenTelemetry to observe our application, and the default openTelemetry prop for mysql is PREFERRED, and mysql connector/j OpenTelemetry features become available due to usage of OpenTelemetry in our applications.

When we heap dump any application that have OpenTelemetry, we found a memory leak in com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.ConnectionFinalizerPhantomReference

When we examine this class objects, we found the root of the leak:

ConnectionFinalizerPhantomReference -> (com.mysql.cj.jdbc.ConnectionImpl) referent -> (com.mysql.cj.NativeSession) session -> (com.mysql.cj.otel.OpenTelemetryHandler) telemetryHandler -> (java.utilWeakHashMap) spans

This spans map never emptied and only used to store com.mysql.cj.telemetry.TelemetrySpan - io.opentelemetry.api.trace.Span relation. The only thing I saw, there is a linkTargest list object and spans map only used to add or remove from that list.

Therefore, the spans map becomes bigger and bigger when time passes. This causes huge memory usages by this map in our applicatons. Somehow the garbage collector did not clean this object.

The linkTargest list object does not have so much area in heap. I think that is because of the algortim when closing connections. You can see that behavior in com.mysql.cj.jdbc.ConnectionImpl#doClose method via calling this.session.getTelemetryHandler().removeLinkTarget(this.connectionSpan).

This kinda empty the list object but the spans map remains same although the connection closed.

How to repeat:
1. Use Mysql 9.1.0 for mysql-connector-j library.
2. Make sure io.opentelemetry.api.GlobalOpenTelemetry class exist in your application.
3. Make sure com.mysql.cj.otel.OpenTelemetryHandler initialized when new session created via com.mysql.cj.NativeSession#NativeSession
4. Just run the program.
5. Wait enough to see the growing size of spans map. It may vary on your application request count. 

Suggested fix:
I suggest when we closing ConnectionImpl, the linkTargets are removing via

this.session.getTelemetryHandler().removeLinkTarget(this.connectionSpan)

After that, we can remove this connectionSpan from spans too. In other words, the removing link target logic is:

public void removeLinkTarget(TelemetrySpan span) {
    Span otelSpan = (Span)this.spans.get(span);
    if (otelSpan != null) {
      this.linkTargets.remove(otelSpan);
    }

why not we remove otelSpan from spans map?

[8 Jul 19:55] Filipe Silva

Hi Ünal Polat,

Thank you for your interest in MySQL Connector/J and for taking the time to report this issue.

As it stands, the current implementation doesn't require the spans map to be explicitly cleared. The use of a WeakHashMap in combination with the garbage collector is intended to handle this automatically.

The fix you suggested doesn’t have a meaningful effect, as it would only remove a single entry from the map—the one corresponding to the span promoted as a link target (i.e., the connection creation span). All other spans are short-lived and, once the operation is complete, no strong references remain. As such, they should be eligible for garbage collection and automatically removed from the map.

Are you certain that the garbage collector is not cleaning up the spans map? In my observations, this behavior appears to be working as expected.

Would you be able to provide a minimal test case that reproduces the issue without involving Spring Boot? Ideally, it would include only Connector/J, OpenTelemetry, and a bit of test code to help isolate the behavior. Also, please use the latest MySQL Connector/J version.