Description:
Hi,
After upgrading Spring Boot 3.4.x, mysql version upgraded to 9.1.0 from 8.3.0.
In mysql 9.1.0, OpenTelemetry features comes in mysql connector/j library:
https://dev.mysql.com/doc/connector-j/en/connector-j-opentelemetry.html
We use OpenTelemetry to observe our application, and the default openTelemetry prop for mysql is PREFERRED, and mysql connector/j OpenTelemetry features become available due to usage of OpenTelemetry in our applications.
When we heap dump any application that have OpenTelemetry, we found a memory leak in com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.ConnectionFinalizerPhantomReference
When we examine this class objects, we found the root of the leak:
ConnectionFinalizerPhantomReference -> (com.mysql.cj.jdbc.ConnectionImpl) referent -> (com.mysql.cj.NativeSession) session -> (com.mysql.cj.otel.OpenTelemetryHandler) telemetryHandler -> (java.utilWeakHashMap) spans
This spans map never emptied and only used to store com.mysql.cj.telemetry.TelemetrySpan - io.opentelemetry.api.trace.Span relation. The only thing I saw, there is a linkTargest list object and spans map only used to add or remove from that list.
Therefore, the spans map becomes bigger and bigger when time passes. This causes huge memory usages by this map in our applicatons. Somehow the garbage collector did not clean this object.
The linkTargest list object does not have so much area in heap. I think that is because of the algortim when closing connections. You can see that behavior in com.mysql.cj.jdbc.ConnectionImpl#doClose method via calling this.session.getTelemetryHandler().removeLinkTarget(this.connectionSpan).
This kinda empty the list object but the spans map remains same although the connection closed.
How to repeat:
1. Use Mysql 9.1.0 for mysql-connector-j library.
2. Make sure io.opentelemetry.api.GlobalOpenTelemetry class exist in your application.
3. Make sure com.mysql.cj.otel.OpenTelemetryHandler initialized when new session created via com.mysql.cj.NativeSession#NativeSession
4. Just run the program.
5. Wait enough to see the growing size of spans map. It may vary on your application request count.
Suggested fix:
I suggest when we closing ConnectionImpl, the linkTargets are removing via
this.session.getTelemetryHandler().removeLinkTarget(this.connectionSpan)
After that, we can remove this connectionSpan from spans too. In other words, the removing link target logic is:
public void removeLinkTarget(TelemetrySpan span) {
Span otelSpan = (Span)this.spans.get(span);
if (otelSpan != null) {
this.linkTargets.remove(otelSpan);
}
why not we remove otelSpan from spans map?
© 2025, Oracle Corporation and/or its affiliates