MySQL Bugs: #112412: Improve open telemetry documentation in MySQL

Bug #112412	Improve open telemetry documentation in MySQL
Submitted:	21 Sep 10:09	Modified:	11 Oct 9:52
Reporter:	Simon Mudd (OCA)	Email Updates:
Status:	Verified	Impact on me:	None
Category:	MySQL Server: Documentation	Severity:	S3 (Non-critical)
Version:	8.0	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
Related to bug#111395 8.0.33 brought some changes related to open_telemtry and the TELEMETRY_LOG_ADMIN. Privilege which at the time was not documented.

I notice other references to open_telemetry:

https://dev.mysql.com/doc/connector-python/en/connector-python-opentelemetry.html
https://dev.mysql.com/doc/refman/8.1/en/telemetry-trace.html

However, they seem incomplete.

I also notice there is in server code a components/test_server_telemetry_traces/server_telemetry_component.cc file which provides a test server related to telemetry.

Tracing and open_telemetry are really interesting things which a lot of users may find very useful. While it seems there's no public tracing component it would be good:
- to document the interface the component should satisfy
- provide some information examples of what telemetry attributes the server can currently provide
- provide an example in other supported languages such as C connectors/libraries of how this can be used
- basically provide enough high level information on how this might be used and what information can be provided without having to read the code.

Providing such information would no doubt spark an interest from users and potentially trigger some people to consider building a plugin for either internal usage or an open source version which could then be used by others.

The "facility is there" but it's pretty much hidden and exposing the capabilities it provides would be very good. So while bug#111395 implied this may have been written for a specific use case (MySQL heatwave for AWS) the general interest in something like this would grow if more people were aware of it.

How to repeat:
read the source and follow links mentioned above.

Suggested fix:
Improve documentation as indicated above.

- optionally provide a generic opensource telemetry component so that language drivers can plug into this if installed and use it by default. Such a component would only be expected to perhaps write out to a local socket or file, but such code could then be extended for more specific use cases if needed. If the current test component can already do this make it part of the general server code distributed with normal production binaries.

Hi Mr. Mudd,

Thank you very much for your Documentation bug.

We have analysed your report and we consider that it is correct.

This report is now a verified bug for 8.0 and higher versions.

Hi Simon, can you please file a separate bug/request for Connector/Python?

Posted by developer:
 
Hi Simon,
the telemetry trace service was announced in 8.0.33: https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-33.html#mysqld-8-0-33-performance...
and the service is documented here:https://dev.mysql.com/doc/dev/mysql-server/latest/PAGE_MYSQL_SERVER_TELEMETRY_TRACES_SERVI...

Please let us know if this information answers your questions.
Thanks.

Thanks for the pointers. Currently I'm somewhat confused at how to use this from the SQL `mysql` command line. I see there's some information on the span information that can be provided but it's not clear how to invoke this for example from the MySQL client using SQL, or for example from the any library that might be using the MySQL C libraries. e.g. Perl uses DBD::mysql which compiles against the MySQL C libraries so it would be good to know what changes are required to use the  tracing functionality in such cases. I think I've seen an example with the python connector but am not too sure how for example java connectors would be adjusted to use this facility.

Specifically I'd expect that traces may be sent for "some" queries and not others and ideally I'd like to be able to influence efficiently on a per-statement basis so it would be good to understand how this can be done.

The reasons for such behaviour is the amount of load we'd put mysqld under if it were to trace all queries would be too high and often tracing wants to follow a "random span" from the originating query all the way through the call tree to the bottom, thus only a small percentage, determined by the application of queries would expect to be traced and the app would need to be able to handle that.

While I looked briefly at the docs they did not really seem to cover some of the points mentioned here.

Hi Mr. Mudd,

We agree with you and we are reverting the status to "Verified".

Hi Mr. Mudd,

For the time being, this is a documentation bug.

However, if documentation is appended to your requirements, a feature request could be added as a separate bug report.

Hi Mr. Mudd,

We were looking further on how to help you and we concluded that, in absence of more documentation, it might be useful to take a look at the test suite, most of all these ones:

----------------------------------------------------------------
./mysql-test/include/have_no_telemetry_component.inc
./mysql-test/include/mtr_telemetry.sql
./mysql-test/include/have_server_telemetry_traces.inc
./mysql-test/suite/perfschema/r/telemetry_traces_test_client.result
./mysql-test/suite/perfschema/r/telemetry_traces_server.result
./mysql-test/suite/perfschema/t/telemetry_traces_server-master.opt
./mysql-test/suite/perfschema/t/telemetry_traces_test_client.test
./mysql-test/suite/perfschema/t/telemetry_traces_server.test
./mysql-test/suite/perfschema/t/telemetry_traces_test_client-master.opt
./mysql-test/suite/group_replication/r/telemetry_traces_server.result
./mysql-test/suite/group_replication/t/telemetry_traces_server.test
./mysql-test/suite/group_replication/t/telemetry_traces_server-slave.opt
./mysql-test/suite/secondary_engine/r/telemetry_traces_server.result
./mysql-test/suite/secondary_engine/t/telemetry_traces_server-master.opt
./mysql-test/suite/secondary_engine/t/telemetry_traces_server.test
./mysql-test/suite/clone/r/telemetry_traces_server.result
./mysql-test/suite/clone/t/telemetry_traces_server-master.opt
./mysql-test/suite/clone/t/telemetry_traces_server.test
./mysql-test/suite/query_rewrite_plugins/r/telemetry_traces_server.result
./mysql-test/suite/query_rewrite_plugins/t/telemetry_traces_server-master.opt
./mysql-test/suite/query_rewrite_plugins/t/telemetry_traces_server.test
./mysql-test/suite/rpl/r/telemetry_traces_server.result
./mysql-test/suite/rpl/t/telemetry_traces_server.test
./mysql-test/suite/rpl/t/telemetry_traces_server-slave.opt
./mysql-test/suite/federated/r/telemetry_traces_server.result
./mysql-test/suite/federated/t/telemetry_traces_server-master.opt
./mysql-test/suite/federated/t/telemetry_traces_server.test
----------------------------------------------------------------

Please, pay special attention to the example files in the t/ and r/ subdirectories.

Do note that there are some separate client tests. There are also separate tests for replication, group replication, plugins, installed clone etc .......

These examples are , actually, quite revealing  ....