Bug #121045 Server telemetry metrics (mysql.stats, mysql.stats.handler) always report 0
Submitted: 31 Jul 23:14
Reporter: Hunter Kang Email Updates:
Status: Open Impact on me:
None 
Category:MySQL Server: Performance Schema Severity:S3 (Non-critical)
Version:9.7.2, 8.4 (present since 8.2.0) OS:Any
Assigned to: CPU Architecture:Any

[31 Jul 23:14] Hunter Kang
Description:
26 server telemetry (OTEL) metrics under the mysql.stats and
mysql.stats.handler meters always export 0, even under load, while the
matching SHOW GLOBAL STATUS counter reports real activity.

These metrics are read by an asynchronous metric callback
(get_metric_aggregated_integer in sql/mysqld.cc) from a sharded, per-THD
counter buffer (aggregated_stats_buffer), aggregated across shards by
aggregated_stats::get_single_total. Counters that work write their shard
next to the per-THD status_var update, e.g. in sql/sql_parse.cc:

    global_aggregated_stats.get_shard(thd->thread_id()).questions++;

For these 26 metrics the producers write only the per-THD status_var and
never the shard the reader sums, so the exported value is a constant 0.

Two producer paths are affected:
 - THD::inc_status_*() in sql/sql_class.cc for the Select_*, Sort_*, and
   Created_tmp_* family, plus long_query_count (sql/log.cc) and
   max_execution_time_exceeded (sql/sql_class.cc).
 - handler::ha_statistic_increment() in sql/handler.cc for the Handler_*
   read/write family, reached from every storage engine. It updates only
   the per-THD field:

    void handler::ha_statistic_increment(
        ulonglong System_status_var::*offset) const {
      if (table && table->in_use) (table->in_use->status_var.*offset)++;
    }

Affected metrics (26), by family:
 - Handler (12): read_first, read_key, read_last, read_next, read_prev,
   read_rnd, read_rnd_next, write, update, delete, external_lock,
   multi_range_read_init
 - Select (5): select_scan, select_full_join, select_full_range_join,
   select_range, select_range_check
 - Sort (4): sort_rows, sort_scan, sort_range, sort_merge_passes
 - Created tmp (3): created.tmp_tables, created.tmp_disk_tables,
   count_hit_tmp_table_size
 - Other (2): slow_queries (long_query_count), max_execution_time_exceeded

The transaction-family Handler counters (ha_commit, ha_rollback,
ha_prepare, ha_savepoint) and the table_open_cache_* counters do write
their shards and are unaffected.

The defect was introduced with the server telemetry metrics interface
(the sharded aggregated_stats_buffer) in 8.2.0: the shard field, its
reset, its cross-shard aggregation, and its metric binding were added for
these 26, but the producer-side shard write was omitted. It has been
present on every release since.

There is no SQL path to these values (performance_schema.setup_metrics
lists metric names but has no value column), so the zeros are only
observable through a component consuming the
mysql_server_telemetry_metrics_v1 service.

Impact: any OTEL-based monitoring, dashboards, or alerting on these 26
metrics reads a constant 0 regardless of real workload -- blind to full
scans, sorts, temporary-table creation, and handler read/write activity.
No error is raised.

How to repeat:
Reproduced on a 9.7.2 debug build; the same code is present in 8.4.

The metric values require a component that consumes the
mysql_server_telemetry_metrics_v1 service. The bundled test component
components/test_server_telemetry_metrics exposes a UDF,
test_report_single_metric(meter, metric), that returns the current value
of a metric (this component is only present in a debug/source build).

1. Build the server with the test telemetry metrics component and start
   it.

2. Install the component and enable all meters:

     INSTALL COMPONENT 'file://component_test_server_telemetry_metrics';
     UPDATE performance_schema.setup_meters SET enabled = 'YES';

3. Generate activity that increments the counters, e.g. full table scans:

     CREATE TABLE t1 (a INT, b INT);
     INSERT INTO t1 VALUES (1,1),(2,2),(3,3),(4,4),(5,5),(6,6),(7,7),(8,8);
     FLUSH STATUS;
     SELECT * FROM t1 WHERE b > 0;
     SELECT * FROM t1 WHERE b <> 42;

4. Compare the status variable (correct) with the telemetry metric (0):

     SHOW GLOBAL STATUS LIKE 'Select_scan';
     -- e.g. Value = 40

     SELECT test_report_single_metric('mysql.stats', 'select_scan');
     -- returns 0

     SHOW GLOBAL STATUS LIKE 'Handler_read_rnd_next';
     -- e.g. Value = 11346

     SELECT test_report_single_metric('mysql.stats.handler',
                                       'read_rnd_next');
     -- returns 0

The SHOW GLOBAL STATUS counters reflect the activity; the telemetry
metrics stay at 0.

Suggested fix:
Write the shard next to each existing per-THD status_var update, matching
the pattern already used by the counters that work
(global_aggregated_stats.get_shard(thd->thread_id()).<field>).

For the Handler family, handler::ha_statistic_increment() receives a
ulonglong System_status_var::* member pointer that cannot be applied to
aggregated_stats_buffer (different field order, and the shard fields are
std::atomic_uint64_t). The helper can be extended to also take the
matching aggregated_stats_buffer member pointer so both the per-THD and
shard counters are updated.