Description:
26 server telemetry (OTEL) metrics under the mysql.stats and
mysql.stats.handler meters always export 0, even under load, while the
matching SHOW GLOBAL STATUS counter reports real activity.
These metrics are read by an asynchronous metric callback
(get_metric_aggregated_integer in sql/mysqld.cc) from a sharded, per-THD
counter buffer (aggregated_stats_buffer), aggregated across shards by
aggregated_stats::get_single_total. Counters that work write their shard
next to the per-THD status_var update, e.g. in sql/sql_parse.cc:
global_aggregated_stats.get_shard(thd->thread_id()).questions++;
For these 26 metrics the producers write only the per-THD status_var and
never the shard the reader sums, so the exported value is a constant 0.
Two producer paths are affected:
- THD::inc_status_*() in sql/sql_class.cc for the Select_*, Sort_*, and
Created_tmp_* family, plus long_query_count (sql/log.cc) and
max_execution_time_exceeded (sql/sql_class.cc).
- handler::ha_statistic_increment() in sql/handler.cc for the Handler_*
read/write family, reached from every storage engine. It updates only
the per-THD field:
void handler::ha_statistic_increment(
ulonglong System_status_var::*offset) const {
if (table && table->in_use) (table->in_use->status_var.*offset)++;
}
Affected metrics (26), by family:
- Handler (12): read_first, read_key, read_last, read_next, read_prev,
read_rnd, read_rnd_next, write, update, delete, external_lock,
multi_range_read_init
- Select (5): select_scan, select_full_join, select_full_range_join,
select_range, select_range_check
- Sort (4): sort_rows, sort_scan, sort_range, sort_merge_passes
- Created tmp (3): created.tmp_tables, created.tmp_disk_tables,
count_hit_tmp_table_size
- Other (2): slow_queries (long_query_count), max_execution_time_exceeded
The transaction-family Handler counters (ha_commit, ha_rollback,
ha_prepare, ha_savepoint) and the table_open_cache_* counters do write
their shards and are unaffected.
The defect was introduced with the server telemetry metrics interface
(the sharded aggregated_stats_buffer) in 8.2.0: the shard field, its
reset, its cross-shard aggregation, and its metric binding were added for
these 26, but the producer-side shard write was omitted. It has been
present on every release since.
There is no SQL path to these values (performance_schema.setup_metrics
lists metric names but has no value column), so the zeros are only
observable through a component consuming the
mysql_server_telemetry_metrics_v1 service.
Impact: any OTEL-based monitoring, dashboards, or alerting on these 26
metrics reads a constant 0 regardless of real workload -- blind to full
scans, sorts, temporary-table creation, and handler read/write activity.
No error is raised.
How to repeat:
Reproduced on a 9.7.2 debug build; the same code is present in 8.4.
The metric values require a component that consumes the
mysql_server_telemetry_metrics_v1 service. The bundled test component
components/test_server_telemetry_metrics exposes a UDF,
test_report_single_metric(meter, metric), that returns the current value
of a metric (this component is only present in a debug/source build).
1. Build the server with the test telemetry metrics component and start
it.
2. Install the component and enable all meters:
INSTALL COMPONENT 'file://component_test_server_telemetry_metrics';
UPDATE performance_schema.setup_meters SET enabled = 'YES';
3. Generate activity that increments the counters, e.g. full table scans:
CREATE TABLE t1 (a INT, b INT);
INSERT INTO t1 VALUES (1,1),(2,2),(3,3),(4,4),(5,5),(6,6),(7,7),(8,8);
FLUSH STATUS;
SELECT * FROM t1 WHERE b > 0;
SELECT * FROM t1 WHERE b <> 42;
4. Compare the status variable (correct) with the telemetry metric (0):
SHOW GLOBAL STATUS LIKE 'Select_scan';
-- e.g. Value = 40
SELECT test_report_single_metric('mysql.stats', 'select_scan');
-- returns 0
SHOW GLOBAL STATUS LIKE 'Handler_read_rnd_next';
-- e.g. Value = 11346
SELECT test_report_single_metric('mysql.stats.handler',
'read_rnd_next');
-- returns 0
The SHOW GLOBAL STATUS counters reflect the activity; the telemetry
metrics stay at 0.
Suggested fix:
Write the shard next to each existing per-THD status_var update, matching
the pattern already used by the counters that work
(global_aggregated_stats.get_shard(thd->thread_id()).<field>).
For the Handler family, handler::ha_statistic_increment() receives a
ulonglong System_status_var::* member pointer that cannot be applied to
aggregated_stats_buffer (different field order, and the shard fields are
std::atomic_uint64_t). The helper can be extended to also take the
matching aggregated_stats_buffer member pointer so both the per-THD and
shard counters are updated.
Description: 26 server telemetry (OTEL) metrics under the mysql.stats and mysql.stats.handler meters always export 0, even under load, while the matching SHOW GLOBAL STATUS counter reports real activity. These metrics are read by an asynchronous metric callback (get_metric_aggregated_integer in sql/mysqld.cc) from a sharded, per-THD counter buffer (aggregated_stats_buffer), aggregated across shards by aggregated_stats::get_single_total. Counters that work write their shard next to the per-THD status_var update, e.g. in sql/sql_parse.cc: global_aggregated_stats.get_shard(thd->thread_id()).questions++; For these 26 metrics the producers write only the per-THD status_var and never the shard the reader sums, so the exported value is a constant 0. Two producer paths are affected: - THD::inc_status_*() in sql/sql_class.cc for the Select_*, Sort_*, and Created_tmp_* family, plus long_query_count (sql/log.cc) and max_execution_time_exceeded (sql/sql_class.cc). - handler::ha_statistic_increment() in sql/handler.cc for the Handler_* read/write family, reached from every storage engine. It updates only the per-THD field: void handler::ha_statistic_increment( ulonglong System_status_var::*offset) const { if (table && table->in_use) (table->in_use->status_var.*offset)++; } Affected metrics (26), by family: - Handler (12): read_first, read_key, read_last, read_next, read_prev, read_rnd, read_rnd_next, write, update, delete, external_lock, multi_range_read_init - Select (5): select_scan, select_full_join, select_full_range_join, select_range, select_range_check - Sort (4): sort_rows, sort_scan, sort_range, sort_merge_passes - Created tmp (3): created.tmp_tables, created.tmp_disk_tables, count_hit_tmp_table_size - Other (2): slow_queries (long_query_count), max_execution_time_exceeded The transaction-family Handler counters (ha_commit, ha_rollback, ha_prepare, ha_savepoint) and the table_open_cache_* counters do write their shards and are unaffected. The defect was introduced with the server telemetry metrics interface (the sharded aggregated_stats_buffer) in 8.2.0: the shard field, its reset, its cross-shard aggregation, and its metric binding were added for these 26, but the producer-side shard write was omitted. It has been present on every release since. There is no SQL path to these values (performance_schema.setup_metrics lists metric names but has no value column), so the zeros are only observable through a component consuming the mysql_server_telemetry_metrics_v1 service. Impact: any OTEL-based monitoring, dashboards, or alerting on these 26 metrics reads a constant 0 regardless of real workload -- blind to full scans, sorts, temporary-table creation, and handler read/write activity. No error is raised. How to repeat: Reproduced on a 9.7.2 debug build; the same code is present in 8.4. The metric values require a component that consumes the mysql_server_telemetry_metrics_v1 service. The bundled test component components/test_server_telemetry_metrics exposes a UDF, test_report_single_metric(meter, metric), that returns the current value of a metric (this component is only present in a debug/source build). 1. Build the server with the test telemetry metrics component and start it. 2. Install the component and enable all meters: INSTALL COMPONENT 'file://component_test_server_telemetry_metrics'; UPDATE performance_schema.setup_meters SET enabled = 'YES'; 3. Generate activity that increments the counters, e.g. full table scans: CREATE TABLE t1 (a INT, b INT); INSERT INTO t1 VALUES (1,1),(2,2),(3,3),(4,4),(5,5),(6,6),(7,7),(8,8); FLUSH STATUS; SELECT * FROM t1 WHERE b > 0; SELECT * FROM t1 WHERE b <> 42; 4. Compare the status variable (correct) with the telemetry metric (0): SHOW GLOBAL STATUS LIKE 'Select_scan'; -- e.g. Value = 40 SELECT test_report_single_metric('mysql.stats', 'select_scan'); -- returns 0 SHOW GLOBAL STATUS LIKE 'Handler_read_rnd_next'; -- e.g. Value = 11346 SELECT test_report_single_metric('mysql.stats.handler', 'read_rnd_next'); -- returns 0 The SHOW GLOBAL STATUS counters reflect the activity; the telemetry metrics stay at 0. Suggested fix: Write the shard next to each existing per-THD status_var update, matching the pattern already used by the counters that work (global_aggregated_stats.get_shard(thd->thread_id()).<field>). For the Handler family, handler::ha_statistic_increment() receives a ulonglong System_status_var::* member pointer that cannot be applied to aggregated_stats_buffer (different field order, and the shard fields are std::atomic_uint64_t). The helper can be extended to also take the matching aggregated_stats_buffer member pointer so both the per-THD and shard counters are updated.