Bug #105861 MySQL 8.0.27 - Possible memory leak, documentation missing
Submitted: 10 Dec 2021 10:43 Modified: 14 Jul 2022 10:07
Reporter: Carlos Abrantes Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: Documentation Severity:S3 (Non-critical)
Version:8.0 OS:Any
Assigned to: CPU Architecture:Any

[10 Dec 2021 10:43] Carlos Abrantes
Description:
Hi,

running a MySQL server 8.0.27 it seems that the memory used by MySQL is higher than it should and keeps getting higher and higher slowly leaking (around 0.014G per hour) until server gets out of memory.

Looking to performance schema the instrument that stands out as being increasing is the memory/innodb/memory at this moment 32G, other instruments like memory/innodb/buf_buf_pool are steady at the same value since the startup.

I was no able to find any documentation about this instrument.
In the instruments table the "Documentation" field is NULL.

its fundamental to have good documentation about the instruments so we can try to understand better where the problem could be and to even create better bug reports.

How to repeat:
This happens without any special action, just normal usage (insert, updates, deletes, select) i also have json fields and simple master-master replication.

Nothing i can point out as the root cause.

Suggested fix:

1)Provide documentation about memory/innodb/memory that is missing, is should contain what "type of actions" are counted here and if possible and idea of how big this field should be.

2) with the previous maybe understand if there is a real memory lead and fix it
[10 Dec 2021 13:30] MySQL Verification Team
Hi Mr. A,

Thank you for your bug report.

However, this is not a bug.

We check every single release for weeks, of which lots of time is consumed on thorough memory leak testing by several different tools.

On the other hand, P_S consumers are not fully documented. That is because it would require another book to enable beginners to understand the meaning of each possible consumer attribute.

Hence, this can be considered a documentation request, although we are not sure whether and when will it be documented.
[10 Dec 2021 14:07] MySQL Verification Team
Just setting a correct version.
[10 Dec 2021 14:42] Carlos Abrantes
Hi,

indeed having documentation for that instrument would help to understand what values are being counted there and how big or not it can be.

From what i see it is just climbing without any control and limit.

This server was upgraded from 8.0.19 to 8.0.27 exactly because the release notes stated a few memory leaks solved.

Does it make sense, or is it possible to have a MySQL server with 92g ending up using all the 128g of the server?

temptable_max_ram - 1G or log size
innodb_log_file_size - 3G (innodb_log_files_in_group default 2)
innodb_log_buffer_size - 1G

plus a few more for connections and i guess some internals, i would images that more than 30g would be more than enough for this.

So can you help to understand how is this normal since you stated that there is no memory leak?

Thanks,
[10 Dec 2021 14:53] MySQL Verification Team
Hi,

This is normal, since InnoDB uses additional memory for various other tasks. It is all reused. There are some big objects in GIS, JSON etc ..... which can increase memory usage, but it will be re-used.

Anyway, wait that this bug is implemented and documentation updated, so you can read it and find out all that you need .......
[10 Dec 2021 16:20] Carlos Abrantes
Hi,

Ok i will wait to understand it.

this instrument started with less than 10G, it is always increasing over time.
now more than 32G, for a DB that has around 120G (data+index). Sounds very odd to me that it's 32G just for "runtime operations", i m sure that 1/4 of DB is not being used at each moment.

but looking forward for the documentation.

Thanks,
[27 Dec 2021 16:18] Carlos Abrantes
Hi,

After some time/tries i realise that memory/innodb/memory gets released with a simple analyze table <table name>.

I m able to reproduce this in prod and in lab with some load. Any ideas of what it could be?

I can see that memory is not release at OS, but maybe is the allocator (will testing with tcmalloc).

But bottom line what could be causing so much memory being allocated and them is "removed" from dozens of gigas (>60g) to less than one.

I heard that some table cache is also free with analyze, but could the table cache be so big? what determines the max size?

Thanks,
[2 Jan 2022 22:52] Christos Chatzaras
Do you use jemalloc? If yes can you switch to tcmalloc to see if it helps?
[3 Jan 2022 11:11] Carlos Abrantes
Hi,

was using default malloc libc, i also did a quick test with tcmalloc, the same is observed, so  memory/innodb/memory still increases over the time, analyze table free that counter/instrument from several gigabytes to a few megabytes and memory is not released in OS.

Questions:
What could it using so much memory and that gets "released" from MySQL perspective with the analyze table?
Could it be an error and the memory is not really released?Just removed from  memory/innodb/memory counter/instrument?

Thanks,
[4 Jan 2022 9:42] Carlos Abrantes
adding some info:
+-----------------+-------+
| max_connections | 1000  |
| table_open_cache           | 4000  |
| table_open_cache_instances | 16    |
| schema_definition_cache         | 256   |
| table_definition_cache          | 2000  |

| Max_used_connections      | 140                 |
[4 Jan 2022 9:58] MySQL Verification Team
Hi!

If you can run google heap profiler to check where the memory is allocated, it will help us understand the cause.  Do it on non-production system if you can repeat the behaviour there.  For example this is how I usually run it:

HEAP_PROFILE_ALLOCATION_INTERVAL=10000000000 HEAP_PROFILE_INUSE_INTERVAL=1000000 HEAPPROFILE=mybin.hprof LD_PRELOAD=/lib64/libtcmalloc.so ./bin/mysqld --defaults-file=/path/to/my.cnf --console

## run workload and check if tcmalloc is creating new heap profile files in teh datadir....   afterwards shutdown mysqld and generate outputs of heap profile file with biggest allocation:  e.g. 

pprof --lines --pdf --show_bytes --heapcheck --nodefraction=0.05 --edgefraction=0.05 --nodecount=100 --inuse_space ./bin/mysqld  ./data/mybin.hprof.5387.heap  >   mybin.hprof.5387.heap.pdf

pprof --lines  --text --show_bytes --stacks --inuse_space ./bin/mysqld  ./data/mybin.hprof.5387.heap  >   ./mybin.hprof.5387.heap.txt

Upload the .txt and .pdf file here for us to check.
[4 Jan 2022 12:24] Carlos Abrantes
Hi,

I m trying to follow your instructions.
I m running mysql in a docker container, 

Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0001.heap (0 MB currently in use)
....
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0212.heap (735 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0213.heap (736 MB currently in use)

but for some reason the files are being deleted.

HEAPPROFILE -> /var/lib/mysql/heap/mybin.hprof and this is a fs mounted to the container.

when i run ls i can see files there, but seems they are deleted and start always from mybin.hprof.0001.heap.

Even if in the logs i can see /var/lib/mysql/heap/mybin.hprof.0225.heap

Ideas?

Thanks,
[4 Jan 2022 14:21] Carlos Abrantes
Hi,

I manage to get it work, it was related to having the env vars to be used by all sessions or only on mysql running session.

i will run some tests.

Thanks,
[4 Jan 2022 15:48] Carlos Abrantes
Hi,

The test is running in lab env so i have some freedom to play with it.

Is it expected that the performance drop drastically?
I confirmed that with tcmalloc without profiling i can run at least the same load test as with default malloc.
With profiling i can only make 12% the load i was doing before.

Is this expected?

With this in mind i guess memory/innodb/memory will take much more time to grow.

Do you have a value in mind for memory/innodb/memory after which i can generate the reports?

Current values:
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 1000  |
+-----------------+-------+
1 row in set (0.44 sec)

+----------------------------+-------+
| Variable_name              | Value |
+----------------------------+-------+
| table_open_cache           | 4000  |
| table_open_cache_instances | 16    |
+----------------------------+-------+
2 rows in set (0.01 sec)

+---------------------------------+-------+
| Variable_name                   | Value |
+---------------------------------+-------+
| schema_definition_cache         | 256   |
| stored_program_definition_cache | 256   |
| table_definition_cache          | 2000  |
| tablespace_definition_cache     | 256   |
+---------------------------------+-------+
4 rows in set (0.01 sec)

+---------------------------+---------------------+
| Variable_name             | Value               |
+---------------------------+---------------------+
| Max_used_connections      | 109                 |
| Max_used_connections_time | 2022-01-04 17:46:25 |
+---------------------------+---------------------+
2 rows in set (0.07 sec)

| memory/innodb/buf_buf_pool                                                  | 2.04 GiB       |
| memory/innodb/hash0hash                                                     | 57.43 MiB      |
| memory/performance_schema/events_statements_summary_by_digest               | 39.67 MiB      |
| memory/innodb/ut0link_buf                                                   | 24.00 MiB      |
| memory/innodb/lock0lock                                                     | 22.22 MiB      |
| memory/innodb/memory                                                        | 18.99 MiB      |

Thanks,
[4 Jan 2022 16:13] MySQL Verification Team
Yes it's expected that profiling memory will affect performance.  Please see how high does the memory allocation go.  And once done upload the pprof outputs,  and also my.cnf used here..
[4 Jan 2022 16:15] MySQL Verification Team
Don't worry too much about P_S memory counters.  They could be wrong.
Instead,  focus on what "top" reports as RSS and what memory profile outputs in this line:

Dumping heap profile to ...... (X MB currently in use)

when that gets higher than expected for giving configuration, or continues increasing without limit,  we have a problem.
[5 Jan 2022 9:35] Carlos Abrantes
Hi,

until this test i'm sure it grows without a limit.

Now, since i can make around 10-12% of the load (due to the profiling) assuming that the will happen in the same proporcion it will take time until i see very high values.

To avoid being waiting for more days than the necessary, can you help with an estimation of what should be the limit? when can we say for sure that, ok we have a problem.

Maybe with the following information you can help with an estimation:
2G of innodb buffer (97% full)
1000 max clients
130 current connection
| Max_used_connections      | 812                 |

| table_open_cache           | 4000  |
| table_open_cache_instances | 16    |
innodb_log_file_size           = 512M
innodb_log_buffer_size         = 256M

From P_S:

| memory/innodb/buf_buf_pool                                         | 2.04 GiB       |
| memory/innodb/memory                                               | 395.92 MiB     |-->from yesterday 18MB 
| memory/innodb/hash0hash                                            | 57.43 MiB      |
| memory/performance_schema/events_errors_summary_by_thread_by_error | 48.42 MiB      |
| memory/performance_schema/events_statements_summary_by_digest      | 39.67 MiB      |

top reports 3.8G of RES.

Profiling logs:
...
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0635.heap (667039 MB allocated cumulatively, 1280 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0636.heap (676578 MB allocated cumulatively, 1291 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0637.heap (686115 MB allocated cumulatively, 1294 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0638.heap (695652 MB allocated cumulatively, 1298 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0639.heap (705189 MB allocated cumulatively, 1302 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0640.heap (714726 MB allocated cumulatively, 1307 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0641.heap (724262 MB allocated cumulatively, 1312 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0642.heap (733799 MB allocated cumulatively, 1317 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0643.heap (743336 MB allocated cumulatively, 1321 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0644.heap (752872 MB allocated cumulatively, 1325 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0645.heap (762409 MB allocated cumulatively, 1332 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0646.heap (1340 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0647.heap (1341 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0648.heap (1342 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0649.heap (1343 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0650.heap (1345 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0651.heap (1348 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0652.heap (1349 MB currently in use)
Dumping heap profile to /var/lib/mysql/heap/mybin.hprof.0653.heap (1351 MB currently in use)
[6 Jan 2022 18:09] Carlos Abrantes
Zip with the requested  txt and pdf files from pprof

Attachment: mybin.hprof.1080.heap_txt_pdf.zip (application/zip, text), 560.73 KiB.

[6 Jan 2022 18:24] Carlos Abrantes
Hi,

Not sure if at this moment its already consider a good time to take.

Current information:

Thu Jan  6 18:46:52 CET 2022
mysql> SELECT event_name, FORMAT_BYTES(current_alloc)        AS current_alloc2        FROM sys.x$memory_global_by_current_bytes order by current_alloc DESC limit 5
;
+--------------------------------------------------------------------+----------------+
| event_name                                                         | current_alloc2 |
+--------------------------------------------------------------------+----------------+
| memory/innodb/buf_buf_pool                                         | 2.04 GiB       |
| memory/innodb/memory                                               | 928.89 MiB     |
| memory/innodb/hash0hash                                            | 57.43 MiB      |
| memory/performance_schema/events_errors_summary_by_thread_by_error | 48.42 MiB      |
| memory/performance_schema/events_statements_summary_by_digest      | 39.67 MiB      |
+--------------------------------------------------------------------+----------------+
5 rows in set (0.01 sec)

top
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
10790 27        20   0 5497564   4.3g  21880 S 141.9 27.7   3903:48 mysqld

The pdf and txt are from this time.

As i lack the knowledge of pprof, i add also the generation outputs:
bash-4.4# pprof --lines --pdf --show_bytes --heapcheck --nodefraction=0.05 --edgefraction=0.05 --nodecount=100 --inuse_space /usr/sbin/mysqld /tmp/mybin.hprof.1080.heap > /tmp/mybin.hprof.1080.heap.pdf
Using local file /usr/sbin/mysqld.
Argument "MSWin32" isn't numeric in numeric eq (==) at /usr/bin/pprof line 5047.
Argument "linux" isn't numeric in numeric eq (==) at /usr/bin/pprof line 5047.
Using local file /tmp/mybin.hprof.1080.heap.
Warning: address    4fcab:	75 09                	jne    4fcb6 is longer than address length 16
Warning: address    4fcab:	75 09                	jne    4fcb6 is longer than address length 16
Warning: address    c1d7a:	74 0c                	je     c1d88 is longer than address length 16
Warning: address    c1d7a:	74 0c                	je     c1d88 is longer than address length 16
Dropping nodes with <= 98505009 B; edges with <= 98505009 abs(B)

bash-4.4# pprof --lines --text --show_bytes --stacks --inuse_space /usr/sbin/mysqld /tmp/mybin.hprof.1080.heap > /tmp/mybin.hprof.1080.heap.txt
Using local file /usr/sbin/mysqld.
Argument "MSWin32" isn't numeric in numeric eq (==) at /usr/bin/pprof line 5047.
Argument "linux" isn't numeric in numeric eq (==) at /usr/bin/pprof line 5047.
Using local file /tmp/mybin.hprof.1080.heap.
Warning: address    4fcab:	75 09                	jne    4fcb6 is longer than address length 16
Warning: address    4fcab:	75 09                	jne    4fcb6 is longer than address length 16
Warning: address    c1d7a:	74 0c                	je     c1d88 is longer than address length 16
Warning: address    c1d7a:	74 0c                	je     c1d88 is longer than address length 16

In the morning i took another snapshot:
Thu Jan  6 09:36:54 WET 2022
mysql> SELECT event_name, FORMAT_BYTES(current_alloc)        AS current_alloc2        FROM sys.x$memory_global_by_current_bytes order by current_alloc DESC limit 5 ;
+--------------------------------------------------------------------+----------------+
| event_name                                                         | current_alloc2 |
+--------------------------------------------------------------------+----------------+
| memory/innodb/buf_buf_pool                                         | 2.04 GiB       |
| memory/innodb/memory                                               | 807.20 MiB     |
| memory/innodb/hash0hash                                            | 57.43 MiB      |
| memory/performance_schema/events_errors_summary_by_thread_by_error | 48.42 MiB      |
| memory/performance_schema/events_statements_summary_by_digest      | 39.67 MiB      |
+--------------------------------------------------------------------+----------------+

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
10790 27        20   0 5368972   4.2g  21880 S 144.9 26.9   3240:52 mysqld

So seems that the increase in top is in line with the memory/innodb/memory.

Not sure if it can help, but innodb buffer was 97% so i assume we will not see allocations for it (lazy allocation), and load constant, did a delta between a heap files from the 2 moments:

bash-4.4# pprof --base=/tmp/mybin.hprof.0986.heap /usr/sbin/mysqld /tmp/mybin.hprof.1080.heap
Using local file /usr/sbin/mysqld.
Argument "MSWin32" isn't numeric in numeric eq (==) at /usr/bin/pprof line 5047.
Argument "linux" isn't numeric in numeric eq (==) at /usr/bin/pprof line 5047.
Using local file /tmp/mybin.hprof.1080.heap.
Warning: address    4fcab:	75 09                	jne    4fcb6 is longer than address length 16
Warning: address    4fcab:	75 09                	jne    4fcb6 is longer than address length 16
Warning: address    c1d7a:	74 0c                	je     c1d88 is longer than address length 16
Warning: address    c1d7a:	74 0c                	je     c1d88 is longer than address length 16
Welcome to pprof!  For help, type 'help'.
(pprof) text
Total: 115.4 MB
   119.7 103.7% 103.7%    119.7 103.7% mem_heap_create_block_func
     0.0   0.0% 103.7%     -4.0  -3.5% temptable::Handler::create
     0.0   0.0% 103.7%      0.0   0.0% std::set::~set
     0.0   0.0% 103.7%      0.0   0.0% ha_innobase::build_template
     0.0   0.0% 103.7%      0.0   0.0% MDL_ticket::create
     0.0   0.0% 103.7%      0.0   0.0% dd::cache::Shared_multi_map::put@1e40d30
     0.0   0.0% 103.7%      0.0   0.0% std::vector::_M_realloc_insert@e6a810
     0.0   0.0% 103.7%      0.0   0.0% row_create_update_node_for_mysql
     0.0   0.0% 103.7%      0.0   0.0% Json_object::consume
     0.0   0.0% 103.7%      0.0   0.0% open_table_def

I will keep the test running.

Thanks,
[10 Jan 2022 13:27] MySQL Verification Team
Thank you for your contribution, Mr. A.

From what we can conclude , for the time being, it could be the issue of the memory fragmentation due to the Standard Templates Library.
[10 Jan 2022 14:48] Carlos Abrantes
Hi,

Unfortunately MySQL was restarted by mistake 2 days ago, so it is now where it was when i sent the last update.

I will do best to ensure a few more days and update new information, to maybe help you "higher values".

Assuming your suspicious, what can make that fragmentation? I m assuming that bigger MySQL setups exists so this would be already known.

Also assuming your suspicious the "analyze table" that cleans the memory from PS counters does it make sense? Meaning from what i can see, it helps the memory to not grow, from your suspicious does it make sense?

Thanks,
[10 Jan 2022 14:57] MySQL Verification Team
Hi,

Please, just let us know if you see the problem with 8.0.27 or 8.0.28.

ANALYZE does not reduce memory usage significantly ........
[10 Jan 2022 15:01] Christos Chatzaras
Is 8.0.28 released?

In the change log I see:

Changes in MySQL 8.0.28 (Not yet released, General Availability)
[10 Jan 2022 15:09] MySQL Verification Team
Hi,

8.0.27 will do !!!!
[10 Jan 2022 16:45] Carlos Abrantes
Hi,

I'm not sure about the last messages, but this is in 8.0.27.

I m then wondering how ANALYZE brings down PS counters, bug?
Saying this because, running it every hour (analyze) seems to mitigate the issue (but to say for sure i need to run a longer test, more days).

If you want me to test 8.0.28, just let me know.

Thanks,
[11 Jan 2022 13:50] MySQL Verification Team
Hi,

Please wait for 8.0.28 ..... it is possible that it should be out this winter and then let us know .....

Thanks ......
[13 Jan 2022 11:04] Carlos Abrantes
Hi,

Now we a few more days (have in mind that with profile i can't make prod load, in prod it climbs much faster), we can see now that memory/innodb/memory  is bigger than memory/innodb/buf_buf_pool which seems accurate if we look to top (also increasing):

mysql> SELECT event_name, FORMAT_BYTES(current_alloc)        AS current_alloc2        FROM sys.x$memory_global_by_current_bytes order by current_alloc DESC limit 5 ;
+---------------------------------------------------------------+----------------+
| event_name                                                    | current_alloc2 |
+---------------------------------------------------------------+----------------+
| memory/innodb/memory                                          | 2.17 GiB       |
| memory/innodb/buf_buf_pool                                    | 2.04 GiB       |
| memory/innodb/hash0hash                                       | 57.43 MiB      |
| memory/performance_schema/events_statements_summary_by_digest | 39.67 MiB      |
| memory/innodb/ut0link_buf                                     | 24.00 MiB      |
+---------------------------------------------------------------+----------------+
5 rows in set (0.06 sec)

TOP
27981 27        20   0 9427164   5.9g  21300 S 123.5 37.3   8747:48 mysqld

My experiences tell me that running ANALYZE TABLE helps as the memory allocation seems to grown slowly (but not enough days to draw a conclusion).

I will upload new pdf and txt.

Any ETA when .28 will be released and if there is any mention to any memory related issue?

Thanks,
[13 Jan 2022 11:07] Carlos Abrantes
Zip with updated txt and pdf files from pprof

Attachment: mybin.hprof.2060.heap.zip (application/zip, text), 569.19 KiB.

[13 Jan 2022 11:18] MySQL Verification Team
Thanks for the heap profiles!  Do you have any json indexes as described in:
https://dev.mysql.com/doc/refman/8.0/en/create-index.html#create-index-multi-valued

I'm asking because of this:

Leak of 1540226160 bytes in 189310 objects allocated from:
	@ 0231841a unknown
	@ 0000000002318a46 mem_heap_add_block ?
	@ 000000000214b332 std::thread::_State_impl::_M_run ?
	@ 000000000214bf81 innobase_get_multi_value ?
	@ 000000000214d1f6 innobase_get_multi_value ?
	@ 000000000214d8e6 ha_innobase::update_row ?
	@ 000000000101cdc2 handler::ha_update_row ?
	@ 0000000000e8de7c Sql_cmd_update::update_single_table ?
	@ 0000000000e146f5 Sql_cmd_dml::execute ?
	@ 0000000000db1ef7 mysql_execute_command ?
[13 Jan 2022 14:00] Carlos Abrantes
Hi,

If i understood correctly, i was checking and yes indeed we have.

| Table   | Non_unique | Key_name           | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | Visible | Expression                                                                   |

| my_table |          1 | myindex1     |            1 | NULL        | A         |           3 |     NULL |   NULL | YES  | BTREE      |         |               | YES     | cast(json_extract(`my_json`,_utf8mb4\'$.my_json_key1\') as char(40) array) |
| my_table |          1 | myindex2     |            1 | NULL        | A         |          14 |     NULL |   NULL | YES  | BTREE      |         |               | YES     | cast(json_extract(`my_json`,_utf8mb4\'$.my_json_key2\') as char(40) array) |

my_json - json field

we have those 2 indexes, are those wrong? or right and maybe there is a leak in there?

Thanks,
[18 Jan 2022 13:54] MySQL Verification Team
Hi Carlos,

Do you know what queries update/insert the my_table containing that json index?
If you can give us a sample of the json and updates,  we can run some tests.

It appeared high in the memory profile,  so perhaps it leaks.
[18 Jan 2022 15:02] Carlos Abrantes
Hi,

I was about to update, it's clearly different the behaviour with and without those index.

I have dropped them and with a test of just 30m we can see a big difference in the  memory counters.

So i would say we are in the right path.

Thanks,
[18 Jan 2022 15:16] MySQL Verification Team
You are welcome.
[18 Jan 2022 16:33] Carlos Abrantes
Sample with table schema and queries

Attachment: Sample_mysql.txt (text/plain), 2.06 KiB.

[18 Jan 2022 16:37] Carlos Abrantes
Hi,

I added the table schema, select and update that the application does in loop.
In lab load test the json field values do not change, the int are incrementing.

Hope that with this data you are able to pinpoint where is the memory leak.

let me know the next steps, if i can stop the memory profiling or if any extra data can be provided to speed up troubleshooting.

Thanks,
Carlos
[19 Jan 2022 2:03] MySQL Verification Team
Hi Carlos!

Thanks for helping research this serious memory leak.

As this report is already a verified *documentation request*,
I have created a new report with repeatable testcase internally to fix the leak.  

Please check future release notes for when it's fixed:
Bug 33766482 - memory leak with multivalue json indexes
[19 Jan 2022 9:14] Carlos Abrantes
Hi,

i was not able to find that bug, is there any change to see the status/updates of:
Bug 33766482 - memory leak with multivalue json indexes

Basically to understand that when its fix i can assume that will be part of the next release.

Meantime while the its not fixed, is there any workaround we can use to bring down the memory (maybe run some cmd periodically)?with what you know does that "analyse" make sense? if not is it another  bug, since it "clears" the PS counters?

Thanks,
[19 Jan 2022 13:39] MySQL Verification Team
Hi,

Unfortunately, you can not follow the new bug, since it is strictly internal bug and only our developers can access it.

Regarding the workaround, you will have to rewrite your query thoroughly, even splitting it into two different queries and combining the results.
[16 Mar 2022 9:20] Carlos Abrantes
Hi MySQL Team,

Can you please let me know the status of the internal ticket that was open due to this one?
Bug 33766482 - memory leak with multivalue json indexes

Is it fixed? will it be part of 8.0.29? or there is any ETA for it?

Thanks,
[16 Mar 2022 10:21] MySQL Verification Team
Bug 33766482 - memory leak with multivalue json indexes 

is not yet fixed,  but it is in development's short-term queue.

Memory leaks are important bugs to fix, but I cannot give precise version that will have the fix as it's out of our hands (Support Team).
[16 Mar 2022 14:03] MySQL Verification Team
Hi Mr. A,

All you can do is what we are doing ......

Follow Release Notes for each new 8.0 release.

You can find them on dev.mysql.com.
[14 Jul 2022 10:07] Carlos Abrantes
Hi  MySQL Verification Team,

I know i asked before, and you said that we can only check the release notes.

But since you have access to the internal ticket:
Bug 33766482 - memory leak with multivalue json indexes

Is it possible for you to check what the status is?

Thanks,
[14 Jul 2022 10:12] MySQL Verification Team
Bug 33766482 - memory leak with multivalue json indexes 
this is closed as fixed in 8.0.30.
[14 Jul 2022 14:54] MySQL Verification Team
Hi Mr. A,

That bug report is updated only when it is fixed.

We have no access to the schedule of the bugs that are verified. Those schedules are done by each team separately and we do not have access to it.