Description:
Every 2 days, we have a mysql restart with this error :
2024-12-16T11:55:04.425692+01:00 91609 [ERROR] [MY-010934] [Server] Out of memory (Needed 140008 bytes)
2024-12-16T11:55:04.425717+01:00 91609 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2024-12-16T11:55:04.442891+01:00 91744 [ERROR] [MY-010934] [Server] Out of memory (Needed 93352 bytes)
2024-12-16T11:55:04.442922+01:00 91744 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2024-12-16T11:55:04.469293+01:00 91693 [ERROR] [MY-010934] [Server] Out of memory (Needed 62248 bytes)
2024-12-16T11:55:04.469344+01:00 91693 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2024-12-16T11:55:04.577874+01:00 3271 [ERROR] [MY-010934] [Server] Out of memory (Needed 92104 bytes)
2024-12-16T11:55:04.577909+01:00 3271 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2024-12-16T11:55:04.619331+01:00 91693 [ERROR] [MY-010934] [Server] Out of memory (Needed 62248 bytes)
2024-12-16T11:55:04.619365+01:00 91693 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2024-12-16T11:55:05.568483+01:00 1258 [ERROR] [MY-010934] [Server] Out of memory (Needed 274472 bytes)
2024-12-16T11:55:05.568521+01:00 1258 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2024-12-16T11:55:06.260959+01:00 90265 [ERROR] [MY-010934] [Server] Out of memory (Needed 140008 bytes)
2024-12-16T11:55:06.260994+01:00 90265 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2024-12-16T11:55:06.288891+01:00 88019 [ERROR] [MY-013131] [Server] Out of sort memory, consider increasing server sort buffer size!
2024-12-16T11:55:06.324702+01:00 318 [ERROR] [MY-010934] [Server] Out of memory (Needed 88856 bytes)
2024-12-16T11:55:06.324736+01:00 318 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2024-12-16T11:55:06.346401+01:00 91785 [ERROR] [MY-010934] [Server] Out of memory (Needed 62248 bytes)
2024-12-16T11:55:06.346427+01:00 91785 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
10:55:06 UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x7f8723538000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f86d50ed510 thread_stack 0x46000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x20f67a1]
/usr/sbin/mysqld(handle_fatal_signal+0x31b) [0xfdcbdb]
/lib64/libpthread.so.0(+0x12d10) [0x7f8ebed70d10]
/usr/sbin/mysqld(Item_hex_string::make_hex_str(char const*, unsigned long)+0xfe) [0x11359be]
/usr/sbin/mysqld(Item_hex_string::hex_string_init(char const*, unsigned int)+0x1e) [0x1135a4e]
/usr/sbin/mysqld(Item_hex_string::Item_hex_string(YYLTYPE const&, MYSQL_LEX_STRING const&)+0x40) [0x1135bd0]
/usr/sbin/mysqld(MYSQLparse(THD*, Parse_tree_root**)+0x9b29) [0xff7589]
/usr/sbin/mysqld(THD::sql_parser()+0x31) [0xe37ff1]
/usr/sbin/mysqld(parse_sql(THD*, Parser_state*, Object_creation_ctx*)+0x123) [0xea6133]
/usr/sbin/mysqld(mysql_parse(THD*, Parser_state*)+0x165) [0xeadf15]
/usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x1e6a) [0xeb044a]
/usr/sbin/mysqld(do_command(THD*)+0x1a4) [0xeb10c4]
/usr/sbin/mysqld() [0xfcdde0]
/usr/sbin/mysqld() [0x267eada]
/lib64/libpthread.so.0(+0x81ca) [0x7f8ebed661ca]
/lib64/libc.so.6(clone+0x43) [0x7f8ebcc798d3]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Mysql performance_schema and sys tables dont really help. performance_schema tables seems to indicate a problem on sp_head::main_mem_root, but we do not retrieve it on sys tables.
SYS
EVENT_NAME CURRENT_NUMBER_OF_BYTES_USED HIGH_NUMBER_OF_BYTES_USED
memory/innodb/buf_buf_pool 26936082432 26936082432
memory/innodb/hash0hash 738056368 738056368
memory/temptable/physical_ram 507510784 631242752
memory/sql/TABLE 90704745 96655871
memory/innodb/memory 81443992 1275402704
memory/performance_schema/table_handles 47513600 47513600
memory/sql/TABLE_SHARE::mem_root 44002480 44323064
memory/performance_schema/events_statements_summary_by_digest 41600000 41600000
memory/performance_schema/events_errors_summary_by_thread_by_error 35426304 35426304
memory/innodb/buf0dblwr 34566272 39955744
Performance_schema
user event_name CURRENT_NUMBER_OF_BYTES_USED HIGH_NUMBER_OF_BYTES_USED
user1 memory/sql/sp_head::main_mem_root 1589658104 1714391232
user2 memory/innodb/memory 1044595360 1045714408
NULL memory/innodb/hash0hash 738056368 738056368
user2 memory/innodb/ha_innodb 354329064 354496648
user2 memory/sql/Log_event 249360672 251753568
user3 memory/sql/sp_head::main_mem_root 162964744 164225536
user4 memory/sql/String::value 158950960 160924008
user4 memory/sql/thd::main_mem_root 117794592 132273688
We have a server with 64go of RAM and the memory increase by 400mo/30minutes and finally when the command "/bin/systemctl status mysqld.service" return 59,9go, the mysql is restarting.
Also, we already use jemalloc.
The database size is 800go.
Do you have any idea ? it's a database with a lot of activity but it's like the memory was never freed.
How to repeat:
We cannot reproduce it in our environment , it's only on production and we cannot use valgrind/gdb etc.
