Bug #117002 Mysql memory increasing without limits and restart every 2 days
Description:
Every 2 days, we have a mysql restart with this error :
2024-12-16T11:55:04.425692+01:00 91609 [ERROR] [MY-010934] [Server] Out of memory (Needed 140008 bytes)
2024-12-16T11:55:04.425717+01:00 91609 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2024-12-16T11:55:04.442891+01:00 91744 [ERROR] [MY-010934] [Server] Out of memory (Needed 93352 bytes)
2024-12-16T11:55:04.442922+01:00 91744 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2024-12-16T11:55:04.469293+01:00 91693 [ERROR] [MY-010934] [Server] Out of memory (Needed 62248 bytes)
2024-12-16T11:55:04.469344+01:00 91693 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2024-12-16T11:55:04.577874+01:00 3271 [ERROR] [MY-010934] [Server] Out of memory (Needed 92104 bytes)
2024-12-16T11:55:04.577909+01:00 3271 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2024-12-16T11:55:04.619331+01:00 91693 [ERROR] [MY-010934] [Server] Out of memory (Needed 62248 bytes)
2024-12-16T11:55:04.619365+01:00 91693 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2024-12-16T11:55:05.568483+01:00 1258 [ERROR] [MY-010934] [Server] Out of memory (Needed 274472 bytes)
2024-12-16T11:55:05.568521+01:00 1258 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2024-12-16T11:55:06.260959+01:00 90265 [ERROR] [MY-010934] [Server] Out of memory (Needed 140008 bytes)
2024-12-16T11:55:06.260994+01:00 90265 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2024-12-16T11:55:06.288891+01:00 88019 [ERROR] [MY-013131] [Server] Out of sort memory, consider increasing server sort buffer size!
2024-12-16T11:55:06.324702+01:00 318 [ERROR] [MY-010934] [Server] Out of memory (Needed 88856 bytes)
2024-12-16T11:55:06.324736+01:00 318 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2024-12-16T11:55:06.346401+01:00 91785 [ERROR] [MY-010934] [Server] Out of memory (Needed 62248 bytes)
2024-12-16T11:55:06.346427+01:00 91785 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
10:55:06 UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x7f8723538000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f86d50ed510 thread_stack 0x46000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x20f67a1]
/usr/sbin/mysqld(handle_fatal_signal+0x31b) [0xfdcbdb]
/lib64/libpthread.so.0(+0x12d10) [0x7f8ebed70d10]
/usr/sbin/mysqld(Item_hex_string::make_hex_str(char const*, unsigned long)+0xfe) [0x11359be]
/usr/sbin/mysqld(Item_hex_string::hex_string_init(char const*, unsigned int)+0x1e) [0x1135a4e]
/usr/sbin/mysqld(Item_hex_string::Item_hex_string(YYLTYPE const&, MYSQL_LEX_STRING const&)+0x40) [0x1135bd0]
/usr/sbin/mysqld(MYSQLparse(THD*, Parse_tree_root**)+0x9b29) [0xff7589]
/usr/sbin/mysqld(THD::sql_parser()+0x31) [0xe37ff1]
/usr/sbin/mysqld(parse_sql(THD*, Parser_state*, Object_creation_ctx*)+0x123) [0xea6133]
/usr/sbin/mysqld(mysql_parse(THD*, Parser_state*)+0x165) [0xeadf15]
/usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x1e6a) [0xeb044a]
/usr/sbin/mysqld(do_command(THD*)+0x1a4) [0xeb10c4]
/usr/sbin/mysqld() [0xfcdde0]
/usr/sbin/mysqld() [0x267eada]
/lib64/libpthread.so.0(+0x81ca) [0x7f8ebed661ca]
/lib64/libc.so.6(clone+0x43) [0x7f8ebcc798d3]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Mysql performance_schema and sys tables dont really help. performance_schema tables seems to indicate a problem on sp_head::main_mem_root, but we do not retrieve it on sys tables.
SYS
EVENT_NAME CURRENT_NUMBER_OF_BYTES_USED HIGH_NUMBER_OF_BYTES_USED
memory/innodb/buf_buf_pool 26936082432 26936082432
memory/innodb/hash0hash 738056368 738056368
memory/temptable/physical_ram 507510784 631242752
memory/sql/TABLE 90704745 96655871
memory/innodb/memory 81443992 1275402704
memory/performance_schema/table_handles 47513600 47513600
memory/sql/TABLE_SHARE::mem_root 44002480 44323064
memory/performance_schema/events_statements_summary_by_digest 41600000 41600000
memory/performance_schema/events_errors_summary_by_thread_by_error 35426304 35426304
memory/innodb/buf0dblwr 34566272 39955744
Performance_schema
user event_name CURRENT_NUMBER_OF_BYTES_USED HIGH_NUMBER_OF_BYTES_USED
user1 memory/sql/sp_head::main_mem_root 1589658104 1714391232
user2 memory/innodb/memory 1044595360 1045714408
NULL memory/innodb/hash0hash 738056368 738056368
user2 memory/innodb/ha_innodb 354329064 354496648
user2 memory/sql/Log_event 249360672 251753568
user3 memory/sql/sp_head::main_mem_root 162964744 164225536
user4 memory/sql/String::value 158950960 160924008
user4 memory/sql/thd::main_mem_root 117794592 132273688
We have a server with 64go of RAM and the memory increase by 400mo/30minutes and finally when the command "/bin/systemctl status mysqld.service" return 59,9go, the mysql is restarting.
Also, we already use jemalloc.
The database size is 800go.
Do you have any idea ? it's a database with a lot of activity but it's like the memory was never freed.
How to repeat:
We cannot reproduce it in our environment , it's only on production and we cannot use valgrind/gdb etc.
Portions of this website are copyright © 2001, 2002 The PHP Group
Page generated in 0.052 sec. using MySQL 8.4.7-u6-cloud
Timestamp references displayed by the system are UTC.
Content reproduced on this site is the property of the
respective copyright holders.
It is not reviewed in advance
by Oracle and does not necessarily represent the opinion of
Oracle or any other party.