MySQL Bugs: #12251: mysqld in free(): warning: junk pointer, too high to make sense

Bug #12251	mysqld in free(): warning: junk pointer, too high to make sense
Submitted:	28 Jul 2005 21:28	Modified:	17 Sep 2007 17:34
Reporter:	daniel lo	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server	Severity:	S2 (Serious)
Version:	5.0.21, 4.1.13	OS:	FreeBSD (FreeBSD 6, FreeBSD 5.4-Stable)
Assigned to:		CPU Architecture:	Any

Description:
Qualification on the S2: while this is serious, we are currently using an older database and can do so for a while.

I recieve the following error when running mysqld with or without debugging code.

mysqld in free(): warning: junk pointer, too high to make sense
mysqld in free(): warning: modified (chunk-) pointer

This error appears when I open about 200 connections to mysqld in about 10 seconds, and then start closing and opening new connections. Each of these connections is managed by a PERL (5.6) program designed to stress test the database. The perl program executes simple bare bones queries (reads/updates) (no joins, or sub-selects).

I've attempted to use the "DBUG" trace facilities to get further information on this problem, but I have been unsuccessful. I can provide you the trace file if you wish.

I ran mysqld with (line breaks added for readability)

/usr/local/libexec/mysqld
--basedir=/usr/local
--datadir=/var/db/mysql
--pid-file=/var/db/mysql/db11.picturetrail.com.pid
--socket=/var/db/mysql/mysql.sock
--debug=d,info,warning,error,safe:O,/spare/mysqld.trace
--core-file
--log-error=/spare/db11.err &

The trace file did not provide anything immediately "enlightening" to me regarding this memory error.

A mysqld.core file is produced when the debugging code is NOT enabled and no core file is produced when the debugging code is in use.

So, I can provide the following

a trace file.
a core file (without any debugging turned on).

If there is anything else I can do to provide more information on this bug report please let me know.

How to repeat:
There is no way for me to provide a "repeatable" description...

mysqld.core-512kb128jb is 542,445,568 bytes (512 key buffer size, and 128 join buffer size)
mysqld.trace is 11,468,853

One thing you can try to do is to start mysqld with --debug=S (you can use other flags too, of course, e.g. --debug=S:t:d:O,/spare/mysqld.trace)

It will make safemalloc to check the consistency of its memory allocation structures on every DBUG_ENTER/DBUG_RETURN - that will have performance impact, but could track down memory corruption down to a function that caused it.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

This PR was closed due to lack of activity.  In fact, there has been a lot of investigation, but it has been reported against support issue 6194.  I'm reopening it and changing the priority to "critical", since it relates to a crash.  To help maintain an overview, we'll keep the bulk of the information with the support issue until we can identify the cause of the bug, which is proving difficult.

Synopsis:

Under heavy load, mysqld SIGSEGVs out of the FreeBSD library routine mutex_unlock_common.  So far we have not been able to reproduce the problem here, and we can't even identify whether it's a bug in MySQL, in FreeBSD or a quirk of the hardware.  The processor is a single processor Opteron running the i386 version of FreeBSD 5.4.

The title of the bug report is misleading: the message "junk pointer, too high to make sense" does not cause a crash, though it implies some kind of problem with the management of malloc()ed memory.  During the course of investigations we have changed the library configuration, and this message no longer occurs.  This is no reason to believe that we have solved anything.

Customer reverted to FreeBSD 4.11 over the weekend of 8 October.  The server did not crash.

On the face of it, this looks like a FreeBSD 5.x problem.  We have installed FreeBSD 5.4-i386 on an in-house single processor Opteron server, and we're trying to reproduce the problem there.

I just had this problem on a new system I installed.

I created a jail, installed apache 1.3, and php5.
the mysql ext for php is causing this error. This wasn't a problem on my other 5.2.1 or 5.3 or 5.4 (not sure if I installed php on 5.4 though) boxes, so wonder if it's something new that happened in ports.

Thanks for your input on this bug (and sorry for the slow response).
Are you just getting the "junk pointer" message, or are you seeing
server crashes as well?

The junk pointer, usual came about in the libthr or libpthreads package.  We had to step back and go to 4.1.14 on FreeBSD 4.11.  We are installing FreeBSD 6.0 and 4.1.15 on a test machine and we will see how that works.

-daniel

Thanks for the quick reply.  Are you also experiencing server crashes?

With FreeBSD 4.11 we are not expierencing any more crashes.  When we have installed FreeBSD 6.0 and 4.1.(latest) or possibly 5.0.(latest) I will test it again, and see what happens.  Right now, our sys admin is very busy and since we have a working database this is not a top prioirty for her.

-daniel

Daniel, sorry about the confusion.  All our messages go out
individually to each person, so it's difficult to say who they were
addressed to.  My message of [29 Nov 14:00] was addressed to Danny
Swett, in the hope that he might be able to independently reproduce
your problems.  I misread your reply as a reply from Danny (your
similar names added to the confusion, of course).  It's good to hear
that 4.11 is no longer crashing, though.  

I'm putting this bug report back into "need feedback" status pending
a reply from Danny Swett.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

This is FreeBSD prerelease 6.1; MySQL 5.0.18 built from source; PHP 5.1.2.
At least once a day mysqld quits with a signal 6.

mysqld in free(): error: junk pointer, too high to make sense

Fatal error 'Recurse on a private mutex.' at line 988 in file /usr/src/lib/libpthread/thread/thr_mutex.c (errno = 42)
060225 13:36:37  mysqld restarted
060225 13:36:38 [Note] /usr/local/mysql/libexec/mysqld: ready for connections.
Version: '5.0.18-log'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
mysqld in free(): error: junk pointer, too high to make sense
mysqld got signal 6;

The same thing's happening on the backupserver with the same configuration.

It seems that every time when webalizer is resolving hostnames and a http-request comes in that needs a database connection, MySQL crashes with a signal 6.

Hi, just wanted to throw my two cents in here.  I've run into the same problem with freebsd 6.0 and mysql 4.1.18.   Under fairly heavy load the server would crash every 2-6 hours and restart.  (some log output below)

The interesting thing here is that it seems to be tied into hardware.  The machine having the problems is a dell power edge 1850 with SCSI raid 1 (perc 4/sc - amrd driver), dual processor with HT enabled.

I'm running the database successfully with no crashes on a dell SC1425 dual processor, HT enabled, with single SATA disk.  Also freebsd 6.0, but mysql 4.1.16  (this is our production db, so I have little incentive to bump this up to 4.18 to see if it breaks.  my hunch is that it wouldn't). 

In both cases mysql is built with native threading.  Variations of the optimization build settings made no difference.

Time permitting I'll try to get the problematic one back online as a RO slave with more debugging enabled and see if I can find out more.

=====

mysqld in free(): warning: modified (chunk-) pointer
060307 03:08:23  mysqld restarted

=====

mysqld in free(): warning: modified (chunk-) pointer
Fatal error 'Exceeded maximum lock level' at line 278 in file /usr/src/lib/libpthread/thread/thr_cancel.c (errno = 0
)
060307 03:42:19  mysqld restarted

======

mysqld got signal 10;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=838860800
read_buffer_size=6287360
max_used_connections=178
max_connections=300
threads_connected=91
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 924493 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

060307 08:48:36  mysqld restarted

Re-opened on request of submitter.

A similar problem. Dell PowerEdge 2850 (Dual Intel Xeon + HT + RAID 10 over 3+3 SCSI drives, FreeBSD 6.0-RELEASE-p6 + MySQL port version 4.0.26_1.

mysql log:
mysqld in free(): warning: modified (chunk-) pointer
Fatal error 'Thread has returned from _thread_switch' at line 1101 in file /usr/src/lib/libpthread/thread/thr_kern.c (errno =
0)
Fatal error 'Exceeded maximum lock level' at line 61 in file /usr/src/lib/libpthread/thread/thr_sigaction.c (errno = 3)
Fatal error 'Exceeded maximum lock level' at line 61 in file /usr/src/lib/libpthread/thread/thr_sigaction.c (errno = 3)
...
060417 01:43:22  mysqld restarted

Don't know if it's hardware or software related... Does anyone have any ideas?

Updating freebsd to a recent version of RELENG_6 to incorporate the patch as per:

http://www.freebsd.org/cgi/query-pr.cgi?pr=95127

Fixed this problem for me.  It sounds like compiling with linux threads would also solve the problem.

 scott

Could you please try the suggestion done by Scott?
-----------------------------------------------------------------------------------------
 [17 Apr 19:43] Scott Wilson

Updating freebsd to a recent version of RELENG_6 to incorporate the patch as
per:

http://www.freebsd.org/cgi/query-pr.cgi?pr=95127

Fixed this problem for me.  It sounds like compiling with linux threads would
also solve the problem.
-------------------------------------------------------------------------------------------
Thanks in advance.

Hi ... I just want to let you know that I have a similar problem on a "Dell Poweredge 1850 Dual Xeon". MySQL crashes twice a day. Here's a few of the log entries:

mysqld in free(): warning: modified (chunk-) pointer
   5851 Fatal error 'Thread has returned from _thread_switch' at line 1101 in file /usr/src/lib/libpthread/thread/thr_kern.c (errno = 0)
060813 19:50:20  mysqld restarted
060813 19:50:23 [Note] /usr/local/libexec/mysqld: ready for connections.
Version: '5.0.21'  socket: '/tmp/mysql.sock'  port: 3306  FreeBSD port: mysql-server-5.0.21
mysqld in free(): warning: modified (chunk-) pointer
060813 22:46:22  mysqld restarted
060813 22:46:26 [Note] /usr/local/libexec/mysqld: ready for connections.
Version: '5.0.21'  socket: '/tmp/mysql.sock'  port: 3306  FreeBSD port: mysql-server-5.0.21
mysqld in free(): warning: modified (chunk-) pointer
060814 10:42:26  mysqld restarted
060814 10:42:27 [Note] /usr/local/libexec/mysqld: ready for connections.
Version: '5.0.21'  socket: '/tmp/mysql.sock'  port: 3306  FreeBSD port: mysql-server-5.0.21
mysqld in free(): warning: modified (chunk-) pointer
Fatal error 'Can't resume thread in critical region
' at line 1000 in file /usr/src/lib/libpthread/thread/thr_kern.c (errno = 22)
060814 22:44:04  mysqld restarted
060814 22:44:06 [Note] /usr/local/libexec/mysqld: ready for connections.
Version: '5.0.21'  socket: '/tmp/mysql.sock'  port: 3306  FreeBSD port: mysql-server-5.0.21
mysqld in free(): warning: modified (chunk-) pointer

I am running FreeBSD 6.0-RELEASE with MySQL 5.0.21

I have been an administrator of this server for 6 months, but the server has been running for about 1 year, and as far as I know, this has always been a problem. 

I will try to follow the suggestion of upgrading freebsd and report back.

All reporters on FreeBSD 6,

Please, try to updatu FreeBSD to a recent version of RELENG_6 to incorporate the patch as per:

http://www.freebsd.org/cgi/query-pr.cgi?pr=95127

and check if this will fix the problem in your case, for MySQL 4.1.21 and 5.0.24a. Inform about the results.

I am the submitter of this comment: [15 Aug 21:23][name withheld]
I appologize .. I did not mean to be anonymous.

We have now recompiled and reinstalled mysql with linux threads, and this has solved the problem. I would never have guessed it without a hint from this page, so thank you all :-)

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

FreeBSD 5.4-RELEASE-p7
MySQL 4.0.27

Version: '4.0.27'  socket: '/tmp/mysql.sock'  port: 3306  FreeBSD port: mysql-server-4.0.27
mysqld in free(): warning: modified (chunk-) pointer
Fatal error 'Thread has returned from _thread_switch' at line 1101 in file /usr/src/lib/libpthread/thread/thr_kern.c (errno = 0)
0

Thank you, Robert, for the feedback.

But we need information which requested Valeriy in last himself comment: results of test with FreeBSD updated with specified patch and current version of MySQL. Which are 4.0.30, 4.1.23, 5.0.45 and 5.1.20. Please upgrade your software, try with it and say us if problem still exists.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".