MySQL Bugs: #36656: mysql (5.0.51*, 5.1.*) crashes on FreeBSD 7-RELEASE

Bug #36656	mysql (5.0.51, 5.1.) crashes on FreeBSD 7-RELEASE
Submitted:	12 May 2008 1:52	Modified:	20 May 2008 10:53
Reporter:	A B	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	MySQL Server	Severity:	S2 (Serious)
Version:	5.0.51a,b 5.1.24rc1	OS:	FreeBSD (7.0-RELEASE)
Assigned to:		CPU Architecture:	Any
Tags:	assertion failure, Signal 11

Description:
Compiled from source (all attempts - 5.0.51, 5.0.51b, 5.1.24-rc) with:

./configure --with-mysqld-user=mysqld --with-unix-socket-path=/var/db/mysql/mysql.sock --prefix=/usr/local/mysql --sysconfdir=/etc --localstatedir=/var/db/mysql --with-federated-storage-engine

installed (make; make install), then inited with

/usr/local/mysql/bin/mysql_install_db --user=mysqld

and started with:

/usr/local/mysql/bin/mysqld_safe --user=mysqld

It immediately crashes (see log below).

Substituting libkse for libpthread also causes signal 11 crash but without the assertion message.

We use 5.0.51 without any issues on FreeBSD 6.3-RELEASE with the same build/install process. This is possibly a FreeBSD 7 bug, however our other apps run without problems.

Log says:

080512 02:27:30 mysqld started
080512 2:27:30InnoDB: Assertion failure in thread 678433024 in file os0sync.c line 623
InnoDB: Failing assertion: 0 == pthread_mutex_init(fast_mutex, MY_MUTEX_INIT_FAST)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
080512 2:27:30 - mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=0
read_buffer_size=258048
max_used_connections=0
max_connections=500
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 381996 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

080512 02:27:30 mysqld ended

How to repeat:
Compile, install, init, start on FreeBSD 7.0-RELEASE

Solution found:

a wild guess that my_fast_mutexattr was not initialized prior to use as MY_MUTEX_INIT_FAST proved true.

Inserting these two lines in the front of the original line 623 in os0sync.c (this is 5.0.51) solved the problem:

  pthread_mutexattr_init(&my_fast_mutexattr);
  pthread_mutexattr_settype(&my_fast_mutexattr, PTHREAD_MUTEX_ADAPTIVE_NP);

[the original line 623 follows these:
  ut_a(0 == pthread_mutex_init(fast_mutex, MY_MUTEX_INIT_FAST));
]

Thank you for a problem report. Please, send the results of:

uname -a
gcc --version

I want to check if this is 32-bit system and what version of gcc is used.

Here they are:

gcc --version:  gcc (GCC) 4.2.1 20070719  [FreeBSD]

uname -a: FreeBSD xxx.xxx.xxx 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Fri May  2 01:14:11 UTC 2008     root@:/usr/src/sys/i386/compile/CUSTOM  i386

The only difference between CUSTOM config and the distributed config is that CUSTOM has these 10 lines added at the end:

options               IPFIREWALL
options               IPFIREWALL_VERBOSE
options               IPFIREWALL_FORWARD
options               IPDIVERT
options               TCPDEBUG
options               DUMMYNET
options               NETATALK
device                crypto
options               GEOM_ELI
device                atapicam

Many thanks for reporting a bug.

MySQL don't support BSD make, please try GNU make here (gmake).

Officially, we don't support FreeBSD 7.0 yet. But we are already in testing phase that we can support it officially soon.

I think your problem is the same we ran into during tests. Please try the following:

$ ./configure --with-named-thread-libs=-pthread 

$ gmake

$ gmake install

Of course you can add all other options you need for configure there too.

Also you can look into bug #36686 if your problem is related to it.

Please let us know if "--with-named-thread-libs=-pthread" solved your problem or if it is related to the other bug.

There was no difference with your proposed build switches. Exactly the same happened (see errlog below). Also see my previous notes on behaviour when using kse thread library.

I suggest looking at difference between FreeBSD 6 and 7 - define of PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is new 7.

errlog:

080513 18:57:33InnoDB: Assertion failure in thread 678433024 in file os0sync.c line 623
InnoDB: Failing assertion: 0 == pthread_mutex_init(fast_mutex, MY_MUTEX_INIT_FAST)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
080513 18:57:33 - mysqld got signal 11;

Regarding bug #36686  - I don't think it's related. We never had build problems.

Sorry, I really can't repeat this.

$ uname -a
FreeBSD swedishchef 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Sun Feb 24 10:35:36 UTC

I tested MySQL 5.0.51b.

What you can do is to try to install from ports. Or look what the difference at the ports and your settings.

I will set this too need feedback. Please inform us, when you figure out more.

Also I will set category to InnoDB that an InnoDB developer can look into this. Maybe they have further ideas here too.

Sorry again, I have one more question:

Do you use 32 or 64bit? I can't repeat this by using FreeBSD 7.0 64bit.

I did try installing from ports (mysql-5.0.51a-freebsd6.0-i386 - there is no FreeBSD-7.0 port) with the same result.

The machine runs in 32-bit mode. Everything - OS, libs, apps - has been compiled from the source. It's 4-core intel CPU (see below.) It's possible that the problem occurs only on a multi-CPU hardware.

As I mentioned, one relevant difference between FreeBSD 6 and 7 is that the definition of 
PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is new in 7. The fix I posted works for us so far (for some reason, 'make test' miserably fails everything - can't run mysqld - and I don't have time to look into that.)

===

May  5 11:35:35 XXX kernel: Copyright (c) 1992-2008 The FreeBSD Project.
May  5 11:35:35 XXX kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
May  5 11:35:35 XXX kernel: The Regents of the University of California. All rights reserved.
May  5 11:35:35 XXX kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
May  5 11:35:35 XXX kernel: FreeBSD 7.0-RELEASE #0: Fri May  2 01:14:11 UTC 2008
May  5 11:35:35 XXX kernel: root@:/usr/src/sys/i386/compile/CUSTOM
May  5 11:35:35 XXX kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
May  5 11:35:35 XXX kernel: CPU: Intel(R) Xeon(R) CPU           X3353  @ 2.66GHz (2669.35-MHz 686-class CPU)
May  5 11:35:35 XXX kernel: Origin = "GenuineIntel"  Id = 0x10676  Stepping = 6
May  5 11:35:35 XXX kernel: Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
May  5 11:35:35 XXX kernel: Features2=0xce3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,<b19>>
May  5 11:35:35 XXX kernel: AMD Features=0x20100000<NX,LM>
May  5 11:35:35 XXX kernel: AMD Features2=0x1<LAHF>
May  5 11:35:35 XXX kernel: Cores per package: 4
May  5 11:35:35 XXX kernel: real memory  = 3484024832 (3322 MB)
May  5 11:35:35 XXX kernel: avail memory = 3404914688 (3247 MB)
May  5 11:35:35 XXX kernel: ACPI APIC Table: <DELL   PE_SC3  >
May  5 11:35:35 XXX kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
May  5 11:35:35 XXX kernel: cpu0 (BSP): APIC ID:  0
May  5 11:35:35 XXX kernel: cpu1 (AP): APIC ID:  1
May  5 11:35:35 XXX kernel: cpu2 (AP): APIC ID:  2
May  5 11:35:35 XXX kernel: cpu3 (AP): APIC ID:  3
May  5 11:35:35 XXX kernel: ioapic0: Changing APIC ID to 4
May  5 11:35:35 XXX kernel: ioapic0 <Version 2.0> irqs 0-23 on motherboard
May  5 11:35:35 XXX kernel: kbd1 at kbdmux0
May  5 11:35:35 XXX kernel: ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
May  5 11:35:35 XXX kernel: hptrr: HPT RocketRAID controller driver v1.1 (May  2 2008 01:14:01)
May  5 11:35:35 XXX kernel: cryptosoft0: <software crypto> on motherboard
May  5 11:35:35 XXX kernel: acpi0: <DELL PE_SC3> on motherboard
May  5 11:35:35 XXX kernel: acpi0: [ITHREAD]
May  5 11:35:35 XXX kernel: acpi0: Power Button (fixed)
May  5 11:35:35 XXX kernel: Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
May  5 11:35:35 XXX kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
May  5 11:35:35 XXX kernel: acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
May  5 11:35:35 XXX kernel: Timecounter "HPET" frequency 14318180 Hz quality 900
May  5 11:35:35 XXX kernel: cpu0: <ACPI CPU> on acpi0
May  5 11:35:35 XXX kernel: est0: <Enhanced SpeedStep Frequency Control> on cpu0
May  5 11:35:35 XXX kernel: p4tcc0: <CPU Frequency Thermal Control> on cpu0
May  5 11:35:35 XXX kernel: cpu1: <ACPI CPU> on acpi0
May  5 11:35:35 XXX kernel: est1: <Enhanced SpeedStep Frequency Control> on cpu1
May  5 11:35:35 XXX kernel: p4tcc1: <CPU Frequency Thermal Control> on cpu1
May  5 11:35:35 XXX kernel: cpu2: <ACPI CPU> on acpi0
May  5 11:35:35 XXX kernel: est2: <Enhanced SpeedStep Frequency Control> on cpu2
May  5 11:35:35 XXX kernel: p4tcc2: <CPU Frequency Thermal Control> on cpu2
May  5 11:35:35 XXX kernel: cpu3: <ACPI CPU> on acpi0
May  5 11:35:35 XXX kernel: est3: <Enhanced SpeedStep Frequency Control> on cpu3
May  5 11:35:35 XXX kernel: p4tcc3: <CPU Frequency Thermal Control> on cpu3

/mysql/include/my_pthread.h in 5.1:

/* Define mutex types, see my_thr_init.c */
#define MY_MUTEX_INIT_SLOW   NULL
#ifdef PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP
extern pthread_mutexattr_t my_fast_mutexattr;
#define MY_MUTEX_INIT_FAST &my_fast_mutexattr
#else
#define MY_MUTEX_INIT_FAST   NULL
#endif

/mysql/mysys/my_thr_init.c:

#ifdef PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP
  /*
    Set mutex type to "fast" a.k.a "adaptive"

    In this case the thread may steal the mutex from some other thread
    that is waiting for the same mutex.  This will save us some
    context switches but may cause a thread to 'starve forever' while
    waiting for the mutex (not likely if the code within the mutex is
    short).
  */
  pthread_mutexattr_init(&my_fast_mutexattr);
  pthread_mutexattr_settype(&my_fast_mutexattr,
                            PTHREAD_MUTEX_ADAPTIVE_NP);
#endif

InnoDB calls MySQL in the pthread mutex initialization. I guess the problem is that MySQL has defined MY_MUTEX_INIT_FAST as NULL, which does not work on a 32-bit FreeBSD-7.

This bug is not an InnoDB bug, but a bug in the operating system abstraction layer in MySQL, that is, /mysql/include and /mysql/mysys/. MySQL should define MY_MUTEX_INIT_FAST so that it works on all compilation platforms.

It's a duplicate of bug#36428

I suffer from the same problem running MySQL server on FreeBSD-6 after upgrading kernel sources. Both MySQL server versions 5.1.22 and 5.1.35 (built from FreeBSD ports) give the same result. The current mysql-server is:
$mysql --version
mysql Ver 14.14 Distrib 5.1.35, for portbld-freebsd6.3 (i386) using 5.2

This kernel works fine:
$uname -a
FreeBSD XXX.net.ua 6.3-RELEASE-p5 FreeBSD 6.3-RELEASE-p5 #0: Fri Oct 24 17:38:59 EEST 2008 root@XXX.net.ua:/usr/src/sys/i386/compile/XXX i386

while the following generates the memory trap:
$uname -a
FreeBSD XXX.net.ua 6.4-STABLE FreeBSD 6.4-STABLE #1: Tue Jun 23 10:07:24 EEST 2009 root@XXX.net.ua:/usr/src/sys/i386/compile/XXX

MySQL log reports:
090624 10:58:21 mysqld_safe Starting mysqld daemon with databases from /var/db/mysql
090624 10:58:21 InnoDB: Assertion failure in thread 139603968 in file os/os0sync.c line 665
InnoDB: Failing assertion: 0 == pthread_mutex_init(fast_mutex, MY_MUTEX_INIT_FAST)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
090624 10:58:21 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=0
max_threads=151
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 133283 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

090624 10:58:21 mysqld_safe mysqld from pid file /var/db/mysql/XXX.pid ended