MySQL Bugs: #27801: mysqld got signal 11 and crashes during mysql_real

Bug #27801	mysqld got signal 11 and crashes during mysql_real_connect call
Submitted:	13 Apr 2007 4:19	Modified:	11 May 2007 14:24
Reporter:	Dmitriy E	Email Updates:
Status:	Won't fix	Impact on me:	None
Category:	MySQL Server: C API (client library)	Severity:	S1 (Critical)
Version:	5.0.42-BK, 5.0.37, 5.0.27	OS:	Linux
Assigned to:	Antony Curtis	CPU Architecture:	Any
Tags:	crash, linux, mysql_real_connect, so, udf

Description:
At the end of the message there is a sample code how to reproduce the “bug”, which does not let us to connect from loaded .so to mysql server.

We have a UDF compiled as .so file. We do not connect from the UDF call, but we do connect on the library initialization. We do this on purpose becasue we cannot wai for an UDF call, there is no actual call. During this connect MySQL server crashes. 

The almost exact code works on Windows. We use MySQL 5.0.27 on both platfroms
 
Here is the error log:
 
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
 
key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=1
max_connections=100
threads_connected=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x18d6688, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x8164f80
0x815aaf6
0x815db05
0x815de53
0x8254178
0x82557ec
0xd59883
0xde73b6
0x1c933e
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

--- and here is the code ---
#include <pthread.h>
#include "mysql.h"
#include "udf.h"

void CreateThread(void* entryPoint, void *data){

	pthread_t* threadId = new pthread_t[1];
	int rc = 0;
	pthread_attr_t  attr;
	
	if (entryPoint == NULL){
     	return;
	}
    rc = pthread_attr_init(&attr);	
	if (rc) {
		return ; 
	}
    rc = pthread_create(threadId, &attr, (void*(*)(void*))entryPoint, data);
	if(rc){
    	return ;
	}      
	
	return;
}

void ThreadFunc(void *data) {
	MYSQL * mysqlStruct = NULL; 
	
	mysqlStruct = ::mysql_init(NULL);
    
	mysqlStruct = ::mysql_real_connect(mysqlStruct, 
											NULL, 
                                            "root", 
											"", 
											"mysql", 
											0, 
											"/var/lib/mysql/mysql.sock", 
											0);
}
void __attribute__ ((constructor)) dl_init(void){ 
     CreateThread((void*)ThreadFunc,NULL);
 } 
 
void __attribute__ ((destructor)) dl_deinit(void){}

__attribute__((dllexport)) double test(UDF_INIT *initid, UDF_ARGS *args, char *is_null,char *error){return 0.;}

__attribute__((dllexport)) my_bool test_init(UDF_INIT *initid, UDF_ARGS *args, char *message){return 0;}

__attribute__((dllexport)) void test_deinit(UDF_INIT *initid){}

How to repeat:
1. Compile this code as "myfile.so"

2. Register UDF, for example

insert into mysql.func(name, ret, dl, type) values
("process_monitor", 1, "myfile.so", "function");

3. Restart MySQL

Thank you for a problem report. Please, try to repeat with a newer version, 5.0.37. In case of the same problem, please, send the resolved stack trace and exact command line used for compilation.

Hi,

Thanks for looking into this! We are trying 5.0.37 and let you know soon if it works or not, but regardless of the result we need a solution for most 5.x.x versions, not just an unknown fix for a single version.

We checked lots of combinations of how to call mysql_init and mysql_real_connect just to see where the bug could be hidden. It appears that 
mysql_init() always works ok. On the other hand, mysql_real_connect() fails regardless of what we specify for the server localhost, or NULL. However, if we specify a specific port number or socket file path, mysql_real_connect() establishes the connection but then immediately crashes with segmentation fault.

If we specify wrong parameters for the connection, it does not connect and it does not crash.

In the debuger we see the following trace:

thd_increment_bytes_received <- net_real_write
my_net_read 
cli_safe_read
mysql_real_connect

Then the connection thread crashes with signal 1. Finally mysql exits.

Hope this could help you identity the root cause of the problem. Is there anything else specific to this issue that you want us to try, other then to try?

in the last message, ending "other then to try" wasn't intended for input

Any updates on this issue? Is there anybody looking into this?

Note that if you need solutions for different versions and/or in any predictable time ("ASAP") you have to become a customer and got a support contract. 

I am not sure that connecting to server from UDF is really supported and should work in general case. Please, send the resolved stack trace:

0x8164f80
0x815aaf6
0x815db05
0x815de53
0x8254178
0x82557ec
0xd59883
0xde73b6
0x1c933e

Hi,

We actually have MySQL network partner status. Hope this makes us eligible to submit bug reports and expect some response.

Please note that I don't expect this issue to be fixed ASAP, if it is an issue in MySQL, but I do expect that somebody can advise us on how to deal with it and the suggested solution is usable.

Thanks in advance for looking into this. We will provide the requested stack trace.

Dmitriy

Here is the requested stack trace. Hope this can help you to figure out the root cause of the problem.

0x8164f80 handle_segfault + 416
0x815aaf6 thd_increment_bytes_received + 22
0x815db05 _Z12my_real_readP6st_netPm + 453
0x815de53 my_net_read + 515
0x8254178 cli_safe_read + 296
0x82557ec mysql_real_connect + 940
0xd59883 (?)
0xde73b6 (?)
0x1c933e (?)

The stack trace I posted earlier is for version 5.0.27.

Here is a stack trace for version 5.0.37. As you can see, the result is the same; it crashes in the same place.

0x81698b0 handle_segfault + 416
0x815efb6 thd_increment_bytes_received + 22
0x8162035 _Z12my_real_readP6st_netPm + 453
0x8162383 my_net_read + 515
0x82606c8 cli_safe_read + 296
0x8261de5 mysql_real_connect + 997
0x5b293d (?)
0xde73b6 (?)
0x1c933e (?)

In your test case you have:

#include "udf.h"

Please, send content of that file. May I just remove that #include? Please, send also exact command line used to compile myfile.so.

I've got exactly the same results with latest 5.0.42-BK. Even if it (inability to handle "UDFs" in .so libraries like this) is intended behaviour without workarounds, server crash is still a bug, I think.

openxs@suse:~/dbs/5.0> bin/resolve_stack_dump -s /tmp/mysqld5.sym 16019_1.stack

0x81bbeb6 handle_segfault + 646
0x81a8298 thd_increment_bytes_received + 24
0x81b09fb _Z12my_real_readP6st_netPm + 123
0x81b0fcb my_net_read + 331
0x82f500d cli_safe_read + 269
0x82f5cfc mysql_real_connect + 828
0x42896803 _end + 974903663
0x40050aa7 _end + 932674579
0x40247c2e _end + 934735258
openxs@suse:~/dbs/5.0> uname -a
Linux suse 2.6.11.4-20a-default #1 Wed Mar 23 21:52:37 UTC 2005 i686 i686 i386 G
NU/Linux
openxs@suse:~/dbs/5.0> cat udf.c
#include <pthread.h>

#include "mysql.h"
/*#include "udf.h"*/

void CreateThread(void* entryPoint, void *data){

        pthread_t* threadId = (pthread_t*) malloc(sizeof(pthread_t));
        int rc = 0;
        pthread_attr_t  attr;

        if (entryPoint == NULL){
        return;
        }
    rc = pthread_attr_init(&attr);
        if (rc) {
                return ;
        }
    rc = pthread_create(threadId, &attr, (void*(*)(void*))entryPoint, data);
        if(rc){
        return ;
        }

        return;
}

void ThreadFunc(void *data) {
MYSQL * mysqlStruct = NULL;

mysqlStruct = mysql_init(NULL);
mysqlStruct = mysql_real_connect(mysqlStruct,
        "localhost",
        "root",
        "",
        "mysql",
        3306,
        NULL,
        0);
}
void __attribute__ ((constructor)) dl_init(void){
     CreateThread((void*)ThreadFunc,NULL);
 }

void __attribute__ ((destructor)) dl_deinit(void){}
double mytest(UDF_INIT *initid, UDF_ARGS *args, char *is_null,char *error){retur
n 0.;}
my_bool mytest_init(UDF_INIT *initid, UDF_ARGS *args, char *message){return 0;}
void mytest_deinit(UDF_INIT *initid){}

Compiled as:

openxs@suse:~/dbs/5.0> CFG=/home/openxs/dbs/5.0/bin/mysql_config
+ CFG=/home/openxs/dbs/5.0/bin/mysql_config
openxs@suse:~/dbs/5.0> gcc -shared -o udf.so `$CFG --cflags` udf.c `$CFG --libs
`
++ /home/openxs/dbs/5.0/bin/mysql_config --cflags
++ /home/openxs/dbs/5.0/bin/mysql_config --libs
+ gcc -shared -o udf.so -I/home/openxs/dbs/5.0/include/mysql udf.c -L/home/openx
s/dbs/5.0/lib/mysql -lmysqlclient -lz -lcrypt -lnsl -lm

Function mytest created as:

CREATE FUNCTION mytest RETURNS REAL SONAME 'udf.so';

Execution of statement above immediately leads to crash (and infinite restarts + crashed, in case of mysqld_safe used) with resolved stack trace above.

Full trace from gdb based on core file created:

(gdb) bt
#0  0xffffe410 in ?? ()
#1  0x42c675ec in ?? ()
#2  0x0000000b in ?? ()
#3  0x00001de9 in ?? ()
#4  0x40053838 in pthread_kill () from /lib/tls/libpthread.so.0
#5  0x082f7533 in write_core ()
#6  0x081bc054 in handle_segfault ()
#7  <signal handler called>
#8  0x081a8468 in thd_increment_bytes_received ()
#9  0x081b0bcb in my_real_read ()
#10 0x081b119b in my_net_read ()
#11 0x082f50ad in cli_safe_read ()
#12 0x082f5d9c in mysql_real_connect ()
#13 0x42896803 in ThreadFunc () from /home/openxs/dbs/5.0/udf.so
#14 0x40050aa7 in pthread_create () from /lib/tls/libpthread.so.0
#15 0x40247c2e in clone () from /lib/tls/libc.so.6

According to Kostja, it is a plugin problem.

the internal mysqld version of 'libmysql' client lib works on different rules than the standard library we have for C clients.

Expecting this code to work, which creates a thread not managed my mysqld and using mysqld internal funcs, even if they look like the mysql client libs... it is not going to work.

I would personally say that this bug is not a bug.

Note that the following code works as expected:

openxs@suse:~/dbs/5.0> export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:`pwd`
openxs@suse:~/dbs/5.0> CFG=/home/openxs/dbs/5.0/bin/mysql_config
openxs@suse:~/dbs/5.0> gcc -shared -o udf.so `$CFG --cflags` -DDEBUG udf4.c
openxs@suse:~/dbs/5.0> bin/mysqld_safe --core-file --core-file-size=256000000 &

[1] 3450
openxs@suse:~/dbs/5.0> Starting mysqld daemon with databases from /home/openxs/d
bs/5.0/var

openxs@suse:~/dbs/5.0> bin/mysql -uroot test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.0.42-debug Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> show processlist;
+----+------+----------------+-------+---------+------+-------+-----------------
-+
| Id | User | Host           | db    | Command | Time | State | Info
 |
+----+------+----------------+-------+---------+------+-------+-----------------
-+
|  1 | root | localhost:1289 | mysql | Sleep   |    9 |       | NULL
 |
|  2 | root | localhost      | test  | Query   |    0 | NULL  | show processlist
 |
+----+------+----------------+-------+---------+------+-------+-----------------
-+
2 rows in set (0.00 sec)

mysql> select * from mysql.func;
+--------+-----+--------+----------+
| name   | ret | dl     | type     |
+--------+-----+--------+----------+
| mytest |   1 | udf.so | function |
+--------+-----+--------+----------+
1 row in set (0.00 sec)

mysql> exit
Bye
openxs@suse:~/dbs/5.0> cat udf4.c
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h>

#include <unistd.h>

#include "mysql.h"
/*#include "udf.h"*/

#define ABSPATH "/home/openxs/dbs/5.0/lib/mysql"

void CreateThread(void* entryPoint, void *data){

        pthread_t* threadId = (pthread_t*) malloc(sizeof(pthread_t));
        int rc = 0;
        pthread_attr_t  attr;

        if (entryPoint == NULL){
        return;
        }
    rc = pthread_attr_init(&attr);
        if (rc) {
                return ;
        }
    rc = pthread_create(threadId, &attr, (void*(*)(void*))entryPoint, data);
        if(rc){
        return ;
        }

        return;
}

void ThreadFunc(void *data) {
MYSQL * mysqlStruct = NULL;

void *handle = dlopen(ABSPATH "/libmysqlclient_r.so", RTLD_NOW | RTLD_DEEPBIND);

void *myd_init;
void *myd_real_connect;

if (!handle)
{
printf("Failed to open: %s\n", dlerror());
exit(1);
}
myd_init=dlsym(handle, "mysql_init");
if (!myd_init)
{
printf("Failed to load symbol: %s\n", dlerror());
exit(1);
}

myd_real_connect=dlsym(handle, "mysql_real_connect");
if (!myd_real_connect)
{
printf("Failed to load symbol: %s\n", dlerror());
exit(1);
}

mysqlStruct = ((MYSQL *(*)(MYSQL *))myd_init)(NULL);

sleep(2);

mysqlStruct = ((MYSQL *(*)(MYSQL *,
                           const char *,
                           const char *,
                           const char *,
                           const char *,
                           unsigned int,
                           const char *,
                           unsigned long))myd_real_connect)(mysqlStruct,
        "127.0.0.1",
        "root",
        "",
        "mysql",
        3306,
        NULL,
        0);

}
void __attribute__ ((constructor)) dl_init(void){
     CreateThread((void*)ThreadFunc,NULL);
 }

void __attribute__ ((destructor)) dl_deinit(void){}
double mytest(UDF_INIT *initid, UDF_ARGS *args, char *is_null,char *error){retur
n 0.;}
my_bool mytest_init(UDF_INIT *initid, UDF_ARGS *args, char *message){return 0;}
void mytest_deinit(UDF_INIT *initid){}

openxs@suse:~/dbs/5.0> getconf GNU_LIBC_VERSION
glibc 2.3.4

But, in any case, this default linking to wrong (server's) code instead of one from libmysqlclient*, should be at least explicitely documented. Also, crashing of server when one just tried to add some weird UDF, and never called it (!), is also a problem to solve.

A possible solution/workaround is for the plugin to statically link/include the MySQL client library. My only major concern is how subsequent libraries will be resolved. A better long-term fix would be to ensure that the 'internal' versions of the mysql client operate in a different namespace than the public client. Since this would not change any of the published 'apis' it would not cause any breakages. The mysql daemon is mostly a C++ application and if we could wrap all its global declarations in a namespace to render then casually inaccessible, it would also releive similar problems as was encountered by Falcon in its early days with same-named class declarations.

There's no bug here. Functions in the server, even if they have the same name as the client C API functions, are in the server - they can only be called in the server context, and only if certain calling conventions are followed.

In particular, they expect a THD context. If you to use these functions from a separate thread, you need to create a THD for it.