MySQL Bugs: #35882: UDF in Trigger result in a crash on heavy duty

Bug #35882	UDF in Trigger result in a crash on heavy duty
Submitted:	7 Apr 2008 16:55	Modified:	9 Jul 2008 15:36
Reporter:	Alfonso Salgado	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server	Severity:	S1 (Critical)
Version:	5.0.37, 5.0.51a	OS:	Linux (SUSE 10 )
Assigned to:		CPU Architecture:	Any

Description:
MySql crashes periodicaly using UDF in triggers.

-Trigger is like:
BEGIN
DECLARE myStrData VARCHAR(128);
...
SELECT getMyData(param1,param2,myhost,myport,param3,'I') INTO myStrData;
...
END

UDF sends information through a socket TCP.

LOGS:

...
07/04/2008, 10:11.12 getMyData
Numero de argumentos: 6
Argumento 0
Expresion = param1
Tipo REAL
Valor = 336720.490000
Argumento 1
Expresion = param2
Tipo REAL
Valor = 6290731.110000
Argumento 2
Expresion = '192.168.0.213'
Tipo STRING
Valor = 192.168.0.213
Argumento 3
Expresion = 6090
Tipo INT
Valor = 6090
Argumento 4
Expresion = param3
Tipo STRING
Valor = param3
Argumento 5
Expresion = 'I'
Tipo STRING
Valor = I
HOST = 192.168.6.213
Port = 6090
Connect
String to send
Sendign 336720.4900000000 6290731.1100000003 param3 I
Close socket
return
Close socket
return
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=524288000
read_buffer_size=2093056
max_used_connections=162
max_connections=250
threads_connected=160
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 3070998 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x2aaad02359e0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x468e8170, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
(nil)
Stack trace seems successful - bottom reached
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x2aab08e55850 is invalid pointer
thd->thread_id=686
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
Number of processes running now: 0
080407 10:11:12 mysqld restarted

-Server memory is 8GB
-Kernel 2.6.16.21-0.8-smp
-x86_64

How to repeat:
Use a UDF that retuns a string in a server on heavy duty

Thank you for a problem report. Please, try to repeat with a newer version, 5.0.51a, and ifnform about the results.

-This happened with mysql 5.0.51a too .
-It ocurres when there are more than one thread running.

MySql crashed in the next scenario:
-Two databases with trigger (AFTER UPDATE) that calls a UDF with socket connection.
-Two clients simultaneously updating the table one to each database. This ocurres only when thereare a big amount of data.

Mysql was compiled with --with-debug=full and we disable safe-malloc.

The result was:

080507 12:14:45 - mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1073741824
read_buffer_size=2093056
max_used_connections=23
max_connections=100
threads_connected=23
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 2072172 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x2aaaf200b348
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x445da520, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x5945f1
0x2aaaf167f6de
0x51473d
0x4eb626
0x727e59
0x730903
0x730950
0x57e120
0x61f18b
0x61cb3b
0x621577
0x5ada43
0x72811f
0x728a61
0x729153
0x72b2d4
0x72cb59
0x738de2
0x63e48e
0x5aff14
0x5b5a80
0x5b8676
0x5b9f11
0x5bb1db
0x2b1fb1367193
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x2aaaf230f538 is invalid pointer
thd->thread_id=103
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Memory status:
Non-mmapped space allocated from system: 82423808
Number of free chunks: 60
Number of fastbin blocks: 0
Number of mmapped regions: 18
Space in mmapped regions: -2019606528
Maximum total allocated space: 0
Space available in freed fastbin blocks: 0
Total allocated space: 82156640
Total free space: 267168
Top-most, releasable space: 76096
Estimated memory (with thread stack): -1931939840

Number of processes running now: 0
080507 12:14:45 mysqld restarted

Hello Alfonso,

can you configure mysqld to produce a core file on crashes
( http://dev.mysql.com/doc/refman/5.0/en/server-options.html#option_mysqld_core-file )
and provide us with both the produced core file and the
exact mysqld binary you are using for analysis?

Enabling a server trace file might also produce some 
useful extra information:
http://dev.mysql.com/doc/refman/5.0/en/making-trace-files.html

Hello, 

The files required was uploaded as bug-mysqld-35882.tar.gz and bug-corefile-35882.tar.gz.

Please confirm me if the upload was succed.

Thanks.

Hello Alfonso,

the files arrived just fine, i can't extract any useful information from the core dump though. The backtrace information produced by gdb only shows references to system library functions at some points but none of the thread stacks shows any calls to actual mysql server functions even though the mysqld binary was not stripped from its symbol tables.

This might be due to our systems being too different (i'm checking the core file on Ubuntu 8.04 / amd64 right now) or due to stack corruption during the crash.

To further check this i need to know the exact operating system version you are using (openSuse or SLES, version number, output of "uname -a") so that we can arrange for an appropriate test machine.

If you have gdb installed we'd also be interested in what output you are getting from

  $ gdb mysqld core.17310
  [...]
  (gdb) thread apply all backtrace

Hello Hartmut,

About the operating system:
1)SLES 10
2)uname -a:
 Linux prometeo 2.6.16.21-0.8-smp #1 SMP Mon Jul 3 18:25:39 UTC 2006 x86_64 x86_64 x86_64 GNU/Linux

The output I'm getting with gdb is in attached file

Regards

gdb output file

Attachment: gdb_out.txt (text/plain), 21.63 KiB.

Ok, the actual problem is in this stack frame entry:

#3  0x00002aaaf0e156de in getDireccionDistancia (initid=<value optimized out>, args=0x430ca80,
    result=0x4471f120 "", res_length=0x4471f0c8, null_value=0x4471f0d7 "", error=0x430cae0 "") at test.c:262

So you need to take a look at what your getDireccionDistancia() function is doing in test.c on line 262 ...

Hello Hartmut,

The source code is:

int sockfd, portno, n;
struct sockaddr_in serv_addr;
struct hostent *server;
char buffer[256];
//OBTENEMOS NOMBRE DEL SERVIDOR
server=gethostbyname(args->args[2]);
if (server == NULL) 
{
     fprintf(stderr,"Error server\n");
     (*error)=0;
     (*null_value)=1;
     return NULL;
}
//Puerto
portno =  (int)*((long long*) args->args[3]);
serv_addr.sin_family = AF_INET;   
serv_addr.sin_port = htons(portno); 
serv_addr.sin_addr = *((struct in_addr *)server->h_addr);(Line 262)
memset(&(serv_addr.sin_zero),0x0,8); 
sockfd = socket(AF_INET, SOCK_STREAM, 0);
if (sockfd < 0)
{
    (*error)=0;
    (*null_value)=1;
    return NULL;	
}

Regards

Just a blind educated guess, but shouldn't you be using the thread safe 
gethostbyname_r() instead of gethostbyname() as mysqld is multi threaded?

As the crash happens on dereferencing an element of the hostent structure
returned by gethostbyname() a reuse of gethostbyname()s static result buffer
could well be blamed for the problem ...

You are completely right.
Thank you very much.

I will make some test more in order to confirm you that the problem was that function.

Regards

The error was than function.

Regards.