Bug #16359 Cluster: Server API Crash upon query
Submitted: 11 Jan 2006 2:54 Modified: 5 Aug 2010 21:10
Reporter: Adam Dixon (Candidate Quality Contributor) Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:mysql-5.0 OS:Linux (RHEL4 64bit)
Assigned to: Assigned Account CPU Architecture:Any
Tags: 5.0.15, mysql-cluster-ndb-7.0.6

[11 Jan 2006 2:54] Adam Dixon
Description:
I am having my cluster mysql API servers crash out of the blue it seems.

-----
[New Thread 1085729120 (LWP 13094)]
[New Thread 1088338272 (LWP 13095)]
[New Thread 1085528416 (LWP 13108)]
[New Thread 1086933344 (LWP 13109)]
[New Thread 1087736160 (LWP 13110)]

Program received signal SIGUSR1, User defined signal 1.
[Switching to Thread 1089743200 (LWP 13082)]
0x000000364640b0ff in __read_nocancel () from /lib64/tls/libpthread.so.0
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n)
Please answer y or n.
The program being debugged has been started already.
Start it from the beginning? (y or n) Program not restarted.
(gdb) bt full

#0  0x000000364640b0ff in __read_nocancel () from /lib64/tls/libpthread.so.0
No symbol table info available.
#1  0x0000000000995c2c in vio_read (vio=0x2a99910ad8, buf=0x2a99561b58 "\001",
    size=4) at viosocket.c:44
        r = 1089740694
        _db_func_ = 0x40f41f50 ""
        _db_file_ = 0x40f41fa0 "\234�C"
        _db_level_ = 0
        _db_framep_ = (char **) 0x2a995d9518
        __PRETTY_FUNCTION__ = "vio_read"
#2  0x00000000005bd2f3 in my_real_read (net=0x2a995da2b0, complen=0x40f42048)
    at net_serv.cc:700
        length = -1
        i = 0
        len = 18446744073709551615
        remain = 4
        pos = (uchar *) 0x2a99561b58 "\001"
        retry_count = 0
        alarmed = 0x40f41fa8
        alarm_buff = {expire_time = 1136904860, alarmed = 1,
  thread = 1089743200, malloced = 0 '\0'}
        net_blocking = 0 '\0'
#3  0x00000000005bd8f5 in my_net_read (net=0x2a995da2b0) at net_serv.cc:874
        len = 10787703
        complen = 0
#4  0x00000000005de5e8 in do_command (thd=0x2a995d9518) at sql_parse.cc:1463
        packet = 0x0
        old_timeout = 30
        packet_length = 17
        net = (NET *) 0x2a995da2b0
        command = COM_QUERY
        _db_func_ = 0x101099561b78 <Address 0x101099561b78 out of bounds>
        _db_file_ = 0x3bf8ac8 ""
        _db_level_ = 42
        _db_framep_ = (char **) 0x40f420d0
#5  0x00000000005dd892 in handle_one_connection (arg=0x2a995d9518)
    at sql_parse.cc:1143
        error = 0
        net = (NET *) 0x2a995da2b0
        sctx = (Security_context *) 0x2a995daf40
        thd = (class THD *) 0x2a995d9518
        launch_time = 0
        set = {__val = {0 <repeats 16 times>}}
#6  0x000000364640613a in start_thread () from /lib64/tls/libpthread.so.0
No symbol table info available.
#7  0x00000036451c52b3 in clone () from /lib64/tls/libc.so.6
No symbol table info available.
#8  0x0000000000000000 in ?? ()
No symbol table info available.
-----

How to repeat:
Working on it.
[12 Jan 2006 8:22] Valeriy Kravchuk
Thank you for a problem report. Please, try to find a repeatable sequence of actions that demonstrates this crash. And, better, on a newer version. Your 5.0.15 is really old.
[24 Jan 2006 18:15] Hartmut Holzgraefe
I tried to reproduce your EXPLAIN crash using "CREATE TABLE SubAccounts" as found on

http://www.koders.com/sql/fid12F1624946C5DC40C71DEB3193D37C60A1CB3ED0.aspx

(using google), but it worked fine for me without problems. As your gdb backtrace shows that
the crash already happens in the parser state i think that it doesn't matter that i could not fill the
table with data.

You reported this bug for MySQL 5.0.15, can you try to reproduce this with a more current version?
I'd also like to know whether you are using a server binary provided by us (and if so the actual package name if possible), or whether you compile yourself
[2 Feb 2006 2:56] Adam Dixon
It seems that the problem only occurs when two different servers which run the same application at exactly the same time the server crashes. The applications run every 15minnutes, and I can make the server crash every single time if I have them run at the same time (ntp etc) However if i split them apart by 5 minutes they run perfectly ok. No problems.

So I ran mysqld in gdb again, and waited for cron to run the script on each of the servers;

Perhaps more backtrace output can help you. See my Attached file.
[2 Feb 2006 2:56] Adam Dixon
bt full output

Attachment: 20060202gdb.txt (text/plain), 8.48 KiB.

[2 Feb 2006 6:34] Adam Dixon
I did notice mentions of sql cache c files in the gdb backtrace, so I put SQL_NO_CACHE in the queries and I havent had a crash yet (both running at the same time and everything) This may also assist you.
[21 Feb 2006 7:42] Valeriy Kravchuk
OK, so the problem may be query cache related... Please, send your my.cnf files content from the failing servers. And, for the third time, I ask you to try to reproduce this with a newer version, 5.0.18.
[22 Mar 2006 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".