| Bug #8221 | mysqld hangs on startup | ||
|---|---|---|---|
| Submitted: | 31 Jan 2005 22:33 | Modified: | 27 Aug 2005 19:47 |
| Reporter: | Dean Grimes | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server: InnoDB storage engine | Severity: | S1 (Critical) |
| Version: | 4.1.9 | OS: | Other (SCO OpenServer 5) |
| Assigned to: | Heikki Tuuri | CPU Architecture: | Any |
[31 Jan 2005 22:33]
Dean Grimes
[1 Feb 2005 13:08]
Heikki Tuuri
Hi! Please compile mysqld with debug info, run in a debugger, and submit the stack trace, so that we see where it hangs. It seems to work in some SCO computers, not in others. Regards, Heikki
[1 Feb 2005 16:48]
Dean Grimes
Another interesting note: If I compile with debug, mysqld starts and runs with InnoDB support. If I don't use debug, then mysqld just hangs.
[1 Feb 2005 16:55]
Heikki Tuuri
Hi! Do you mean that if you do: CFLAGS="-g -O3" CXXFLAGS="-O3 -g" ./configure make the resulting binary does not hang? That is rather strange if compiler flags affect this. Regards, Heikki
[1 Feb 2005 16:59]
Dean Grimes
Another thing I'm finding that I need to do in order recompile: I have to remove the entire mysql-4.1.9 tree and then untar fresh from the tarball. If I don't do that I get a 'No rule to make my_init.lo in libmysql' error and the make fails. Simply untaring without removing the tree first doesn't fix it. Looking at the make display, I noticed that when it runs fresh it symbolic links a bunch of files into the libmysql directory. This doesn't happen on subsequent makes.
[1 Feb 2005 17:00]
Dean Grimes
Actually, I configured with configure --prefix=/d/mysql --with-debug
[1 Feb 2005 17:03]
Heikki Tuuri
Dean, what compiler do you use? Please show the exact steps. Regards, Heikki
[1 Feb 2005 17:39]
Dean Grimes
rm -r mysql-4.1.9 tar xvf mysql-4.1.9.tar cd mysql-4.1.9 configure --prefix=/d/mysql --with-debug Compiler: Reading specs from /usr/gnu/lib/gcc-lib/i586-pc-sco3.2v5.0/2.95.3/specs gcc version 2.95.3 20010315 (release) I'm able to start mysqld: (carn700b:root)[/d/mysql/libexec] mysqld --verbose --skip-bdb -u root 050201 11:06:47 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 050201 11:06:47 InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 0 43716. InnoDB: Doing recovery: scanned up to log sequence number 0 43716 InnoDB: Last MySQL binlog file position 0 79, file name ./carn700b-bin.000006 050201 11:06:47 InnoDB: Flushing modified pages from the buffer pool... 050201 11:06:47 InnoDB: Started; log sequence number 0 43716 mysqld: ready for connections. Version: '4.1.9-debug-log' socket: '/d/mysql/var/mysql.sock' port: 3307 Sourc e distribution If I don't use --with-debug, mysqld just hangs.
[1 Feb 2005 17:44]
Heikki Tuuri
Dean, please use that CFLAGS... trick to add the debug info in the non-debug binary. Then gdb mysql When it hangs, send a signal kill -11 <processnumber here> to break gdb. It may also be possible to break the gdb execution simply with ctrl-C. Then (gdb) bt full to print the stack trace. Regards, Heikki
[1 Feb 2005 18:24]
Dean Grimes
(carn700b:root)[/d/mysql/libexec] gdb --args mysqld --skip-bdb -u root
GNU gdb 5.2.1
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i586-pc-sco3.2v5.0"...
(gdb) run
Starting program: /d/mysql/libexec/mysqld --skip-bdb -u root
Program received signal SIGSEGV, Segmentation fault.
0x800b2e10 in pthread_sigsuspend () from /usr/lib/libgthreads.so
(gdb) bt full
#0 0x800b2e10 in pthread_sigsuspend () from /usr/lib/libgthreads.so
No symbol table info available.
#1 0x800b29b7 in pthread_p_sigsuspend (set=0x0) at signal.c:3381
No locals.
#2 0x800ac796 in pthread_handle_many_process_signals () at signal.c:753
sig = -2146721376
#3 0x800b74f5 in pthread_sched_new_signals () from /usr/lib/libgthreads.so
No symbol table info available.
#4 0x800b733d in pthread_sched () from /usr/lib/libgthreads.so
No symbol table info available.
#5 0x800aa379 in pthread_cond_wait () from /usr/lib/libgthreads.so
No symbol table info available.
#6 0x082b49e1 in os_event_wait () at os0sync.c:335
__c = 0
os_sync_mutex = (struct os_mutex_struct *) 0x844c010
os_thread_count = 0
os_event_count = 10000
os_mutex_count = 2
os_fast_mutex_count = 10006
os_sync_mutex_inited = 1
os_event_list = {count = 10000, start = 0x859e36c, end = 0x844c0fc}
os_mutex_list = {count = 2, start = 0x844c0d8, end = 0x844c010}
#7 0x082a3251 in sync_array_wait_event (arr=0x844c034, index=0)
at sync0arr.c:433
---Type <return> to continue, or q <return> to quit---
cell = (struct sync_cell_struct *) 0x845358c
event = (struct os_event_struct *) 0x844c0fc
#8 0x082a6fc6 in mutex_spin_wait (mutex=0x843ae80,
file_name=0x83ec64f "sync0sync.c", line=233) at sync0sync.c:484
index = 0
i = 4
#9 0x082a6044 in mutex_create_func (mutex=0x843ac00,
cfile_name=0xffffffff <Address 0xffffffff out of bounds>, cline=1208)
at sync0sync.c:260
No locals.
#10 0x082aaf21 in sync_init () at sync0sync.c:1208
i = 105264
#11 0x08178252 in srv_boot () at srv0srv.c:877
No locals.
#12 0x0817ce5f in innobase_start_or_create_for_mysql () at srv0start.c:1105
ret = (struct buf_pool_struct *) 0x2a28
create_new_db = 0
log_file_created = 0
log_created = 0
log_opened = 0
min_flushed_lsn = {high = 0, low = 0}
max_flushed_lsn = {high = 0, low = 0}
sum_of_new_sizes = 0
sum_of_data_file_sizes = 138479840
---Type <return> to continue, or q <return> to quit---
tablespace_size_in_header = 0
err = 0
i = 138615777
srv_file_per_table_original_value = 0
mtr = {state = 1819374457, memo = {heap = 0x7261762f,
used = 137297967, data = <incomplete type>, base = {count = 138475349,
start = 0x1047658, end = 0x8}, list = {prev = 0x7, next = 0x80478d4}},
log = {heap = 0x4, used = 0, data = <incomplete type>, base = {
count = 138723162, start = 0x8047894, end = 0x8134383}, list = {
prev = 0x0, next = 0x8436ae0}}, modifications = 1073741824,
n_log_recs = 138479552, log_mode = 138479528, start_lsn = {high = 138479540,
low = 134510740}, end_lsn = {high = 135480254, low = 138664184},
magic_n = 138479560}
#13 0x08134617 in innobase_init () at ha_innodb.cc:1049
current_dir = "./"
err = 10792
default_path = 0x8431be1 ""
#14 0x0812db0f in ha_init () at handler.cc:260
have_yes = SHOW_OPTION_YES
known_extensions = {count = 0, name = 0x8396c33 "known_exts",
type_names = 0x0, type_lengths = 0x0}
sys_table_types = {{type = 0x83968cf "MyISAM", value = 0x840fce0,
comment = 0x8396900 "Default engine as of MySQL 3.23 with great performance"
, db_type = DB_TYPE_MYISAM}, {type = 0x83968bf "HEAP", value = 0x840fce0,
---Type <return> to continue, or q <return> to quit---
comment = 0x8396937 "Alias for MEMORY", db_type = DB_TYPE_HEAP}, {
type = 0x8396948 "MEMORY", value = 0x840fce0,
comment = 0x8396960 "Hash based, stored in memory, useful for temporary tabl
es", db_type = DB_TYPE_HEAP}, {type = 0x839699a "MERGE", value = 0x840fce0,
comment = 0x83969a0 "Collection of identical MyISAM tables",
db_type = DB_TYPE_MRG_MYISAM}, {type = 0x83969c6 "MRG_MYISAM",
value = 0x840fce0, comment = 0x83969d1 "Alias for MERGE",
db_type = DB_TYPE_MRG_MYISAM}, {type = 0x83969e1 "ISAM",
value = 0x840c440,
comment = 0x8396a00 "Obsolete storage engine, now replaced by MyISAM",
db_type = DB_TYPE_ISAM}, {type = 0x8396a30 "MRG_ISAM", value = 0x840c440,
comment = 0x8396a40 "Obsolete storage engine, now replaced by MERGE",
db_type = DB_TYPE_MRG_ISAM}, {type = 0x83968e1 "InnoDB",
value = 0x840c444,
comment = 0x8396a80 "Supports transactions, row-level locking, and foreign k
eys", db_type = DB_TYPE_INNODB}, {type = 0x8396abb "INNOBASE",
value = 0x840c444, comment = 0x8396ac4 "Alias for INNODB",
db_type = DB_TYPE_INNODB}, {type = 0x8396ad5 "BDB", value = 0x840c448,
comment = 0x8396ae0 "Supports transactions and page-level locking",
db_type = DB_TYPE_BERKELEY_DB}, {type = 0x8396b0d "BERKELEYDB",
value = 0x840c448, comment = 0x8396b18 "Alias for BDB",
db_type = DB_TYPE_BERKELEY_DB}, {type = 0x8396b26 "NDBCLUSTER",
value = 0x840c44c,
comment = 0x8396b40 "Clustered, fault-tolerant, memory-based tables",
---Type <return> to continue, or q <return> to quit---
db_type = DB_TYPE_NDBCLUSTER}, {type = 0x8396b6f "NDB", value = 0x840c44c,
comment = 0x8396b73 "Alias for NDBCLUSTER", db_type = DB_TYPE_NDBCLUSTER},
{type = 0x8396b88 "EXAMPLE", value = 0x840c63c,
comment = 0x8396b90 "Example storage engine",
db_type = DB_TYPE_EXAMPLE_DB}, {type = 0x8396ba7 "ARCHIVE",
value = 0x840c640, comment = 0x8396baf "Archive storage engine",
db_type = DB_TYPE_ARCHIVE_DB}, {type = 0x8396bc6 "CSV", value = 0x840c644,
comment = 0x8396bca "CSV storage engine", db_type = DB_TYPE_CSV_DB}, {
type = 0x0, value = 0x0, comment = 0x0, db_type = DB_TYPE_UNKNOWN}}
ha_row_type = {0x8396100 "", 0x8396bdd "FIXED", 0x8396be3 "DYNAMIC",
0x8396beb "COMPRESSED", 0x8396101 "?", 0x8396101 "?", 0x8396101 "?"}
tx_isolation_typelib = {count = 4, name = 0x8396100 "",
type_names = 0x840fe2c, type_lengths = 0x0}
ha_read_count = 0
ha_write_count = 0
ha_delete_count = 0
ha_update_count = 0
ha_read_key_count = 0
ha_read_next_count = 0
ha_read_prev_count = 0
ha_read_first_count = 0
ha_read_last_count = 0
ha_read_rnd_count = 0
ha_read_rnd_next_count = 0
---Type <return> to continue, or q <return> to quit---
ha_discover_count = 0
ha_commit_count = 0
ha_rollback_count = 0
tx_isolation_names = {0x8396bf6 "READ-UNCOMMITTED",
0x8396c07 "READ-COMMITTED", 0x8396c16 "REPEATABLE-READ",
0x8396c26 "SERIALIZABLE", 0x0}
known_extensions_id = 0
#15 0x080b23ca in init_server_components () at mysqld.cc:2680
No locals.
#16 0x080b27d8 in main (argc=4, argv=0x8047b14) at mysqld.cc:2959
No locals.
#17 0x0804b1a5 in _start ()
No symbol table info available.
(gdb)
[2 Feb 2005 15:48]
Heikki Tuuri
Hi! I guess SCO returns a non-standard return value from pthread_mutex_trylock(). I have to tweak configure.in. Thank you, Heikki
[7 Mar 2005 18:55]
Dean Grimes
Hi, I'm at a point in my project design where I need to decide if I should use InnoDB tables. I need to know if this will be fixed within the next few months. If not, then I'll fore go using InnoDB tables and opt for MySQL tables instead. I would really like to use InnoDB because I need to make the application multi-user. If there is anything you need from me, just let me know. Thanks, Dean
[2 May 2005 17:34]
Heikki Tuuri
Hi! Please post what man pthread_mutex_trylock prints in SCO Unix. Regards, Heikki
[5 May 2005 20:58]
Cuffiette
Hi Heikki, I apologize but in SCO package there are not the man pages. I attached the source code from the SCO Pthread 3.14 osr507 (ftp://ftp.zenez.com/pub/zenez/prgms/FSU-threads-3.14-osr507.tar.gz). Please look at http://bugs.mysql.com/bug.php?id=3961 also. I hope that it can be useful. ======== file disp.c ================================================== /*------------------------------------------------------------*/ /* * pthread_mutex_trylock - */ int pthread_mutex_trylock(mutex) pthread_mutex_t *mutex; { #ifdef _POSIX_THREADS_PRIO_PROTECT if (mutex->protocol == PRIO_PROTECT) return slow_mutex_trylock(mutex); #endif if (test_and_set(&mutex->lock)) { mutex->owner = mac_pthread_self(); return(EBUSY); } return(0); } ============ file mutex.c ============================================= /*------------------------------------------------------------*/ /* * pthread_mutex_trylock - Just try to lock the mutex. If * lock succeeds return. Otherwise return right away without * getting the lock. (Fast mutex_trylock without * error checks can be found in pthread_sched.S.) */ #ifdef NOERR_CHECK int slow_mutex_trylock(mutex) #else int pthread_mutex_trylock(mutex) #endif pthread_mutex_t *mutex; { pthread_t p = mac_pthread_self(); #ifndef NOERR_CHECK if (mutex == NO_MUTEX) #if !defined(DRAFT7) set_errno(EINVAL); return -1; #else /* !DRAFT7 */ return(EINVAL); #endif /* DRAFT7 */ #ifdef _POSIX_THREADS_PRIO_PROTECT if (mutex->protocol == PRIO_PROTECT && p->attr.prio > mutex->prio_ceiling) #if !defined(DRAFT7) set_errno(EINVAL); return -1; #else /* !DRAFT7 */ return(EINVAL); #endif /* DRAFT7 */ #endif /* _POSIX_THREADS_PRIO_PROTECT */ #endif /* !NOERR_CHECK */ SET_KERNEL_FLAG; if (mutex->lock) { CLEAR_KERNEL_FLAG; #if !defined(DRAFT7) set_errno(EBUSY); return -1; #else /* !DRAFT7 */ return(EBUSY); #endif /* DRAFT7 */ } mutex->lock = TRUE; mutex->owner = p; #ifdef NOERR_CHECK mac_change_lock_prio(mutex, p); #else #ifdef _POSIX_THREADS_PRIO_PROTECT if (mutex->protocol == PRIO_PROTECT) mac_change_lock_prio(mutex, p); #endif #endif CLEAR_KERNEL_FLAG; return(0); } ====== file opt.S ==================================================== ! int pthread_mutex_trylock(mutex) ! pthread_mutex_t *mutex; ! ! When mutex can be locked immediately, execute as leaf and return 1; ! otherwise return 0 ENTRY(pthread_mutex_trylock) #ifdef _POSIX_THREADS_PRIO_PROTECT ! load protocol type of mutex ld [%o0+mutex_protocol],%o1 cmp %o1, PRIO_PROTECT ! test if PRIO_PROTECT bne fast_trylock ! yes nop ! Delay save %sp,-SA(MINFRAME),%sp ! Get a new window call NAME(slow_mutex_trylock)! return(slow_mutex_lock(mutex)); mov %i0, %o0 ! Delay: Pass parameter mov %o0, %i0 ! Return value ret restore fast_trylock: #endif /* _POSIX_THREADS_PRIO_PROTECT */ ldstub [%o0+mutex_lock],%o1 ! test_and_set(&mutex->lock) tst %o1 ! prev. lock == 0 ? bne mutex_trylocked sethi %hi(NAME(pthread_kern)),%o1 ! Get kernel address or %o1,%lo(NAME(pthread_kern)),%o1 ld [%o1+pthread_self],%o1 st %o1,[%o0+mutex_owner] ! mutex->owner = mac_pthread_self() retl clr %o0 ! return(0); mutex_trylocked: set NAME(errno),%o1 ! errno = EBUSY; mov EBUSY,%o0 st %o0,[%o1] retl mov -1,%o0 ! return(-1);
[9 Aug 2005 17:19]
Dean Grimes
Has any progress been made on this issue? We will be going to pilot in the first of second week of September. I need to know if this is going to be resolved. Thanks, Dean
[9 Aug 2005 17:33]
Heikki Tuuri
Hi!
No progress made. Can you empirically check what pthread_mutex_trylock() returns in the SCO-compiled mysqld binary?
MySQL assumes:
/*
Convert pthread_mutex_trylock to return values according to latest POSIX
RETURN VALUES
0 If we are able successfully lock the mutex.
EBUSY Mutex was locked by another thread
# Other error number returned by pthread_mutex_trylock()
(Not likely)
*/
It is also possible that the hang has some other cause. trylock was just my first guess.
Regards,
Heikki
[9 Aug 2005 17:35]
Heikki Tuuri
Hi! Looking at the code in mutex.c, maybe you should compile with DRAFT7 as defined? Regards, Heikki
[11 Aug 2005 21:38]
Dean Grimes
Hi, I compiled the latest version of pthreads and recompiled MySQL but that didn't do the trick. So I tried your suggestion of compiling with DRAFT7 and that seems to have fixed the problem. I'll do some extensive testing next week and if everything is working OK, I'll close this issue. Thanks for pointing me in the right direction. Dean
[12 Aug 2005 14:06]
Heikki Tuuri
Dean, thank you. I will look if we can add that define of DRAFT7 to the configure.in in SCO. Regards, Heikki
[12 Aug 2005 14:54]
Heikki Tuuri
Dean, since you have to compile mutex.c with DRAFT7 defined, it does not help to add the define to MySQL's configure.in. Maybe we should document here: http://dev.mysql.com/doc/mysql/en/sco.html that if you use pthreads, you should compile the pthread library with DRAFT7 defined? Regards, Heikki
[12 Aug 2005 15:27]
Dean Grimes
That's correct. I just added it to the compile of the FSU-threads. I edited the Makefile for SCO and added -DDRAFT7 to the CFLAGS. So far everything seems to be OK. I finish my testing by Monday and I expect to close this issue then. Thanks, Dean
[17 Aug 2005 19:50]
Heikki Tuuri
Dean, maybe we should add to the manual: " If you use SCO OpenServer 5, you may need to recompile FSU threads with -DDRAFT7 in CFLAGS. Otherwise, InnoDB may hang at a mysqld startup. " Does the above sound sensible? I am no SCO expert. Regards, Heikki
[18 Aug 2005 12:56]
Dean Grimes
Hi, Yes, I think that would be a good idea. Also, I upgraded FSU Pthreads from 3.8c to 3.14; which is the latest release. Although I don't believe that had any bearing on the issue. However, all of my testing has indicated that the problem is history. I've tested on SCO OpenServer versions 5.04, 5.06 and 5.07. I have not tested 5.0.5 but I suspect that it would work just fine on that release as well. Thanks for all your help, Dean
[22 Aug 2005 15:04]
Heikki Tuuri
Dear docs team, please add to: http://dev.mysql.com/doc/mysql/en/sco.html " If you use SCO OpenServer 5, you may need to recompile FSU pthreads with -DDRAFT7 in CFLAGS. Otherwise, InnoDB may hang at a mysqld startup. " Regards, Heikki
[27 Aug 2005 19:47]
Paul DuBois
Thank you for your bug report. This issue has been addressed in the documentation. The updated documentation will appear on our website shortly, and will be included in the next release of the relevant product(s).
