Bug #26939 | We randomly get "MySQL Error 1030: Got error 127 from storage engine" on MyISAM | ||
---|---|---|---|
Submitted: | 7 Mar 2007 23:20 | Modified: | 10 Jan 2008 4:56 |
Reporter: | Steve Meyers | Email Updates: | |
Status: | No Feedback | Impact on me: | |
Category: | MySQL Server: MyISAM storage engine | Severity: | S3 (Non-critical) |
Version: | 4.1.22, 5.0.45 | OS: | Linux (RHEL 3) |
Assigned to: | CPU Architecture: | Any | |
Tags: | 127, myisam, NPTL, regression |
[7 Mar 2007 23:20]
Steve Meyers
[7 Mar 2007 23:55]
Steve Meyers
I forgot to mention that CHECK TABLE and REPAIR TABLE both claim that the table(s) in question are fine.
[8 Mar 2007 15:39]
MySQL Verification Team
steve, thanks for reporting this. try set thread_cache_size=0 in my.cnf and check if this problem continues. please upload existing my.cnf to this bug report too.
[8 Mar 2007 16:52]
Steve Meyers
[client] port = 3306 socket = /var/lib/mysql/mysql.sock [mysqld] wait_timeout = 1000 net_read_timeout= 1000 connect_timeout = 1000 max_connections = 750 max_connect_errors = 1000000000 port = 3306 socket = /var/lib/mysql/mysql.sock skip-locking key_buffer = 384M max_allowed_packet = 16M sort_buffer_size = 4M read_buffer_size = 4M myisam_sort_buffer_size = 64M thread_cache = 8 query_cache_size = 32M thread_concurrency = 8 open_files_limit = 8192 table_cache = 2048 tmpdir = /data/mysql-tmp relay-log = relay-bin.nnnnnn relay-log-index = relay-bin.index server-id = 45 report-host = db45 slave_net_timeout = 300 innodb_data_home_dir = /var/lib/mysql/ innodb_data_file_path = ibdata1:10M:autoextend:max:1024M innodb_log_group_home_dir = /var/lib/mysql/ innodb_log_arch_dir = /var/lib/mysql/ [mysqldump] quick max_allowed_packet = 16M [mysql] no-auto-rehash [isamchk] key_buffer = 256M sort_buffer_size = 256M read_buffer = 2M write_buffer = 2M [myisamchk] key_buffer = 256M sort_buffer_size = 256M read_buffer = 2M write_buffer = 2M [mysqlhotcopy] interactive-timeout
[8 Mar 2007 19:41]
Valeriy Kravchuk
Have you tried to restart server with thread_cache_size=0? We need to know if it makes any difference in your case.
[9 Mar 2007 18:43]
Steve Meyers
We tried setting thread_cache_size=0, and we're still getting the errors.
[4 Apr 2007 17:11]
Steve Meyers
We've had to downgrade all of our systems to 4.1.21, this is causing our applications to crash frequently.
[4 Apr 2007 18:59]
Valeriy Kravchuk
Please, send the results of: getconf GNU_LIBPTHREAD_VERSION and upload your entire error log from 4.1.22 (gzipped).
[4 Apr 2007 19:26]
Steve Meyers
The error log is very short, so I'll just include it here: [root@db49 mysql]# getconf GNU_LIBPTHREAD_VERSION NPTL 0.60 [root@db49 mysql]# more db49.err 070402 14:15:48 mysqld started 070402 14:15:49 InnoDB: Started; log sequence number 0 3662762173 /usr/sbin/mysqld: ready for connections. Version: '4.1.22-standard' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Edition - Standard (GPL) 070402 14:15:49 [Note] Slave SQL thread initialized, starting replication in log 'binary-log.000109' at position 770739272, relay log './relay-bin.000 325' position: 760043807 070402 14:15:49 [Note] Slave I/O thread: connected to master 'repl@db35:3306', replication started in log 'binary-log.000109' at position 770760530 070404 1:15:02 [ERROR] Got error 127 when reading table './adventist_site/billing_complete' 070404 1:15:11 [ERROR] Got error 127 when reading table './black_site/billing_complete' 070404 1:15:12 [ERROR] Got error 127 when reading table './interracial_site/billing_complete' 070404 1:15:14 [ERROR] Got error 127 when reading table './jewish_site/billing_complete' 070404 1:15:15 [ERROR] Got error 127 when reading table './latin_site/billing_complete' 070404 1:15:15 [ERROR] Got error 127 when reading table './bbw_site/billing_complete' 070404 1:15:16 [ERROR] Got error 127 when reading table './military_site/billing_complete' 070404 1:15:16 [ERROR] Got error 127 when reading table './christian_site/billing_complete' 070404 1:15:16 [ERROR] Got error 127 when reading table './lds_site/billing_complete' 070404 1:15:17 [ERROR] Got error 127 when reading table './singleparent_site/billing_complete' 070404 1:15:18 [ERROR] Got error 127 when reading table './ldsso_site/billing_complete' 070404 10:58:23 [Note] /usr/sbin/mysqld: Normal shutdown 070404 10:58:23 [Note] Slave SQL thread exiting, replication stopped in log 'binary-log.000111' at position 537452987 070404 10:58:23 [ERROR] Slave I/O thread killed while reading event 070404 10:58:23 [ERROR] Slave I/O thread exiting, read up to log 'binary-log.000111', position 537452987 070404 10:58:26 InnoDB: Starting shutdown... 070404 10:58:29 InnoDB: Shutdown completed; log sequence number 0 3662762173 070404 10:58:29 [Note] /usr/sbin/mysqld: Shutdown complete 070404 10:58:29 mysqld ended
[4 Apr 2007 19:28]
Steve Meyers
I should note that we had about 20 servers with 4.1.22 on it. The above was from just one of them.
[5 Apr 2007 16:18]
Valeriy Kravchuk
NPTL 0.60 is not any good (for MySQL, at least). Please, send the results of: uname -a getconf GNU_LIBC_VERSION Will it be possible for you to upgrade glibc, if needed?
[5 Apr 2007 16:30]
Steve Meyers
Upgrading is not really an option at this point. We're on a standard RHEL 3 system, which the binaries we use are supposedly built for (by you). We'll be upgrading to RHEL 5 in the next 3-6 months probably. [root@db49 root]# uname -a Linux db49 2.4.21-47.0.1.ELsmp #1 SMP Thu Oct 19 10:34:18 EDT 2006 x86_64 x86_64 x86_64 GNU/Linux [root@db49 root]# getconf GNU_LIBC_VERSION glibc 2.3.2
[6 Apr 2007 10:28]
Valeriy Kravchuk
Yes, surely we do provide binaries specially for RHEL3. But I want to check the difference between the systems) we use and your one (any Upgrades installed). Please, send the results of: rpm -q -a | grep -i mysql rpm -q -a | grep glibc Can you, please, also try to add: export LD_ASSUME_KERNEL=2.4.1 at the beginning of the script used to start MySQL server and check if it will influence thew problem with errors 127 in any way?
[6 Apr 2007 16:25]
Steve Meyers
I have re-upgraded to 4.1.22 on one system, and added "export LD_ASSUME_KERNEL=2.4.1" to the top of the init script and restarted it. I will report back either later today or Monday on it. [smeyers@db49 smeyers]$ rpm -q -a | grep -i mysql MySQL-conf-1.0-5.15 MySQL-shared-standard-4.1.22-0.rhel3 MySQL-client-standard-4.1.22-0.rhel3 MySQL-server-standard-4.1.22-0.rhel3 [smeyers@db49 smeyers]$ rpm -q -a | grep glibc glibc-common-2.3.2-95.44 glibc-2.3.2-95.44 glibc-2.3.2-95.44
[6 Apr 2007 23:20]
Steve Meyers
It appears that we are not getting errors with 4.1.22 if the LD_ASSUME_KERNEL is set. I should point out that we did not get errors at all with 4.1.21, regardless of that setting.
[7 Apr 2007 6:19]
Valeriy Kravchuk
That proves that NPTL version (0.60) is a problem. With that variable set you switched MySQL server to LinuxThreads usage. I can not tell you for sure what exact fix from those listed at http://dev.mysql.com/doc/refman/4.1/en/news-4-1-22.html lead to this problem in your case (as we still need a repeatable test case), but NPTL 0.60 was never good for MySQL, especially on 64-bit platforms. As we have a workaround, I set this bug report to S3. Please, inform me if you'll get any similar problem with LD_ASSUME_KERNEL set.
[9 Apr 2007 14:18]
Steve Meyers
I was wrong. We did see some error 127s over the weekend.
[9 Apr 2007 15:09]
Valeriy Kravchuk
Please, send SHOW CREATE TABLE and SHOW TABLE STATUS results for all the tables you got error 127 for.
[25 Apr 2007 12:46]
Dmitry Panov
I have exactly the same problem with 2 different mysql versions running in significantly different environments. First server is running debian sarge: Linux hostinghi 2.6.8-2-686-smp #1 SMP Thu May 19 17:27:55 JST 2005 i686 GNU/Linux glibc 2.3.2 NPTL 0.60 The second server is running debian etch: Linux hosting3 2.6.18-4-amd64 #1 SMP Wed Feb 21 14:29:38 UTC 2007 x86_64 GNU/Linux glibc 2.3.6 NPTL 2.3.6 The table in question is quite large (3GB+ data and 4GB+ full-text index). MySQL 5.0.16 (static, downloaded from mysql.com) is running fine on the first server. However version 5.0.37 (static and glibc 2.3 downloaded from the site) running on the same server as well as debian etch version (based on 5.0.32) running on the second server give random "error 127". No crash occurs and a subsequent query may succeed. The error message looks as follows: 070422 7:01:07 [ERROR] Got error 127 when reading table './intotalnews/Story_Fulltext'
[25 Apr 2007 20:03]
Dmitry Panov
Just tried dropping the full-text index. It didn't solve the problem. Should I open a new bug for this?
[29 Apr 2007 17:09]
Dmitry Panov
ok, that's what I'll do
[9 May 2007 23:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".
[10 May 2007 4:52]
Steve Meyers
mysql> show create table archive_messages\G *************************** 1. row *************************** Table: archive_messages Create Table: CREATE TABLE `archive_messages` ( `messageid` int(10) unsigned NOT NULL auto_increment, `boardid` int(10) unsigned NOT NULL default '0', `parent` int(10) unsigned NOT NULL default '0', `thread` int(10) unsigned NOT NULL default '0', `subject` varchar(255) NOT NULL default '', `userid` int(10) unsigned NOT NULL default '0', `username` varchar(26) NOT NULL default '', `gender` tinyint(3) unsigned NOT NULL default '0', `posted_time` datetime NOT NULL default '0000-00-00 00:00:00', `views` mediumint(8) unsigned NOT NULL default '0', `reply_count` mediumint(8) unsigned NOT NULL default '0', `last_viewed_time` datetime NOT NULL default '0000-00-00 00:00:00', `last_edited_time` datetime NOT NULL default '0000-00-00 00:00:00', `last_replied_time` datetime NOT NULL default '0000-00-00 00:00:00', `last_replied_username` varchar(26) NOT NULL default '', `last_replied_messageid` int(10) unsigned NOT NULL default '0', `sticky` tinyint(1) NOT NULL default '0', `last_child` tinyint(1) NOT NULL default '0', `status` tinyint(3) unsigned NOT NULL default '1', `real_user` tinyint(3) unsigned NOT NULL default '1', `master_update_ymdt` datetime NOT NULL default '0000-00-00 00:00:00', PRIMARY KEY (`messageid`), KEY `parent` (`parent`,`thread`), KEY `userid` (`userid`), KEY `thread` (`thread`), KEY `boardid` (`boardid`), FULLTEXT KEY `username` (`username`,`subject`) ) ENGINE=MyISAM AUTO_INCREMENT=1352617 DEFAULT CHARSET=utf8 1 row in set (0.00 sec) mysql> show create table archive_message_bodies\G *************************** 1. row *************************** Table: archive_message_bodies Create Table: CREATE TABLE `archive_message_bodies` ( `messageid` int(10) unsigned NOT NULL default '0', `boardid` int(10) unsigned NOT NULL default '0', `thread` int(10) unsigned NOT NULL default '0', `body` text NOT NULL, PRIMARY KEY (`messageid`), FULLTEXT KEY `body` (`body`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 1 row in set (0.00 sec)
[2 Aug 2007 13:28]
Roald van Loon
Hello, I'd like to re-open this bug as I have seen exactly the same error since we upgraded our database replication-slave servers from 4.1.21 to 4.1.22. All replication-slave servers are: slave50 ~ # uname -a Linux slave50 2.6.17-gentoo-r8 #1 SMP Wed Oct 18 17:04:43 CEST 2006 x86_64 Intel(R) Xeon(R) CPU 5140 @ 2.33GHz GNU/Linux With 16Gb of ram, serving 132 tables (~15GB data, ~11GB index) with MyISAM SE. All running NPTL (and glibc) 2.5, and all running approx 1000~2000 q/sec and 50~100 conn/sec. After running a REPAIR when we hit an error-127, we notice a dataloss of a few records compared to the (InnoDB) master.
[7 Aug 2007 6:48]
MySQL Verification Team
Hi! I think we should wait for bug #29838 to be pushed, then retest. Does disabling concurrent inserts makes any differences to the frequency of 127 errors?
[7 Aug 2007 6:52]
MySQL Verification Team
Hi again! Does your application ever receive bogus "table is marked as crashed" error messages (corresponding to the 127 in the logs) ? Which queries return these errors? Are they queries which could possibly access data that has been very recently inserted? Such as SELECT .. MAX(auto_inc_col) .. ?
[7 Aug 2007 9:16]
Roald van Loon
Hi, All databases have one thread inserting data (replication slave sql thread), and a lot of threads accessing data (reads) especially recently added data. As a matter of fact, the tables which produce these errors are very frequently being accessed by aggregating functions (mostly count()'s). The most interesting part of this issue is probably the fact that none of the issues occur in 4.1.21, but occur in 4.1.22.
[7 Aug 2007 12:10]
Roald van Loon
Btw; table never get marked as crashed.
[9 Aug 2007 15:14]
Steve Meyers
We just upgraded to 5.0.45 from 4.1.21, and the problem has started happening again. Instead of error 127, though, we get: Table 'b' is marked as crashed and should be repaired Same tables as last time. In the error log, we see the 127s: 070809 7:58:17 [ERROR] Got error 127 when reading table './christian_boards/archive_messages' 070809 8:13:41 [ERROR] Got error 127 when reading table './christian_boards/archive_messages' 070809 8:29:22 [ERROR] Got error 127 when reading table './black_boards/archive_messages' 070809 8:35:21 [ERROR] Got error 127 when reading table './christian_boards/archive_messages' 070809 8:46:11 [ERROR] Got error 127 when reading table './lds_boards/archive_message_bodies' Again, the table is just fine. The table does not need to be repaired, and the same query works perfectly if run again.
[5 Nov 2007 13:31]
Arnaud d'Ussel
Hi, I would like to add something about this bug as we have exactly the same problem with our 5.0.32 version of mysql, running on Linux netprod-sgbd1n 2.6.18-4-amd64 #1 SMP Fri May 4 00:37:33 UTC 2007 x86_64 GNU/Linux, NPTL 2.3. We have : table is marked as crashed as result of mysql_query and 071105 12:34:53 [ERROR] Got error 127 when reading table './unerencontre/ur_user_account' in the mysql-error.log it appears only on table very frequently used, from 150 Mo to 2Go. Most of the time, the check table is ok, but sometimes, the table is corrupted and we have to repari it.
[10 Dec 2007 4:56]
Valeriy Kravchuk
Please, try to repeat with a new version, 5.0.51, and inform about the results.
[11 Jan 2008 0:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".
[14 Mar 2008 16:56]
Mihai Costache
same problem here MySQL 4.1.22 OS = Gentoo Linux uname -a Linux 2.6.22-hardened-r8 #1 SMP i686 Intel(R) Xeon(R) CPU 5160 @ 3.00GHz GenuineIntel GNU/Linux getconf GNU_LIBPTHREAD_VERSION NPTL 2.6.1 getconf GNU_LIBC_VERSION glibc 2.6.1 mysql log: 080212 7:57:44 InnoDB: Started; log sequence number 1 3547151937 /usr/sbin/mysqld: ready for connections. Version: '4.1.22-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Gentoo Linux mysql-4.1.22-r1 ........ 080231 20:50:45 [ERROR] Got error 127 when reading table './db-name/table1_name' ........ 080305 18:28:11 [ERROR] Got error 127 when reading table './db-name/table2_name' ........ a few errors like these from above per day .... until now no error like "table is marked as crashed" in our logs until now all tables are OK on CHECK and ANALYZE TABLE checking upgrade to 5.0.51a is no option for us now ... maybe in one year my.cnf: [client] port = 3306 socket = /var/run/mysqld/mysqld.sock [mysql] character-sets-dir=/usr/share/mysql/charsets default-character-set=utf8 [mysqladmin] character-sets-dir=/usr/share/mysql/charsets default-character-set=utf8 [mysqlcheck] character-sets-dir=/usr/share/mysql/charsets default-character-set=utf8 [mysqldump] character-sets-dir=/usr/share/mysql/charsets default-character-set=utf8 [mysqlimport] character-sets-dir=/usr/share/mysql/charsets default-character-set=utf8 [mysqlshow] character-sets-dir=/usr/share/mysql/charsets default-character-set=utf8 [myisamchk] character-sets-dir=/usr/share/mysql/charsets [myisampack] character-sets-dir=/usr/share/mysql/charsets [mysqld_safe] err-log = /var/log/mysql/mysql.err [mysqld] max_connections = 200 character-set-server = utf8 default-character-set = utf8 user = mysql port = 3306 socket = /var/run/mysqld/mysqld.sock pid-file = /var/run/mysqld/mysqld.pid log-error = /var/log/mysql/mysqld.err log-slow-queries = /var/log/mysql/mysql_slow_queries long_query_time = 3 basedir = /usr datadir = /var/lib/mysql skip-locking skip-name-resolve tmp_table_size = 64M key_buffer = 256M max_allowed_packet = 16M table_cache = 512 sort_buffer_size = 1M join_buffer_size = 1M read_buffer_size = 1M read_rnd_buffer_size = 1M bulk_insert_buffer_size = 8M myisam_sort_buffer_size = 16M language = /usr/share/mysql/english thread_cache_size = 16 thread_concurrency = 4 bind-address = 0.0.0.0 log-bin = /var/lib/mysql/master-binary-log server-id = 1 max_binlog_size = 100M binlog-ignore-db = test tmpdir = /tmp/ innodb_buffer_pool_size = 256M innodb_additional_mem_pool_size = 4M innodb_data_home_dir = /var/lib/mysql/ innodb_log_arch_dir = /var/lib/mysql/ innodb_log_group_home_dir = /var/lib/mysql/ innodb_data_file_path = ibdata1:512M;ibdata2:512M;ibdata3:512M;ibdata4:512M;ibdata5:512M;ibdata6:10M:autoextend innodb_log_file_size = 64M innodb_log_buffer_size = 10M set-variable = innodb_log_files_in_group=2 innodb_flush_log_at_trx_commit = 1 innodb_lock_wait_timeout = 50 innodb_thread_concurrency = 8 [mysqldump] quick max_allowed_packet = 16M [mysql] [isamchk] key_buffer = 32M sort_buffer_size = 32M read_buffer = 2M write_buffer = 2M [myisamchk] key_buffer = 32M sort_buffer_size = 32M read_buffer = 2M write_buffer = 2M [mysqlhotcopy] interactive-timeout