[Mon Aug 08 16:44:23 root@dc1stagedb03:/opt/mysql-data/stagedb03b/logs ] # cat mysqld.log
110806 21:52:29 mysqld_safe Starting mysqld daemon with databases from /opt/mysql-data/stagedb03b/data
110806 21:52:29 [Note] Plugin 'BLACKHOLE' is disabled.
110806 21:52:29 [Note] Plugin 'FEDERATED' is disabled.
110806 21:52:29  InnoDB: Initializing buffer pool, size = 30.0G
110806 21:52:40  InnoDB: Completed initialization of buffer pool
110806 21:52:43  InnoDB: Started; log sequence number 99 1777751779
110806 21:52:43 [Note] Event Scheduler: Loaded 0 events
110806 21:52:43 [Note] /opt/mysql/mysql-advanced-5.1.58-solaris10-x86_64/bin/mysqld: ready for connections.
Version: '5.1.58-enterprise-commercial-advanced-log'  socket: '/opt/mysql-data/stagedb03b/logs/mysqld.sock'  port: 3306  MySQL Enterprise Server - Advanced Edition (Commercial)
110808 10:15:52 [ERROR] /opt/mysql/mysql-advanced-5.1.58-solaris10-x86_64/bin/mysqld: Out of memory (Needed 5100000 bytes)
110808 10:15:52 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
110808 11:49:51 [ERROR] /opt/mysql/mysql-advanced-5.1.58-solaris10-x86_64/bin/mysqld: Out of memory (Needed 5116320 bytes)
110808 11:49:51 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
110808 11:55:19 [Note] /opt/mysql/mysql-advanced-5.1.58-solaris10-x86_64/bin/mysqld: Normal shutdown

110808 11:55:19 [Note] Event Scheduler: Purging the queue. 0 events
110808 11:55:21  InnoDB: Starting shutdown...
110808 11:55:27  InnoDB: Shutdown completed; log sequence number 129 782093679
110808 11:55:29 [Note] /opt/mysql/mysql-advanced-5.1.58-solaris10-x86_64/bin/mysqld: Shutdown complete

110808 11:55:42 mysqld_safe mysqld from pid file /opt/mysql-data/stagedb03b/logs/mysqld.pid ended
110808 11:56:17 mysqld_safe Starting mysqld daemon with databases from /opt/mysql-data/stagedb03b/data
110808 11:56:17 [Note] Plugin 'BLACKHOLE' is disabled.
110808 11:56:17 [Note] Plugin 'FEDERATED' is disabled.
110808 11:56:18  InnoDB: Initializing buffer pool, size = 30.0G
110808 11:56:29  InnoDB: Completed initialization of buffer pool
110808 11:56:31  InnoDB: Started; log sequence number 129 782093679
110808 11:56:31 [Note] Event Scheduler: Loaded 0 events
110808 11:56:31 [Note] /opt/mysql/mysql-advanced-5.1.58-solaris10-x86_64/bin/mysqld: ready for connections.
Version: '5.1.58-enterprise-commercial-advanced-log'  socket: '/opt/mysql-data/stagedb03b/logs/mysqld.sock'  port: 3306  MySQL Enterprise Server - Advanced Edition (Commercial)
110808 12:00:43 [ERROR] /opt/mysql/mysql-advanced-5.1.58-solaris10-x86_64/bin/mysqld: Out of memory (Needed 5769120 bytes)
110808 12:00:43 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
110808 12:02:43 [ERROR] /opt/mysql/mysql-advanced-5.1.58-solaris10-x86_64/bin/mysqld: Out of memory (Needed 5434560 bytes)
110808 12:02:43 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
110808 12:02:52 - mysqld got signal 10 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.

key_buffer_size=2147483648
read_buffer_size=2097152
max_used_connections=13
max_threads=1000
threads_connected=13
It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 12347370 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong...
/opt/mysql/mysql-advanced-5.1.58-solaris10-x86_64/bin/mysqld:my_print_stacktrace+0x20
/opt/mysql/mysql-advanced-5.1.58-solaris10-x86_64/bin/mysqld:handle_segfault+0x2c3
/lib/amd64/libc.so.1:0xdc6c6
/lib/amd64/libc.so.1:0xd0fa2
/lib/amd64/libc.so.1:0xdc2ce [ Signal 10 (BUS)]
/lib/amd64/libc.so.1:0xd87b5
/lib/amd64/libc.so.1:pthread_create+0xfe
/opt/mysql/mysql-advanced-5.1.58-solaris10-x86_64/bin/mysqld:__1cbIcreate_thread_to_handle_connection6FpnDTHD__v_+0x8d
/opt/mysql/mysql-advanced-5.1.58-solaris10-x86_64/bin/mysqld:0x318f02
/opt/mysql/mysql-advanced-5.1.58-solaris10-x86_64/bin/mysqld:handle_connections_sockets+0x3b9
/opt/mysql/mysql-advanced-5.1.58-solaris10-x86_64/bin/mysqld:main+0x425
/opt/mysql/mysql-advanced-5.1.58-solaris10-x86_64/bin/mysqld:0x25fccc
Please read http://dev.mysql.com/doc/refman/5.1/en/resolve-stack-dump.html
and follow instructions on how to resolve the stack trace.
Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.
110808 12:02:53 mysqld_safe mysqld restarted
110808 12:02:53 [Note] Plugin 'BLACKHOLE' is disabled.
110808 12:02:53 [Note] Plugin 'FEDERATED' is disabled.
110808 12:02:54  InnoDB: Initializing buffer pool, size = 30.0G
110808 12:03:05  InnoDB: Completed initialization of buffer pool
InnoDB: Log scan progressed past the checkpoint lsn 129 825721258
110808 12:03:06  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 129 830963712
InnoDB: Doing recovery: scanned up to log sequence number 129 836206592
InnoDB: Doing recovery: scanned up to log sequence number 129 841449472
InnoDB: Doing recovery: scanned up to log sequence number 129 846692352
InnoDB: Doing recovery: scanned up to log sequence number 129 851935232
InnoDB: Doing recovery: scanned up to log sequence number 129 857178112
InnoDB: Doing recovery: scanned up to log sequence number 129 862420992
InnoDB: Doing recovery: scanned up to log sequence number 129 867663872
InnoDB: Doing recovery: scanned up to log sequence number 129 869194817
110808 12:03:10  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
110808 12:03:26  InnoDB: Started; log sequence number 129 869194817
110808 12:03:26 [Note] Event Scheduler: Loaded 0 events
110808 12:03:26 [Note] /opt/mysql/mysql-advanced-5.1.58-solaris10-x86_64/bin/mysqld: ready for connections.
Version: '5.1.58-enterprise-commercial-advanced-log'  socket: '/opt/mysql-data/stagedb03b/logs/mysqld.sock'  port: 3306  MySQL Enterprise Server - Advanced Edition (Commercial) [Mon Aug 08 16:44:24 root@dc1stagedb03:/opt/mysql-data/stagedb03b/logs ] #