Description:
2016-08-29 11:33:44 38700 [ERROR] /usr/sbin/mysqld: Sort aborted: Error writing file '/scratch/mysql_tmp/MY3DVh8D' (Errcode: 28 - No space left on device)
2016-08-29 11:33:44 38700 [ERROR] Invalid (old?) table or database name 'lost+found'
2016-08-29 11:33:47 38700 [ERROR] /usr/sbin/mysqld: Incorrect key file for table '/scratch/mysql_tmp/#sql_972c_11.MYI'; try to repair it
2016-08-29 11:33:47 38700 [ERROR] Got an error from unknown thread, /pb2/build/sb_0-9307864-1369151250.06/rpm/BUILD/mysqlcom-pro-5.6.12/mysqlcom-pro-5.6.12/storage/myisam/mi_write.c:226
2016-08-29 11:33:47 38700 [ERROR] /usr/sbin/mysqld: Incorrect key file for table '/scratch/mysql_tmp/#sql_972c_18.MYI'; try to repair it
2016-08-29 11:33:47 38700 [ERROR] Got an error from unknown thread, /pb2/build/sb_0-9307864-1369151250.06/rpm/BUILD/mysqlcom-pro-5.6.12/mysqlcom-pro-5.6.12/storage/myisam/mi_write.c:226
2016-08-29 11:33:48 38700 [ERROR] /usr/sbin/mysqld: Incorrect key file for table '/scratch/mysql_tmp/#sql_972c_17.MYI'; try to repair it
2016-08-29 11:33:48 38700 [ERROR] Got an error from unknown thread, /pb2/build/sb_0-9307864-1369151250.06/rpm/BUILD/mysqlcom-pro-5.6.12/mysqlcom-pro-5.6.12/storage/myisam/mi_write.c:226
2016-08-29 11:33:51 38700 [Warning] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statement is unsafe because it invokes a trigger or a stored function that inserts into an AUTO_INCREMENT column. Inserted values cannot be logged correctly. Statement: update VISIT_CHECKIN set ADHOC_VISIT=null, appointment_id=2992569, appointment_time='1970-01-01 10:50:00', APPT_COMMENTS='1 yr', DOC_SPECIALITY='OPTOM', bin_time='1970-01-01 10:54:00', CHECK_OUT_TIME=null, checked_in_by_userid='pwhite', checked_in_by_username='white patrice', COMMENTS='', consultation_end_time='1970-01-01 11:33:00', consultation_start_time='1970-01-01 11:08:00', IS_DELETED=0, discharge_ma=null, DISCHARGE_MA_CODE=null, doctor_code='TC2', doctor_id=123819, DRUG_HISTORY_VIEWED=null, ELIGIB_STATUS=null, first_visit=0, HIGH_RISK=1, insurance_code='HG', insurance_id=12464, MEDICAL_ASST_CODE='MD', MA_ROOM_IN_TIME=null, medical_assistant=2449, OFFICE_ID=10083, offsite_visit=null, patient_daily_no=74, PAT_FULL_NAME='Hodge, Carla', PATIEN
2016-08-29 11:33:51 38700 [Warning] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statements writing to a table with an auto-increment column after selecting from another table are unsafe because the order in which rows are retrieved determines what (if any) rows will be written. This order cannot be predicted and may differ on master and the slave. Statement: update VISIT_CHECKIN set ADHOC_VISIT=null, appointment_id=2992569, appointment_time='1970-01-01 10:50:00', APPT_COMMENTS='1 yr', DOC_SPECIALITY='OPTOM', bin_time='1970-01-01 10:54:00', CHECK_OUT_TIME=null, checked_in_by_userid='pwhite', checked_in_by_username='white patrice', COMMENTS='', consultation_end_time='1970-01-01 11:33:00', consultation_start_time='1970-01-01 11:08:00', IS_DELETED=0, discharge_ma=null, DISCHARGE_MA_CODE=null, doctor_code='TC2', doctor_id=123819, DRUG_HISTORY_VIEWED=null, ELIGIB_STATUS=null, first_visit=0, HIGH_RISK=1, insurance_code='HG', insurance_id=12464, MEDICAL_ASST_CODE='MD', MA_ROOM_IN_TIME=null, medica
15:33:53 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=33554432
read_buffer_size=131072
max_used_connections=1289
max_threads=1400
thread_count=822
connection_count=822
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 46106133 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x3e1063a0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
2016-08-29 11:33:54 38700 [Warning] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statements writing to a table with an auto-increment column after selecting from another table are unsafe because the order in which rows are retrieved determines what (if any) rows will be written. This order cannot be predicted and may differ on master and the slave. Statement: CALL NBSavePhysicalExamFull(918232, 13, 3, 784562, 'GU:', 13,'evelync', 1, NULL )
2016-08-29 11:33:56 38700 [ERROR] /usr/sbin/mysqld: Incorrect key file for table '/scratch/mysql_tmp/#sql_972c_5.MYI'; try to repair it
2016-08-29 11:33:56 38700 [ERROR] Got an error from unknown thread, /pb2/build/sb_0-9307864-1369151250.06/rpm/BUILD/mysqlcom-pro-5.6.12/mysqlcom-pro-5.6.12/storage/myisam/mi_write.c:226
stack_bottom = 7f9c38a5ce48 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8fef65]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x691bd4]
/lib64/libpthread.so.0[0x37fd00f710]
/usr/sbin/mysqld(_ZNK7handler22ha_statistic_incrementEM17system_status_vary+0x8)[0x5d4708]
/usr/sbin/mysqld(_ZN9ha_myisam9write_rowEPh+0x20)[0xaebe40]
/usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0xff)[0x5dcc3f]
/usr/sbin/mysqld[0x6ec7ed]
/usr/sbin/mysqld[0x6ebdd1]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1c1)[0x6ec301]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x2aa)[0x6ecb0a]
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x250)[0x72d2c0]
/usr/sbin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x187)[0x72db37]
/usr/sbin/mysqld[0x709f1d]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0xb46)[0x70be76]
/usr/sbin/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x60)[0x83bf20]
/usr/sbin/mysqld(_ZN12sp_lex_instr23reset_lex_and_exec_coreEP3THDPjb+0x220)[0x83c240]
/usr/sbin/mysqld(_ZN12sp_lex_instr29validate_lex_and_execute_coreEP3THDPjb+0x8f)[0x83d6bf]
/usr/sbin/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x1a0)[0x83e640]
/usr/sbin/mysqld(_ZN7sp_head7executeEP3THDb+0x782)[0x698622]
/usr/sbin/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x6c7)[0x699b07]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x102a)[0x70c35a]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x32f)[0x71014f]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xe14)[0x711054]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0xcf)[0x6debef]
/usr/sbin/mysqld(handle_one_connection+0x47)[0x6ded17]
/lib64/libpthread.so.0[0x37fd0079d1]
/lib64/libc.so.6(clone+0x6d)[0x37fcce88fd]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f963a8edb98): is an invalid pointer
Connection ID (thread ID): 4883063
Status: NOT_KILLED
How to repeat:
Cannot repeat : as it seems that within this issue occurring
it correlates w/ a query that generated a number of additional connections and within each connection the same query occurred with an ORDER by clause that created multiple tmp file and filled the tmp disk:
that's what I am seeing but hopefully you can shed additional light on this issue as well.
