Description:
A SIGSEGV exception caused MySQL to crash when executing a heavy (concurrent) workload against the IBMDB2I storage engine.
The stack looks like this:
ISF=2BF41090 TS=80007BF41090 NIA=D5957CCC REWIND__17IOASYNCREADBUFFERFV+0X30
ISF=2BF410E0 TS=80007BF410E0 NIA=D59589B4 NEWREADREQUEST__17IOASYNCREADBUFFERFULCUIBT1IT6+0X13C
ISF=2BF41180 TS=80007BF41180 NIA=D5912B00 DOINITIALREAD__10HA_IBMDB2IFCUIULIT4+0X124
ISF=2BF411E0 TS=80007BF411E0 NIA=D5911FB8 INDEX_READ__10HA_IBMDB2IFPUCPCUCUI16HA_RKEY_FUNCTION+0X3C8
ISF=2BF412A0 TS=80007BF412A0 NIA=D591FFF8 INDEX_READ_MAP__7HANDLERFPUCPCUCUL16HA_RKEY_FUNCTION+0X74
ISF=2BF412F0 TS=80007BF412F0 NIA=101427C0 READ_RANGE_FIRST__7HANDLERFPC12ST_KEY_RANGEPC12ST_KEY_RANGEBT3+0X17C
ISF=2BF41360 TS=80007BF41360 NIA=101421F8 READ_MULTI_RANGE_FIRST__7HANDLERFPP18ST_KEY_MULTI_RANGEP18ST_KEY_MULTI_RANGEUIBP17ST_HA +
NDLER_BUFFER+0X194
ISF=2BF413D0 TS=80007BF413D0 NIA=10012BE4 GET_NEXT__18QUICK_RANGE_SELECTFV+0X34C
ISF=2BF41460 TS=80007BF41460 NIA=103258CC RR_QUICK__FP14ST_READ_RECORD+0X90
ISF=2BF414B0 TS=80007BF414B0 NIA=1029E6F4 JOIN_INIT_READ_RECORD__FP13ST_JOIN_TABLE+0XBC
ISF=2BF41500 TS=80007BF41500 NIA=102A54E8 SUB_SELECT__FP4JOINP13ST_JOIN_TABLEB+0X13C
ISF=2BF41560 TS=80007BF41560 NIA=1029BB40 DO_SELECT__FP4JOINP4LISTXT4ITEM_P8ST_TABLEP9PROCEDURE+0X344
ISF=2BF415E0 TS=80007BF415E0 NIA=102981FC EXEC__4JOINFV+0X8FC
ISF=2BF41760 TS=80007BF41760 NIA=102872A0 MYSQL_SELECT__FP3THDPPP4ITEMP10TABLE_LISTUIR4LISTXT4ITEM_P4ITEMT4P8ST_ORDERT8T6T8ULP13S +
ELECT_RESULTP18ST_SELECT_LEX_UNITP13ST_SELECT_LEX+0X308
ISF=2BF417F0 TS=80007BF417F0 NIA=10286DF4 HANDLE_SELECT__FP3THDP6ST_LEXP13SELECT_RESULTUL+0X18C
ISF=2BF41880 TS=80007BF41880 NIA=1024EBD0 EXECUTE_SQLCOM_SELECT__FP3THDP10TABLE_LIST+0X34C
ISF=2BF41D20 TS=80007BF41D20 NIA=10246EC0 MYSQL_EXECUTE_COMMAND__FP3THD+0X6C4
ISF=2BF43080 TS=80007BF43080 NIA=102448D0 MYSQL_PARSE__FP3THDPCCUIPPCC+0X1F4
ISF=2BF43160 TS=80007BF43160 NIA=102423C0 DISPATCH_COMMAND__F19ENUM_SERVER_COMMANDP3THDPCUI+0X8E0
ISF=2BF43A50 TS=80007BF43A50 NIA=10243E40 DO_COMMAND__FP3THD+0X16C
ISF=2BF43AB0 TS=80007BF43AB0 NIA=104F5548 HANDLE_ONE_CONNECTION+0X188
ISF=2BF43B10 TS=80007BF43B10 NIA=D04DAC64 _PTHREAD_BODY+0X104
ISF=2BF43B60 TS=80007BF43B60 NIA=00000000
I believe that the root cause of the crash is a failure to detect an out-of-memory condition when the storage engine allocates a buffer for reading rows from the table.
How to repeat:
Reproducing the workload that generated this failure will be difficult. However, the bug can be simulated by forcing a malloc failure in ValidatedPointer<T>::alloc().
1. Create an IBMDB2I table
2. Insert a row into the table.
3. Using a debugger, put a breakpoint on line 91 of db2i_ValidatedPointer.h
4. Select from the table. This should hit the breakpoint.
5. Override the value of address returned by malloc_aligned to be 0.
6. Continue. This should generate a SIGSEGV and a stack similar to the one described above.
Suggested fix:
Improve the error handling in the storage engine to detect the out of memory condition and correctly report the failure without crashing.
