MySQL Bugs: #73979: wrong stack size calculation leads to stack overflow in pinbox allocator

Bug #73979	wrong stack size calculation leads to stack overflow in pinbox allocator
Submitted:	19 Sep 2014 10:37	Modified:	20 Feb 2015 18:12
Reporter:	Vlad Lesin	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Performance Schema	Severity:	S3 (Non-critical)
Version:	5.5, 5.6	OS:	Any
Assigned to:	Marc ALFF	CPU Architecture:	Any

Description:
1) The value in pins->stack_ends_here is actually incorrect. This value is calculated when the thread is initialized in my_thread_init(). The problem is that the value doesn't take into account existing stack usage and just adds the thread stack size to calculate the beginning of the stack.

2) _lf_pinbox_real_free() is using alloca() in a very unsafe way. alloca() should only be used by leaf functions, since you cannot predict the amount of stack space that non-leaf functions will use. As pins->stack_ends_here is calculated in wrong way and there is no window for the qsort() which is invoked from _lf_pinbox_real_free() after alloca(), stack overflow can take place.

See this https://bugs.launchpad.net/percona-server/+bug/1351148 analysis for more detailed description.

How to repeat:
There is not test case to repeat it.

Suggested fix:
1) Take into accout the current stack offset when stack size is calculated in my_thread_init();

2) Don't allocate the whole free stack space in _lf_pinbox_real_free(), take some window for qsort();

See this https://code.launchpad.net/~vlad-lesin/percona-server/5.6-553-bug-1351148 patch.

Hello and thank you for your questions.

First of all, the function (my_thread_init) is called immediately upon the entry to the new thread, so there is no stack usage, or it is very, very small. Insignificantly small.

Regarding alloca(), the function that you are mentioning is not using it. But, yes, it is used in other places. We took care to use it only for pointers for which we know that can not be large. Where we were unsure, we inserted stack size checks. Also, the return from the function is ALWAYS checked and the appropriate action is taken. So far, we did not have any reports of problems with this setup. Where we had, we inserted stack size checks and users would increase the thread stack size. There are practically no current reports on the problem in this area. I am the one who was involved in writing this part of code (as well) and as I am with MySQL for 16 years, I would have remembered any problems in this area.

I am leaving this bug in "Need Feedback" status, as it is always possible that we missed something.

Sinisa, are you saying that _lf_pinbox_real_free is _not_ calling alloca followed by qsort?

Actually, it does. Simply, I was not directed to the proper source.

Your bug is justified. I can not give it any higher priority, since the usage of concurrent hashes is rather scarce. MUCH more important then that, chances of entering into problems due to this miscalculation are minutely small.

Verified as described.

Thanks for the report.

Noted in 5.7.6 changelog.

Miscalculation of memory requirements for qsort operations could
result in stack overflow errors in situations with a large number of
concurrent server connections.