Bug #66631 MySQL-Proxy crash under multi-thread
Submitted: 31 Aug 2012 9:34 Modified: 26 Dec 2012 12:42
Reporter: Yang Yue Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Proxy Severity:S1 (Critical)
Version:0.8.2, 0.8.3 OS:Linux
Assigned to: CPU Architecture:Any
Tags: crash
Triage: Needs Triage: D1 (Critical)

[31 Aug 2012 9:34] Yang Yue
Description:
MySQL-Proxy will crash when it is started in multi-thread mode.

How to repeat:
1. set proxy-lua-script = test.lua
2. set event-threads = 32
3. start mysql-proxy
4. send sql query to mysql-proxy with high concurrency

test.lua
function read_query( packet )
        proxy.queries:append(1, packet, { resultset_is_needed = true })
        proxy.connection.backend_ndx = 1 
        return proxy.PROXY_SEND_QUERY
end

function read_query_result( inj ) 
        proxy.connection.backend_ndx = 0 
end
[31 Aug 2012 9:43] Yang Yue
read_query_result in test.lua
proxy.connection.backend_ndx = 0

src/network-conn-pool-lua.c:206
int network_connection_pool_lua_add_connection(network_mysqld_con *con)

src/network-conn-pool-lua.c:220
chassis_event_add_local(con->srv, &(con->server->event));

src/chassis-event-thread.c:211
void chassis_event_add_local(chassis *chas, struct event *ev)

src/chassis-event-thread.c:192
void chassis_event_add_local_with_timeout(chassis G_GNUC_UNUSED *chas, struct event *ev, struct timeval *tv)

src/chassis-event-thread.c:206
chassis_event_op_apply(op, event_base);
[18 Dec 2012 18:54] Sveta Smirnova
Thank you for the report.

Verified as described. For load I used file which will be attached in few seconds. I tried to run it in several times in parallel terminals.
[18 Dec 2012 18:55] Sveta Smirnova
test case

Attachment: bug66631.cpp (application/octet-stream, text), 1.56 KiB.

[26 Dec 2012 12:42] Yang Yue
Hi, I have fixed this bug by myself about two months ago.

Function chassis_event_add_local in chassis-event-thread.c,the event_base to remove the timeout connections is one of the threads' event_base.

I create a new thread named remove_thread, it creates an event_base named remove_base, all EV_READ in timeout connections are added to this event_base, then mysql-proxy won't crash.

But I don't know why this method can avoid crash clearly, maybe the ping event and the timeout event in one event_base lead to conflict?
[16 Jan 2013 17:33] Sveta Smirnova
More testing found this is repeatable on Oracle Linux 6 and not repeatable on Red Hat 5, Solaris and few other OSes