MySQL Bugs: #66631: MySQL-Proxy crash under multi-thread

Bug #66631	MySQL-Proxy crash under multi-thread
Submitted:	31 Aug 2012 9:34	Modified:	26 Dec 2012 12:42
Reporter:	Yang Yue	Email Updates:
Status:	Verified	Impact on me:	None
Category:	MySQL Proxy	Severity:	S1 (Critical)
Version:	0.8.2, 0.8.3	OS:	Linux
Assigned to:		CPU Architecture:	Any
Tags:	crash

Description:
MySQL-Proxy will crash when it is started in multi-thread mode.

How to repeat:
1. set proxy-lua-script = test.lua
2. set event-threads = 32
3. start mysql-proxy
4. send sql query to mysql-proxy with high concurrency

test.lua
function read_query( packet )
        proxy.queries:append(1, packet, { resultset_is_needed = true })
        proxy.connection.backend_ndx = 1 
        return proxy.PROXY_SEND_QUERY
end

function read_query_result( inj ) 
        proxy.connection.backend_ndx = 0 
end

read_query_result in test.lua
proxy.connection.backend_ndx = 0

src/network-conn-pool-lua.c:206
int network_connection_pool_lua_add_connection(network_mysqld_con *con)

src/network-conn-pool-lua.c:220
chassis_event_add_local(con->srv, &(con->server->event));

src/chassis-event-thread.c:211
void chassis_event_add_local(chassis *chas, struct event *ev)

src/chassis-event-thread.c:192
void chassis_event_add_local_with_timeout(chassis G_GNUC_UNUSED *chas, struct event *ev, struct timeval *tv)

src/chassis-event-thread.c:206
chassis_event_op_apply(op, event_base);

Thank you for the report.

Verified as described. For load I used file which will be attached in few seconds. I tried to run it in several times in parallel terminals.

test case

Attachment: bug66631.cpp (application/octet-stream, text), 1.56 KiB.

Hi, I have fixed this bug by myself about two months ago.

Function chassis_event_add_local in chassis-event-thread.c，the event_base to remove the timeout connections is one of the threads' event_base.

I create a new thread named remove_thread, it creates an event_base named remove_base, all EV_READ in timeout connections are added to this event_base, then mysql-proxy won't crash.

But I don't know why this method can avoid crash clearly, maybe the ping event and the timeout event in one event_base lead to conflict?

More testing found this is repeatable on Oracle Linux 6 and not repeatable on Red Hat 5, Solaris and few other OSes