MySQL Bugs: #59648: my_strtoll10_mb2: Assertion `(*endptr

Bug #59648	my_strtoll10_mb2: Assertion `(*endptr - s) % 2 == 0' failed.
Submitted:	21 Jan 2011 7:52	Modified:	6 Apr 2011 17:55
Reporter:	Shane Bester (Platinum Quality Contributor)	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Charsets	Severity:	S1 (Critical)
Version:	mysql-trunk-debug	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
Version: '5.6.2-m5-valgrind-max-debug'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
[New Thread 0x7fffe3f98710 (LWP 343)]
mysqld: ./strings/ctype-ucs2.c:733: my_strtoll10_mb2: Assertion `(*endptr - s) % 2 == 0' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe3f98710 (LWP 343)]
0x00000030e14329a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64        return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0  in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  in abort () at abort.c:92
#2  in __assert_fail at assert.c:81
#3  in my_strtoll10_mb2  at ./strings/ctype-ucs2.c:733
#4  in Item_func_signed::val_int_from_str at ./sql/item_func.cc:971
#5  in Item_func_unsigned::val_int at ./sql/item_func.cc:1039
#6  in Item::send at ./sql/item.cc:5968
#7  in Protocol::send_result_set_row at ./sql/protocol.cc:848
#8  in select_send::send_data at ./sql/sql_class.cc:1866
#9  in JOIN::exec at ./sql/sql_select.cc:2794
#10 in mysql_select at ./sql/sql_select.cc:3554
#11 in handle_select at ./sql/sql_select.cc:323
#12 in execute_sqlcom_select at ./sql/sql_parse.cc:4513
#13 in mysql_execute_command at ./sql/sql_parse.cc:2096
#14 in mysql_parse  at ./sql/sql_parse.cc:5550
#15 in dispatch_command at ./sql/sql_parse.cc:1078
#16 in do_command at ./sql/sql_parse.cc:815
#17 in do_handle_one_connection at ./sql/sql_connect.cc:748
#18 in handle_one_connection at ./sql/sql_connect.cc:684
#19 in start_thread at pthread_create.c:301
#20 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) frame 3
#3  in my_strtoll10_mb2 at ./strings/ctype-ucs2.c:733
733         DBUG_ASSERT((*endptr - s) % 2 == 0);
(gdb) p *endptr-s
$1 = 9

5.5.10-debug didn't crash.

How to repeat:
#on debug build of mysql-trunk
select convert(des_encrypt(0,char('1' using ucs2)),unsigned);

Suggested fix:
maybe review fix for bug #59145 ?

Thank you for the bug report. Verified with current mysql-trunk on Mac OS X:

macbook-pro:trunk openxs$ bin/mysql -uroot test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.2-m5-debug Source distribution

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> select convert(des_encrypt(0,char('1' using ucs2)),unsigned);
ERROR 2013 (HY000): Lost connection to MySQL server during query
mysql> 110121 18:06:52 mysqld_safe mysqld restarted

mysql> exit
Bye
macbook-pro:trunk openxs$ tail -80 data/macbook-pro.err 
...
ysqld: ready for connections.
Version: '5.6.2-m5-debug'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
Assertion failed: ((*endptr - s) % 2 == 0), function my_strtoll10_mb2, file /Users/openxs/bzr2/mysql-trunk-work/strings/ctype-ucs2.c, line 733. 
...

please add another testcase to the sweet:

select convert(quote(char(0xf5 using ucs2)),signed);

another testcase to test the fix when available:

select char(coalesce(null,char(count('%s') using ucs2),1,@@global.license,null) using cp850);

Noted in 5.1.57, 5.5.11, 5.6.2 changelogs.

An incorrect character set pointer passed to my_strtoll10_mb2()
caused an assertion to be raised.

CHANGESET - http://lists.mysql.com/commits/132371