Bug #36983 stack corruption and crash with des_encrypt crashes on invalid inputs
Submitted: 26 May 2008 15:34 Modified: 22 Sep 2008 16:38
Reporter: Shane Bester (Platinum Quality Contributor) Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: DML Severity:S1 (Critical)
Version:5.1.26bk OS:Any
Assigned to: CPU Architecture:Any
Tags: DES_ENCRYPT, regression

[26 May 2008 15:34] Shane Bester
Description:
got a new crash with des_encrypt with corrupted stack trace.

How to repeat:
select des_encrypt(0xe8877854e034b0502fd8908edd480c7192c02a9bf182dcf160006f57d5b26806381ecf6acfd46b7ed690401c37aa922274b794403a5dcde84c6e703412484498a154d3640ab095a72a3363140eafe5ff4b295abcf8aa101fde969de0c3fa3c0f405cc4519cfa664f7a2cf888b8,0xac90f09431e289748358fe9000ee10ca5f7e5388281ca8c0327c24558fa0b0b5940419779a5a82d272a40a0328f840063e244012a86e4050f470f0e0907bc056609680181010);
[26 May 2008 15:35] MySQL Verification Team
please make sure the above sql is all on 1 line when running it.  silly wrapping.
[26 May 2008 18:32] Sveta Smirnova
Thank you for the report.

I can not repeat described behavior nor on Mac neither on Linux. Which operation system do you use? Do yo paste the statement into command line client?
[26 May 2008 18:48] MySQL Verification Team
paste into cmdline client all on one line.  this is with the -debug binaries from todays BK.  when run under valgrind, I got illegal instruction errors.. OS is:
[sbester@box1 6.0]$ cat /proc/version 
Linux version 2.6.18-1.2798.fc6xen (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 4.1.1 20061011 (Red Hat 4.1.1-30)) #1 SMP Mon Oct 16 15:11:19 EDT 2006
[
[26 May 2008 20:23] MySQL Verification Team
I could not repeat the crash on FC8 64-bit with latest source and debug build.
[27 May 2008 5:49] MySQL Verification Team
assigning back to myself to make better testcase \o/
[27 May 2008 8:45] MySQL Verification Team
using same source code on a windows build, i cannot repeat the crash, even after hundreds of thousands of tests.  But in older linux builds on the same box this crashes happens now, the SQL doesn't crash, so there's some explanation required somewhere.