MySQL Bugs: #31220: SQLFetch or SQLFetchScroll returns negative data length using SQL_C

Bug #31220	SQLFetch or SQLFetchScroll returns negative data length using SQL_C_WCHAR
Submitted:	27 Sep 2007 2:22	Modified:	14 Mar 2008 19:41
Reporter:	Viktor Ferenczi
Status:	Closed
Category:	Connector/ODBC	Severity:	S2 (Serious)
Version:	3.51.20r750	OS:	Linux (Ubuntu Feisty, up-to-date)
Assigned to:	Bugs System	Target Version:
Tags:	SQLFetch, SQLFetchScroll, negative, LENGTH, string, varchar, buffer, overflow, segfault, segmentation fault, crash
Triage:	D3 (Medium)

Description:
Calling SQLFetch or SQLFetchScroll returns negative data length in certain circumstances.
I've attached the complete unixODBC trace of my test. At the end my application segfaulted
due to the unexpected negative length. The core dump has been analysed to point out the
problem. Strings are encoded with 16bit unicode characters (SQL_C_WCHAR).

CPU: Intel Core Duo T5500
Memory: 1.5Gbytes
OS: Ubuntu Feisty 7.04, up-to-date, 32 bit
MySQL: 5.0.38
ODBC manager: unixODBC 2.2.11-13
MySQL ODBC connector: mysql-connector-odbc-3.51.20r750.tar.gz
The ODBC driver is compiled from source against the development headers installed by
Ubuntu's package manager (apt).

Driver settings:

[mysql]
Description             = mysql
Driver          = /usr/local/lib/libmyodbc3.so
Driver64                = /usr/lib
Setup           = /usr/local/lib/libmyodbc3S.so
Setup64         = /usr/lib
UsageCount              = 1
CPTimeout               =
CPReuse         =

System DSN used:

[test_mysql]
Driver          = mysql
DATABASE                = test
DESCRIPTION             = mysql
PWD             = test
SERVER          = localhost
UID             = test

How to repeat:
Construct a test case to reproduce the SQL commands related to the last table used
accoring to the attached trace.

unixODBC trace output

Attachment: mysql-unixODBC-trace.log.bz2 (application/x-bzip, text), 25.33 KiB.

Test case with _W functions

Attachment: bug31220.c (text/plain), 2.73 KiB.

The test case above returns non-negative buffer length

Tested the program in Windows: got the correct length, in HP-UX the Buffer length is
1073745752.

Bogdan, one problem with your test case is that cLength is the wrong type. It should be
SQLLEN, not SQLINTEGER.

Jim, I agree. Thanks for your note.
Unfortunately, SQLLEN has not changed the situation. Results are the same.

fix + test

Attachment: bug31220.diff (application/octet-stream, text), 1.91 KiB.

Committed as rev1054, will be released in 3.51.24.

A note has been added to the 3.51.24 changelog: 

Calling SQLFetch or SQLFetchScroll would return negative data lengths when using
SQL_C_WCHAR.