MySQL Bugs: #15547: Can't initialize character set latin1 (path: /usr/share/mysql/charsets/)

Bug #15547	Can't initialize character set latin1 (path: /usr/share/mysql/charsets/)
Submitted:	7 Dec 2005 9:06	Modified:	29 Jun 2007 16:42
Reporter:	Philippe Jalaber	Email Updates:
Status:	Verified	Impact on me:	None
Category:	Connector / ODBC	Severity:	S1 (Critical)
Version:	3.51	OS:	Linux (Linux)
Assigned to:	Bogdan Degtyariov	CPU Architecture:	Any

Description:
I wrote a simple test program that uses unixODBC 2.2.11 to connect to a MySQL
4.1.15 database with MyODBC 3.51.12 connector.  The program runs on linux
Mandrake 10.1 with 2.4.28 kernel with tls and O(1) patch, glibc 2.3.3-21mdk. unixODBC, MySQL and MyODBC have been compiled from tarballs with gcc 3.4.1.
The program launches 10 threads, each thread creates 6 connections to the
database and close them. If you allocate the SQLHENV in the main program and use
it in each thread in order to allocate each SQLHDBC, everything is OK. Now if
you allocate a new SQLHENV in each thread, ODBC reports the following error when
creating the 5-6 last connections:

Character set 'latin1' is not a compiled character set and is not specified in
the '/usr/share/mysql/charsets/Index.xml' file
(1) HYT00:2019:[unixODBC][MySQL][ODBC 3.51 Driver]Can't initialize character set
latin1 (path: /usr/share/mysql/charsets/)

I also tested the same program with a PostgreSQL database, it works fine. So
unixODBC does not seem to be the problem.

How to repeat:
here is the test program (the program is linked with tls pthread library)
to compile it:  gcc test.c -o test -lodbc -lpthread
In order to easily reproduce the error I launch the program like this: while true; do ./test; done
Hope it helps.

#include <stdlib.h>
#include <stdio.h>
#include <sql.h>
#include <sqlext.h>
#include <pthread.h>

static void odbc_sqlerror(SQLSMALLINT type, SQLHANDLE handle){
        SQLCHAR             state[7];
        SQLCHAR             text[1024];
        SQLSMALLINT         len;
        int                 i=0;
        SQLINTEGER          native;  
        while( SQL_SUCCEEDED(SQLGetDiagRec(type, handle, ++i, state, &native,
text, sizeof(text), &len))) {
                fprintf(stderr, "(%d) %s:%ld:%s\n", i, state, native, text);
        }
}

static SQLHDBC new_hdbc(const char *db, SQLHENV henv){
        SQLRETURN retcode;
        SQLHDBC hdbc;
        retcode = SQLAllocHandle(SQL_HANDLE_DBC, henv, &hdbc);
        if ((retcode != SQL_SUCCESS) && (retcode != SQL_SUCCESS_WITH_INFO)) {
                odbc_sqlerror(SQL_HANDLE_DBC, hdbc);
                return NULL;
        }        
        
        retcode = SQLConnect(hdbc, (char*)db, SQL_NTS, 
                             NULL, 0, NULL, 0);        
        if ((retcode != SQL_SUCCESS) && (retcode != SQL_SUCCESS_WITH_INFO)) {
                odbc_sqlerror(SQL_HANDLE_DBC, hdbc);
                SQLFreeHandle(SQL_HANDLE_DBC, hdbc);
                return NULL;
        }

        return hdbc;
}

static void free_hdbc(SQLHDBC hdbc){
        int retcode;
        retcode = SQLDisconnect(hdbc);
        if ((retcode != SQL_SUCCESS) && (retcode != SQL_SUCCESS_WITH_INFO)) {
                odbc_sqlerror(SQL_HANDLE_DBC, hdbc);
        }
       
        SQLFreeHandle(SQL_HANDLE_DBC, hdbc);
        if ((retcode != SQL_SUCCESS) && (retcode != SQL_SUCCESS_WITH_INFO)) {
                odbc_sqlerror(SQL_HANDLE_DBC, hdbc);
        }
}

static SQLHENV new_henv(void){
       SQLHENV henv;       
       SQLRETURN retcode;

        retcode = SQLAllocHandle(SQL_HANDLE_ENV, SQL_NULL_HANDLE, &henv);
        if ((retcode != SQL_SUCCESS) && (retcode != SQL_SUCCESS_WITH_INFO)) {
                odbc_sqlerror(SQL_HANDLE_ENV, henv);
                return NULL;
        }
  
        SQLSetEnvAttr(henv, SQL_ATTR_ODBC_VERSION, (SQLPOINTER)SQL_OV_ODBC3,
0);
        return henv;
}

static void free_henv(SQLHDBC henv){
        int retcode;  
        retcode = SQLFreeHandle(SQL_HANDLE_ENV, henv);
        if ((retcode != SQL_SUCCESS) && (retcode != SQL_SUCCESS_WITH_INFO)) {
                odbc_sqlerror(SQL_HANDLE_ENV, henv);
        }
}

void *routine(void *arg){
#define NB_CNX 6
        SQLHENV henv;
        SQLHDBC hdbc[NB_CNX];
        int i;
        int j;
        int release_henv = 0;

        henv = (SQLHENV)arg;
        if (henv == NULL) {
                release_henv = 1;
                if ((henv = new_henv()) == NULL) {
                        return NULL;
                }
        }
                
        for (i = 0; i < NB_CNX; i++) {
                printf("Thread %x: opening connection %d\n", pthread_self(),
i);
                if ((hdbc[i] = new_hdbc("DB", henv)) == NULL) {
                        return NULL;
                }
        }
                
        for (i = 0; i < NB_CNX; i++) {
                printf("Thread %x: closing connection %d\n", pthread_self(),
i);
                free_hdbc(hdbc[i]);
        }

        if (release_henv) {
                free_henv(henv);
        }
                
        return NULL;
}

int main(){
#define NB_THREAD 10
        pthread_t thread[NB_THREAD];
        int i;
        struct timeval tv;
        SQLHENV henv;

        /* setting this variable to 1 will force the program to use
           only one SQLHENV allocated in the main program. however if set to 0
           the program will allocate an SQLHENV in each thread
        */
        int one_henv = 0;

        if (one_henv) {
                if ((henv = new_henv()) == NULL) {
                        return 0;
                }
        }

        for (i = 0; i < NB_THREAD; i++) {                
                pthread_create(&thread[i], NULL, routine, (one_henv) ? henv :
NULL);
        }
        for (i = 0; i < NB_THREAD; i++) {
                pthread_join(thread[i], NULL);
        }

        if (one_henv) {
                free_henv(henv);
        }

Philippe, I ran your example and it works fine for me.

The strange thing is - if you build mysql and myodbc from sources, the Index.xml must be in  the /usr/local/mysql/share/mysql/charsets. The fact you program finds it in the /usr/share/mysql/charsets/ makes me believe you have other mysql libraries (than you build from sources). Could you provide here output of

ldd test

I meant of course output of 

ldd /usr/local/lib/libmyodbc3.so

I made a mistake. I did not build from tarballs but I used rpm packages. 

My myODBC driver has mysqlclient linked as static, I 'm sure that I link with right version.
I made the same test with shared library but the result is the same ;(

Please make sure that you run the test program in infinite loop, because it doesn't fail immediately.

Yes, I was able to reproduce it. 

$ while ( true ); do  ./test $i > /dev/null ; done
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql/share/mysql/charsets/Index.xml' file
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql/share/mysql/charsets/Index.xml' file
Segmentation fault
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql/share/mysql/charsets/Index.xml' file
(1) HYT00:2019:[unixODBC][MySQL][ODBC 3.51 Driver]Can't initialize character set latin1 (path: /usr/local/mysql/share/mysql/charsets/)
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql/share/mysql/charsets/Index.xml' file
(1) HYT00:2019:[unixODBC][MySQL][ODBC 3.51 Driver]Can't initialize character set latin1 (path: /usr/local/mysql/share/mysql/charsets/)
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql/share/mysql/charsets/Index.xml' file
(1) HYT00:2019:[unixODBC][MySQL][ODBC 3.51 Driver]Can't initialize character set latin1 (path: /usr/local/mysql/share/mysql/charsets/)
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql/share/mysql/charsets/Index.xml' file
(1) HYT00:2019:[unixODBC][MySQL][ODBC 3.51 Driver]Can't initialize character set latin1 (path: /usr/local/mysql/share/mysql/charsets/)
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql/share/mysql/charsets/Index.xml' file
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql/share/mysql/charsets/Index.xml' file
Segmentation fault
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql/share/mysql/charsets/Index.xml' file
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql/share/mysql/charsets/Index.xml' file
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql/share/mysql/charsets/Index.xml' file
Segmentation fault
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql/share/mysql/charsets/Index.xml' file
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql/share/mysql/charsets/Index.xml' file
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql/share/mysql/charsets/Index.xml' file
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql/share/mysql/charsets/Index.xml' file
(1) HYT00:2019:[unixODBC][MySQL][ODBC 3.51 Driver]Can't initialize character set latin1 (path: /usr/local/mysql/share/mysql/charsets/)
(1) HYT00:2019:[unixODBC][MySQL][ODBC 3.51 Driver]Can't initialize character set latin1 (path: /usr/local/mysql/share/mysql/charsets/)
(1) HYT00:2019:[unixODBC][MySQL][ODBC 3.51 Driver]Can't initialize character set latin1 (path: /usr/local/mysql/share/mysql/charsets/)
(1) HYT00:2019:[unixODBC][MySQL][ODBC 3.51 Driver]Can't initialize character set latin1 (path: /usr/local/mysql/share/mysql/charsets/)

Have the same trouble on linux RedHat entreprise 4. I cannot make the MyOdbc driver run correctly in a multithread environment. I wrote also a little test program. sometimes it runs, sometimes it fails with the "Character set 'latin1' ..." message, sometimes it fails with a segmentation fault.
I used MySql 5.0.19, MyOdbc 3.51.12 and UnixOdbc 2.2.11.
My ODBC DSN uses the /usr/lib/libmyodbc3_r.so.

I also tried recompiling myodbc but the compilation failed with error telling that there are multiple definition of adler32.

Is there any workarround to that blocking situation ?

to reproduce the pb:

testodbc -s sourcename -u user -p password -h numthreads

Here is the source of my testodbc program:

#include <pthread.h>
#include <stdio.h>
#include <sql.h>        // core
#include <sqlext.h>     // extensions
#include <string>

using namespace std;

// Thread mutex
	pthread_mutex_t m_mMutex;

// End Thread flag
int EndThread = 0;
int NumErrors = 0;

// Num threads
int NumThreads = 1;

SQLCHAR	Source[255];	// Data source name
SQLCHAR	User[255];		// Data source user
SQLCHAR	Password[255];	// Data source password
SQLCHAR	Table[255];		// Data source table

void DisplaySqlError(SQLHENV henv, SQLHDBC hdbc, SQLHSTMT hstmt);
int DisplaySqlStringInfo(SQLHENV henv, SQLHDBC hdbc, SQLUSMALLINT InfoType);
int DoConnect(bool WithInfo, bool WithSearch);
void* ThreadMain(void *Value);

int main(int argc, char* argv[])
{
	if(argc <=1)
	{
		printf("usage: testodbc -s DataSource -u User [-p Password] [-t Table] [-h NumThreads]\n");
		return 0;
	}
	Source[0] = '\0';
	User[0] = '\0';
	Password[0] = '\0';
	int i;
	for(i=1;i<argc;i++)
	{
		if(strcmp(argv[i], "-s") == 0)
		{
			if(i==(argc-1))
			{
				printf("usage: testodbc -s DataSource -u User [-p Password] [-t Table] [-h NumThreads]\n");
				return 0;
			}
			i++;
			strcpy((char *) Source, argv[i]);
		}
		else if(strcmp(argv[i], "-u") == 0)
		{
			if(i==(argc-1))
			{
				printf("usage: testodbc -s DataSource -u User [-p Password] [-t Table] [-h NumThreads]\n");
				return 0;
			}
			i++;
			strcpy((char *) User, argv[i]);
		}
		else if(strcmp(argv[i], "-p") == 0)
		{
			if(i==(argc-1))
			{
				printf("usage: testodbc -s DataSource -u User [-p Password] [-t Table] [-h NumThreads]\n");
				return 0;
			}
			i++;
			strcpy((char *) Password, argv[i]);
		}
		else if(strcmp(argv[i], "-t") == 0)
		{
			if(i==(argc-1))
			{
				printf("usage: testodbc -s DataSource -u User [-p Password] [-t Table] [-h NumThreads]\n");
				return 0;
			}
			i++;
			strcpy((char *) Table, argv[i]);
		}
		else if(strcmp(argv[i], "-h") == 0)
		{
			if(i==(argc-1))
			{
				printf("usage: testodbc -s DataSource -u User [-p Password] [-t Table] [-h NumThreads]\n");
				return 0;
			}
			i++;
			NumThreads = atoi(argv[i]);
			if(NumThreads <= 0)
				NumThreads = 1;
			if(NumThreads > 500)
				NumThreads = 500;
		}
	}

	if(!DoConnect(true, false))
		return 0;

	pthread_mutex_init(&m_mMutex, NULL);

	EndThread = NumThreads;
	int EndThreadHere = EndThread;
	for(i=0;i<NumThreads;i++)
	{
		pthread_attr_t	attr;
		pthread_attr_init(&attr);
		pthread_attr_setscope(&attr, PTHREAD_SCOPE_SYSTEM);		// the unix thread has a system scope for best performances
		pthread_t thid;
		pthread_create(&thid, &attr, ThreadMain, NULL);
		pthread_attr_destroy(&attr);
	}
	while(EndThreadHere)
	{
		pthread_mutex_lock(&m_mMutex);
		EndThreadHere = EndThread;
		pthread_mutex_unlock(&m_mMutex);
	}

	if(NumErrors)
		printf("\nProgram terminated with %d errors\n", NumErrors);
	else
		printf("\nProgram terminated successfully\n");

	pthread_mutex_unlock(&m_mMutex);
	pthread_mutex_destroy(&m_mMutex);

	return 0;
}

void* ThreadMain(void *Value)
{
	int RetVal = DoConnect(false, true);

	pthread_mutex_lock(&m_mMutex);
	if(!RetVal)
		NumErrors++;
	EndThread--;
	pthread_mutex_unlock(&m_mMutex);

	pthread_detach(pthread_self());
	pthread_exit(NULL);

	return (void *)(0);			// end of thread (unix)
}

int DoConnect(bool WithInfo, bool WithSearch)
{
	SQLHENV     henv;	/* environment handle */
	SQLHDBC     hdbc;	/* connection handle */
	SQLHSTMT    hstmt;	/* statement handle */
	SQLRETURN   retcode;	/* return code */
	SQLCHAR SQLStmt[255];

	henv = NULL;
	hdbc = NULL;
	hstmt = NULL;

	retcode = SQLAllocEnv(&henv);
	retcode = SQLAllocConnect(henv, &hdbc);
	retcode = SQLConnect(hdbc, Source, SQL_NTS, User, SQL_NTS, Password, SQL_NTS);

	if (retcode != SQL_SUCCESS)
	{
		printf("*** Cannot connect data source\n");
		DisplaySqlError(henv, hdbc, hstmt);
		SQLFreeConnect(hdbc);
		SQLFreeEnv(henv);
		return 0;
	}
	if(WithInfo)
	{
		printf("Database name: ");
		DisplaySqlStringInfo(henv, hdbc,SQL_DBMS_NAME);
		printf("\nDatabase version: ");
		DisplaySqlStringInfo(henv, hdbc,SQL_DBMS_VER);
		printf("\nDriver name: ");
		DisplaySqlStringInfo(henv, hdbc,SQL_DRIVER_NAME);
		printf("\nDriver version: ");
		DisplaySqlStringInfo(henv, hdbc,SQL_DRIVER_VER);
		printf("\nDriver ODBC version: ");
		DisplaySqlStringInfo(henv, hdbc,SQL_ODBC_VER);
		printf("\n");
	}
	if(WithSearch)
	{
		if(strlen((char *)Table))
		{
			retcode = SQLAllocHandle(SQL_HANDLE_STMT, hdbc,&hstmt);
			if (retcode == SQL_SUCCESS)
			{
				strcpy((char *)SQLStmt, "SELECT * FROM ");
				strcat((char *)SQLStmt, (char *)Table);
				retcode = SQLExecDirect(hstmt, SQLStmt, SQL_NTS);
				// Display The Results Of The SQL Query
				if (retcode == SQL_SUCCESS)
					printf("Database search successful\n");
				else
				{
					printf("*** Error searching database\n");
					DisplaySqlError(henv, hdbc, hstmt);
					SQLDisconnect(hdbc);
					SQLFreeConnect(hdbc);
					SQLFreeEnv(henv);
					return 0;
				}
			}
			// Free The SQL Statement Handle
			if(hstmt != NULL)
				SQLFreeHandle(SQL_HANDLE_STMT, hstmt);
		}
	}

	SQLDisconnect(hdbc);
	SQLFreeConnect(hdbc);
	SQLFreeEnv(henv);

	return 1;
}

void DisplaySqlError(SQLHENV henv, SQLHDBC hdbc, SQLHSTMT hstmt)
{
	SQLCHAR	Message[SQL_MAX_MESSAGE_LENGTH + 1];
	SQLCHAR	State[10];
	SQLINTEGER NativeError;
	SQLSMALLINT TextLength;

	Message[0] = '\0';
	SQLError(henv, hdbc, hstmt, State, &NativeError, Message, SQL_MAX_MESSAGE_LENGTH+1, &TextLength);

	printf("SQL error: %s\n", (const char *)Message);
}

int DisplaySqlStringInfo(SQLHENV henv, SQLHDBC hdbc, SQLUSMALLINT InfoType)
{
	SQLRETURN   retcode;	/* return code */
	SQLCHAR    info[255];	/* info string for SQLGetInfo */
	SQLSMALLINT cbInfoValue;

	info[0] = '\0';
	retcode = SQLGetInfo(hdbc,InfoType, &info, sizeof(info),&cbInfoValue);
	if (retcode == SQL_SUCCESS)
		printf((const char *)info);
	else
	{
		printf("*** Cannot get info from data source\n");
		DisplaySqlError(henv, hdbc, NULL);
		return 0;
	}
	return 1;
}

makefile:

CXX = g++
CXXFLAGS = -I- -I./ -D_REENTRANT -D_GNU_SOURCE -DLINUX
LDFLAGS = -L./ -lpthread -ldl -lodbc

testodbc :: testodbc.o
	$(CXX) -o $@ $^ $(LDFLAGS)
	strip $@

testodbc.o : testodbc.cpp
	$(CXX) -c -o $@ $< $(CXXFLAGS)

Philippe, I tested your code, the result is:

On my redhat9 system (libc-2.3.2, libpthread-0.29), it works fine. 

But on the other system (libc-2.3.4, libpthread-2.3.4), the same situation arise s just as yours. I am not very sure if it is related to the libc version ?

I tried reproduce the initial problem using both test snippets in the bug
report (with minor modifications), and tried it on SLES9 x86, RHEL4 x86
and x86_64, and could not reproduce the problem.

I did hit another bug#25621, but it is unrelated.

But I could repeat the craches with segmentation faults reported,
especially on RHEL4 x86, so this bug report is left open.

The library initialization in Connector/ODBC 3.51 is not very thread-safe on non-Windows platforms. A work-around is to make sure you start up one thread before the others, to avoid the race conditions.

To fix this, we'll need to use pthread_once() on non-Windows platforms to make sure we only do the library initialization once, use a mutex-protected count of threads so we can shut the driver down at the right time, and make sure to call mysql_thread_init() in each new thread.

Bug#6536 was marked as duplicate of this one.

Bug #45058 is a related bug in libmysql -- once that fix propagates, we can use the same new pthread_once() functionality to protect C/ODBC's init.

Still doesn't seem to be fixed.

Any update on this one? 
One of our perl scripts using the following Perl libraries started seg-faulting and dumping core due to this error:

DBI: 1.623
DBD::mysql: 4.022
OS: Cent OS 6.4

I know these are slightly older versions, but we are waiting for this bug to be fixed so we can upgrade.