Bug #45058 init_available_charsets uses double checked locking
Submitted: 25 May 2009 3:52 Modified: 19 Jul 2010 20:14
Reporter: Mark Callaghan Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server Severity:S3 (Non-critical)
Version:5.0, 5.1 OS:Any
Assigned to: Staale Smedseng
Tags: CHECKED, double, init_available_charsets, locking
Triage: Triaged: D2 (Serious)

[25 May 2009 3:52] Mark Callaghan
Description:
init_available_charsets uses double checked locking. I don't have a test case to show that this is a problem, but for the reasons described in this paper -- 
 http://www.aristeia.com/Papers/DDJ_Jul_Aug_2004_revised.pdf -- when one thread has the mutex and is initializing structure, the assignment to charset_initialized that it does may be visible to other threads before all of the other writes that it does. This will allow other threads to read invalid data.

static my_bool init_available_charsets(myf myflags)
#endif
{
  char fname[FN_REFLEN + sizeof(MY_CHARSET_INDEX)];
  my_bool error=FALSE;
  /*
    We have to use charset_initialized to not lock on THR_LOCK_charset
    inside get_internal_charset...
  */
  if (!charset_initialized)
  {
    CHARSET_INFO **cs;
    /*
      To make things thread safe we are not allowing other threads to interfere
      while we may changing the cs_info_table
    */
    pthread_mutex_lock(&THR_LOCK_charset);
    if (!charset_initialized)
    {
<snip>
      strmov(get_charsets_dir(fname), MY_CHARSET_INDEX);
      error= my_read_charset_file(fname,myflags);
      charset_initialized=1;
    }
    pthread_mutex_unlock(&THR_LOCK_charset);
  }
  return error;
}

How to repeat:
Review the code and the referenced paper
[25 May 2009 6:05] Sveta Smirnova
Thank you for the report.

The paper shows the problem if one uses object which method going to return as flag:

Singleton* Singleton::instance() { 
if (pInstance == 0) { // 1st test 
Lock lock; 
if (pInstance == 0) { // 2nd test 
pInstance = new Singleton; 
} 
} 
return pInstance; 
} 

while MySQL uses separate flag charset_initialized which sets to 1 after all work is done. Why do you think MySQL code affected by the same problem?
[25 May 2009 6:33] Jonas Oreland
the code needs a memory barrier before the flag is set,

      error= my_read_charset_file(fname,myflags);
>>    mb();
      charset_initialized=1;

this is a problem, as a CPU might(will) reorder writes to memory,
which can be prevented by a memory barrier
(and since we probably can't enforce a read-protocol on the reader,
 it likely will have to be a full barrier)

an portable(but expensive) way of making a memory barrier to to lock/unlock
a mutex
[25 Jun 2009 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[25 Jun 2009 23:11] Mark Callaghan
Move the charset_initialized assignment after pthread_mutex_unlock
[29 Jun 2009 10:11] Sveta Smirnova
Mark, Jonas,

thank you for the feedback. Verified as described.
[27 Jul 2009 21:12] Davi Arnaut
The proposed solution just goes from one gray scenario to another. AFAIR, pthread makes no guarantees with respect to variables modified outside of the of the lock. Also, whether a memory barrier is issued, it is implementation specific.

As others have mentioned, this whole optimization seems unwarranted. It would perhaps be better to just initialize it once at server start.
[11 Sep 2009 17:24] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/83079

3120 Staale Smedseng	2009-09-11
      Bug #45058 init_available_charsets uses double checked locking
      
      As documented in the bug report, the double checked locking
      pattern has inherent issues, and cannot guarantee correct
      initialization.
      
      This patch modifies the charset initialization code to do its
      work during server/client initialization, and not through
      calls to get_charset_*(). This is achieved by splitting
      init_available_charsets() into two functions:
      
       * load_compiled_charsets(): loads the compiled character
         sets, called from my_init(),
       * load_available_charsets(): loads charset index file after
         any options regarding charset directory have been
         processed; called explicitly in mysqld.cc, mysqldump.cc
         a.o., and implicitly in clients through
         mysql_library_init().
      
      The split is necessary to support functions relying on
      charsets that are used early on, while giving the possibility
      of changing the source charsets_dir (e.g., during options
      parsing) before loading charset definitions.
     @ client/mysqldump.c
        Adding explicit loading of charset info from file.
     @ extra/charset2html.c
        Adding explicit loading of charset info from file.
     @ extra/comp_err.c
        Adding explicit loading of charset info from file.
     @ include/my_sys.h
        init_available_charsets() is split into two functions.
     @ libmysql/libmysql.c
        Adding explicit loading of charset info from file.
     @ mysys/charset.c
        init_available_charsets() is split into two functions. Calls
        to init_available_charsets() are dropped from the get_charset*() functions.
     @ mysys/my_init.c
        Adding explicit loading of compiled charsets in my_init().
     @ netware/libmysqlmain.c
        Charset loading now happens in my_init() and mysql_server_init().
     @ sql/mysqld.cc
        Adding explicit loading of charset info from file.
[16 Oct 2009 14:54] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/87152

3153 Staale Smedseng	2009-10-16
      Bug #45058 init_available_charsets uses double checked locking
            
      As documented in the bug report, the double checked locking
      pattern has inherent issues, and cannot guarantee correct
      initialization.
      
      This is a new patch with an alternative solution, intended to
      avoid any possible API incompatibilities introduced in the
      initial patch.
      
      This patch replaces the logic in init_available_charsets()
      with the use of pthread_once(3). A wrapper function,
      my_pthread_once(), is introduced and is used in lieu of direct
      calls to init_available_charsets().
      
      For the Windows platform, the implementation in lp:sysbench is
      ported. For single-thread use, a simple define calls the
      function and sets the pthread_once control variable.
     @ include/my_no_pthread.h
        Dummy my_pthread_once() for single thread use.
     @ include/my_pthread.h
        Declaration for new function my_pthread_once().
     @ mysys/charset.c
        Logic in init_available_charsets() is simplified. 
        Using my_pthread_once() for all calls to this func.
[19 Dec 2009 8:29] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20091219082307-f3i4fn0tm8trb3c0) (version source revid:alik@sun.com-20091216180721-eoa754i79j4ssd3m) (merge vers: 6.0.14-alpha) (pib:15)
[19 Dec 2009 8:33] Bugs System
Pushed into 5.5.1-m2 (revid:alik@sun.com-20091219082021-f34nq4jytwamozz0) (version source revid:alexey.kopytov@sun.com-20091216134707-o96eqw0u2ynvo9gm) (merge vers: 5.5.0-beta) (pib:15)
[19 Dec 2009 8:37] Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20091219082213-nhjjgmphote4ntxj) (version source revid:alik@sun.com-20091216180221-a5ps59gajad3pip9) (pib:15)
[8 Jan 2010 21:24] Paul Dubois
Noted in 5.5.1, 6.0.14 changelogs.

It was possible for init_available_charsets() not to initialize
correctly.  

Setting report to NDI pending push to Celosia.
[15 Jan 2010 8:58] Bugs System
Pushed into 5.1.43 (revid:joro@sun.com-20100115085139-qkh0i0fpohd9u9p5) (version source revid:staale.smedseng@sun.com-20091212181125-w4deewgjviz548i0) (merge vers: 5.1.42) (pib:16)
[15 Jan 2010 18:21] Paul Dubois
Noted in 5.1.43 changelog.

Setting report to NDI pending push to Celosia.
[20 Feb 2010 5:15] [ name withheld ]
According to the discussion at
https://bugzilla.redhat.com/show_bug.cgi?id=566547
as well as several reports linked from there, the version of this patch pushed into 5.1.43 does not work.
It breaks akonadi and probably other applications.  I have not tried to trace the logic changes in detail but it looks to me like there is a code path wherein the charset info doesn't get initialized.
[20 Feb 2010 15:52] Bartosz Fabianowski
Using MySQL 5.5.1, KDE's akonadi fails in the same way as with MySQL 5.1.43. So this change must have introduced problems in the 5.5 branch as well. Here is a backtrace:

#0  0x0000000802f77344 in my_strcasecmp_8bit () from /usr/local/lib/mysql/libmysqlclient_r.so.16
#1  0x0000000802f6bd8b in get_collation_number () from /usr/local/lib/mysql/libmysqlclient_r.so.16
#2  0x0000000802f6bdd0 in get_charset_by_name () from /usr/local/lib/mysql/libmysqlclient_r.so.16
#3  0x0000000802f7f4f2 in mysql_init_character_set () from /usr/local/lib/mysql/libmysqlclient_r.so.16
#4  0x0000000802f81650 in mysql_real_connect () from /usr/local/lib/mysql/libmysqlclient_r.so.16
#5  0x0000000802e0ef66 in QMYSQLDriver::open () from /usr/local/lib/qt4/plugins/sqldrivers/libqsqlmysql.so
#6  0x0000000800f0ba61 in QSqlDatabase::open () from /usr/local/lib/qt4/libQtSql.so.4
#7  0x000000080076d55c in Akonadi::DataStore::open (this=0x802c7df80) at /usr/ports/databases/akonadi/work/akonadi-1.3.1/server/src/storage/datastore.cpp:87
#8  0x000000080076da9d in DataStore (this=0x802c7df80) at /usr/ports/databases/akonadi/work/akonadi-1.3.1/server/src/storage/datastore.cpp:64
#9  0x000000080076db7c in Akonadi::DataStore::self () at /usr/ports/databases/akonadi/work/akonadi-1.3.1/server/src/storage/datastore.cpp:140
#10 0x0000000800790cbb in Akonadi::CacheCleaner::run (this=0x8028aa340) at /usr/ports/databases/akonadi/work/akonadi-1.3.1/server/src/cachecleaner.cpp:43
#11 0x00000008009ad793 in QThreadPrivate::start () from /usr/local/lib/qt4/libQtCore.so.4
#12 0x0000000801cc32e1 in pthread_getprio () from /lib/libthr.so.3
#13 0x0000000000000000 in ?? ()
Cannot access memory at address 0x7fffff9fe000
[20 Feb 2010 19:18] Rex Dieter
This fix seems to have caused problems initializing akonadi(mysql) databases used in kde.  Errors reported include:
[akonadiserver] Character set 'latin1' is not a compiled character set and is
not specified in the '/usr/share/mysql/charsets/Index.xml' file
[akonadiserver] Database error: Cannot open database.

See also downstream reports,
http://bugs.kde.org/show_bug.cgi?id=226960
https://bugzilla.redhat.com/show_bug.cgi?id=566547

In particular, Tom Lane has come up with the minimal set of changes, that if reverted, makes the problem go away wrt akonadi initialization,
https://bugzilla.redhat.com/show_bug.cgi?id=566547#c11

Please advise.
[22 Feb 2010 23:16] Robin Johnson
Another ping for reopening this please. The patch on the RedHat bug is a working fix, verified by others on FreeBSD. The patch is now applied/available to users in Fedora (FC12/13) and Gentoo.
[22 Feb 2010 23:37] Davi Arnaut
The patch is a reversal, it does not explain the problem. Anyone knows what the problem is?
[22 Feb 2010 23:55] Robin Johnson
@Davi:
See the RedHat and KDE reports, that note when akonadi tries to start mysqld, mysqld crashes with:
Character set 'latin1' is not a compiled character set and is not specified in
the '/usr/share/mysql/charsets/Index.xml' file

Despite the latin1 being defined in there, and the files for it existing.
[23 Feb 2010 0:13] Davi Arnaut
I saw that. I was hoping for a explanation on why it breaks.
[23 Feb 2010 0:24] Robin Johnson
As to what akonadi is doing to cause a break, I wish I knew. Not a KDE user here, just the MySQL maintainer @ Gentoo.

@Rex Dieter:
Do you have the parameters that akonadi has to start the mysqld instance?
[26 Feb 2010 13:37] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/101654

3358 Staale Smedseng	2010-02-26
      Bug #45058 init_available_charsets uses double checked locking
      
      A client doing multiple mysql_library_init() and
      mysql_library_end() calls over the lifetime of the process may
      experience lost character set data, potentially even a
      SIGSEGV.
      
      This patch reinstates the reloading of character set data when
      a mysql_library_init() is done after a mysql_library_end().
[26 Feb 2010 14:30] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/101668

3362 Staale Smedseng	2010-02-26
      Bug #45058 init_available_charsets uses double checked locking
      
      A client doing multiple mysql_library_init() and
      mysql_library_end() calls over the lifetime of the process may
      experience lost character set data, potentially even a
      SIGSEGV.
      
      This patch reinstates the reloading of character set data when
      a mysql_library_init() is done after a mysql_library_end().
[1 Mar 2010 20:25] Rex Dieter
Based on some preliminary testing, I can confirm that
http://lists.mysql.com/commits/101654
appears to fix the akonadi/charset issue reported here.  Thank you.
[2 Mar 2010 14:34] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100302142746-u1gxdf5yk2bjrq3e) (version source revid:alik@sun.com-20100301095421-4cz64ibem1h2quve) (merge vers: 6.0.14-alpha) (pib:16)
[2 Mar 2010 20:55] Paul Dubois
Setting report to Need Merge pending push to Celosia.
[5 Mar 2010 10:28] Alex Dupre
I think the committed patch is wrong. You are trying to fake the pthread_once behavior, but actually you cannot reset the charsets_initialized variable. It probably works on Linux, because PTHREAD_ONCE_INIT is defined as '0', but on other systems it could be different (for example on FreeBSD) and fails miserably.
[5 Mar 2010 10:38] Davi Arnaut
We know and tried to workaround it: http://lists.mysql.com/commits/101737
[12 Mar 2010 14:10] Bugs System
Pushed into 5.1.44-ndb-7.0.14 (revid:jonas@mysql.com-20100312135944-t0z8s1da2orvl66x) (version source revid:jonas@mysql.com-20100312115609-woou0te4a6s4ae9y) (merge vers: 5.1.44-ndb-7.0.14) (pib:16)
[12 Mar 2010 14:25] Bugs System
Pushed into 5.1.44-ndb-6.2.19 (revid:jonas@mysql.com-20100312134846-tuqhd9w3tv4xgl3d) (version source revid:jonas@mysql.com-20100312060623-mx6407w2vx76h3by) (merge vers: 5.1.44-ndb-6.2.19) (pib:16)
[12 Mar 2010 14:39] Bugs System
Pushed into 5.1.44-ndb-6.3.33 (revid:jonas@mysql.com-20100312135724-xcw8vw2lu3mijrhn) (version source revid:jonas@mysql.com-20100312103652-snkltsd197l7q2yg) (merge vers: 5.1.44-ndb-6.3.33) (pib:16)
[12 Mar 2010 17:42] Paul Dubois
Fixed in earlier 5.1.x, 5.5.x.
[26 Mar 2010 8:21] Bugs System
Pushed into 5.5.4-m3 (revid:alik@sun.com-20100326080914-2pz8ns984e0spu03) (version source revid:alexey.kopytov@sun.com-20100307164059-cri8typa32cypq0l) (merge vers: 5.5.3-m2) (pib:16)
[26 Mar 2010 8:26] Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100326081116-m3v4l34yhr43mtsv) (version source revid:alik@sun.com-20100325072612-4sds00ix8ajo1e84) (pib:16)
[6 Apr 2010 7:57] Bugs System
Pushed into 5.1.46 (revid:sergey.glukhov@sun.com-20100405111026-7kz1p8qlzglqgfmu) (version source revid:joro@sun.com-20100301084434-ytctk3ceebjvqo7a) (merge vers: 5.1.45) (pib:16)
[17 Jun 2010 11:54] Bugs System
Pushed into 5.1.47-ndb-7.0.16 (revid:martin.skold@mysql.com-20100617114014-bva0dy24yyd67697) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)
[17 Jun 2010 12:32] Bugs System
Pushed into 5.1.47-ndb-6.2.19 (revid:martin.skold@mysql.com-20100617115448-idrbic6gbki37h1c) (version source revid:martin.skold@mysql.com-20100609211156-tsac5qhw951miwtt) (merge vers: 5.1.46-ndb-6.2.19) (pib:16)
[17 Jun 2010 13:19] Bugs System
Pushed into 5.1.47-ndb-6.3.35 (revid:martin.skold@mysql.com-20100617114611-61aqbb52j752y116) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)
[20 Jul 2010 15:34] Domas Mituzas
This is still broken - a multithreaded mysql app breaks just fine with latest 5.1.49-bzr:

$ while true; do env LD_LIBRARY_PATH=/usr/local/mysql-5.1/lib/mysql/ ./pmysql "SET @a=1" < servers; done
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql-5.1/share/mysql/charsets/Index.xml' file
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql-5.1/share/mysql/charsets/Index.xml' file

** (process:6969): WARNING **: Could not connect to 10..........: Can't initialize character set latin1 (path: /usr/local/mysql-5.1/share/mysql/charsets/)

** (process:6969): WARNING **: Could not connect to 10..........: : Can't initialize character set latin1 (path: /usr/local/mysql-5.1/share/mysql/charsets/)
Segmentation fault
Segmentation fault
*** glibc detected *** ./pmysql: munmap_chunk(): invalid pointer: 0x00002aaaac024720 ***
Segmentation fault
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql-5.1/share/mysql/charsets/Index.xml' file
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql-5.1/share/mysql/charsets/Index.xml' file
Character set 'latin1' is not a compiled character set and is not specified in the '/usr/local/mysql-5.1/share/mysql/charsets/Index.xml' file

** (process:8630): WARNING **: Could not connect to 10..........: : Can't initialize character set latin1 (path: /usr/local/mysql-5.1/share/mysql/charsets/)

** (process:8630): WARNING **: Could not connect to 10..........: : Can't initialize character set latin1 (path: /usr/local/mysql-5.1/share/mysql/charsets/)

** (process:8630): WARNING **: Could not connect to 10...........: : Can't initialize character set latin1 (path: /usr/local/mysql-5.1/share/mysql/charsets/)
[20 Jul 2010 16:01] Domas Mituzas
Disregard my previous comment.

libmysqlclient was used instead of libmysqlclient_r...

As I was calling mysql_thread_init(), assertion/warning/etc when using not-thread-safe client would have been especially helpful! I'll open a feature-request for API about that.
[29 Aug 2016 8:56] Chandrakanth Nayak N
hi,
In my code I am using mysql_library_init() function before calling any mysql library functions.
And similarly closing by using mysql_library_end().  But some time we see below error

Character set 'latin1' is not a compiled character set and is not specified in the '/usr/share/mysql/charsets/Index.xml' file
Can't initialize character set latin1 (path: /usr/share/mysql/charsets/)

I am not sure what causing this problem. Is this fix merged in below mysql version. This one causing lot of problem in our environment. 

OS : centos 6.2
MySQL version:
mysql> SHOW VARIABLES LIKE "%version%";
+-------------------------+---------------------+
| Variable_name                   | Value                         |
+-------------------------+---------------------+
| protocol_version                | 10                               |
| version                               | 5.1.52-log                  |
| version_comment              | Source distribution   |
| version_compile_machine | x86_64                       |
| version_compile_os           | unknown-linux-gnu   |
+-------------------------+---------------------+

Even after upgrading to centos 6.8 we are seeing charset latin1 error.

OS : centos 6.8
MySQL version : 5.1.73-log.