MySQL Bugs: #50221: Server refuses to start from non-default installdir on Solaris 32-bit

Bug #50221	Server refuses to start from non-default installdir on Solaris 32-bit
Submitted:	11 Jan 2010 8:01	Modified:	7 Mar 2010 1:17
Reporter:	John Embretsen	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Options	Severity:	S1 (Critical)
Version:	5.5.99-m3 (Celosia)	OS:	Solaris (32-bit)
Assigned to:	Olav Sandstå	CPU Architecture:	Any
Tags:	pb2, test failure

Description:
The MySQL server is refusing to start from non-default install directory. The cause seems to be some erroneous treatment of mysqld variables:

------------------------------------------

./libexec/mysqld --verbose --help

100111  8:51:43 [ERROR] Error message file '/usr/local/share/mysql/english/errmsg.sys' had only 618 error messages,
but it should contain at least 663 error messages.
Check that the above file is the right version for this program!
100111  8:51:43 [Warning] Can't create test file /usr/local/var/techra22.lower-test
100111  8:51:43 [Warning] Can't create test file /usr/local/var/techra22.lower-test
./libexec/mysqld: Can't change dir to '/usr/local/var/' (Errcode: 2)
100111  8:51:43 [Note] Plugin 'FEDERATED' is disabled.
100111  8:51:43 [Note] Plugin 'ndbcluster' is disabled.
./libexec/mysqld: Unknown error 1146
100111  8:51:43 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
100111  8:51:43 [ERROR] ./libexec/mysqld: unknown option '--verbose'
./libexec/mysqld  Ver 5.5.99-debug for sun-solaris2.10 on sparc (Source distribution)
Copyright (C) 2000-2008 MySQL AB, by Monty and others
Copyright (C) 2008,2009 Sun Microsystems, Inc.
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL license

Starts the MySQL database server

Usage: ./libexec/mysqld [OPTIONS]

For more help options (several pages), use mysqld --verbose --help
100111  8:51:43 [ERROR] Aborting

------------------------------------------

Note that in the above example, we are trying to display server help text from inside a directory where a binary built by PB2 is unpacked. 

Same symptoms are observed when trying to run mysql-test-run.pl or mysql_install_db scripts, or other wrappers around mysqld.

This issue has been observed on Solaris 10 32-bit platforms only so far. 

The behavior change occurred upon the push/merge of WL#4738 ("streamline/simplify @@variable creation process"), and was first observed in Pushbuild for Random Query Generator tests on platform "sol10 sparc32 debug_max".

In mysql-next-mr branch, WL#4738 push details were as follows:

Push Revision:	serg@mysql.com-20091226205532-73tbr9vog3o58nx4
Push Date:      2009-12-28 09:45:15

How to repeat:
On a Solaris 32-bit platform, obtain a copy of MySQL server based on mysql-next-mr or mysql-6.0-codebase branches of the above mentioned revision or later (until this bug is fixed).

Try to start the server from a non-default install directory, e.g. by doing:

cd mysql-test
perl ./mysql-test-run.pl 1st

or: 

./libexec/mysqld --verbose --help

or: 

./scripts/mysql_install_db [...]

etc.

Suggested fix:
Do not try to use files from default installations when not necessary or not instructed to do so. Honor settings given on command-line.

Olav has started to do some initial analysis of this issue, so I am assigning to him for now...

Bug can be considered "Verified": I was able to repeat it.

The reason for the mysqld process fails to use the argument values specified on the command line seems to be that only the first 20 of the 92 entries in
the static allocated my_long_options array is initialized correctly. The last 72 entries does not contain the correct values when compiled with Sun Studio compiler for 32 bit Solaris. So when looking up an argument given on the command line mysqld does not find it in its default list of valid arguments and thus ignore them and reverts to default values.

The following code change to mysqld.cc (approximately line 6245) seems to have triggered this bug:

-  {"default-storage-engine", OPT_STORAGE_ENGINE,
-   "Set the default storage engine (table type) for tables.",
-   (uchar**)&default_storage_engine_str, (uchar**)&default_storage_engine_str,
-   0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0},
+  {"default-storage-engine", 0, "The default storage engine for new tables",
+   (uchar**) &default_storage_engine, 0, 0, GET_STR, REQUIRED_ARG,
+   (longlong)"MyISAM", 0, 0, 0, 0, 0 },

It seems like it is the (longlong)"MyISAM" value initialization that is causing the following entries in the my_long_options to contain "garbage". Replacing this with a "0" value "fixes" the problem with the my_long_options array (but also changes the default value of the default-storage-engine).

This might look like a compiler issue with the Sun Studio compiler.

C++ program to show Sun Studio compiler bug

Attachment: olav.cc (text/x-c++src), 507 bytes.

This problem seems to be caused by a bug in the Sun Studio compiler. I have attached a small C++ program based on a simplified version of the my_long_options array in mysqld.cc that illustrates this bug.

To reproduce the bug using the C++ program do:

1. Compile the program using Sun Studio 12 (or Sun Studio 12 update 1) C++ compiler for 32 bit memory model:
     
    CC -I/export/home/tmp/olav/solaris32/include -m32 -mt -g0 olav.cc

2. Run the program:

    a.out

This should crash with a segmentation fault and produce a core file.

Inspecting the core file shows that the my_long_options array contains garbage after the 3rd entry:

(dbx) print my_long_options
my_long_options = (
{
    name      = 0x8050afe "help1"
    id        = 1
    def_value = 0
    min_value = 0
    max_value = 0
}
{
    name      = 0x8050b04 "help2"
    id        = 2
    def_value = 0
    min_value = 0
    max_value = 0
}
{
    name      = 0x8050b0a "help3"
    id        = 3
    def_value = 134548240LL
    min_value = 0
    max_value = 577880320599130112LL
}
{
    name      = 0x4 "<bad address 0x4>"
    id        = 0
    def_value = 0
    min_value = 0
    max_value = 577880346368933888LL
}
{
    name      = 0x5 "<bad address 0x5>"
    id        = 0
    def_value = 0
    min_value = 0
    max_value = 577880372138737664LL
}
{
    name      = 0x6 "<bad address 0x6>"
    id        = 0
    def_value = 0
    min_value = 0
    max_value = 0
}

It seems to me that everything after some place in the middle of the 3rd entry is shifted down by 4 bytes. My guess is that the compiler when executing the following initialization of the 3rd entry:

    {"help3", 3, (long long int)"MyISAM", 0, 0},

is only allocating 4 bytes to the "def_value" member instead of 8 bytes. If I replace (long long int)"MyISAM" with a integer value the program behaves correct.

This program also runs correctly if compiled using Sun Studio C++ for 64 bit memory model as well as when using gcc to compile for 32 bit memory model.

Gcc is not too happy with this code for initialization of the def_value member. When using gcc 3.4.3 to compile 32-bit binaries for Solaris it prints the following warning:

  mysqld.cc:5785: warning: cast from pointer to integer of different size

The produced code works as expected.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/96750

2965 Olav Sandstaa	2010-01-13
      Fix for Bug#50221 Server refuses to start from non-default installdir 
      on Solaris 32-bit
      
      When starting mysqld it did not recognize most of the options given on
      the command line when it was compiled for 32-bit Solaris using Sun
      Studio compiler. The cause for this was that most of the entries in
      the my_long_options array contained "garbage" data. The garbage data
      was caused by a compiler bug. When initilizing the def_value member
      for the "default-storage-engine" entry it was initialized like this:
      
        (longlong)"MyISAM"
      
      i.e. casting a 32 bit pointer to a 64 bit integer value. Due to the
      compiler bug only 4 bytes was allocated (instead of 8 bytes). This
      caused everything following this entry to be stored at a location that
      was 4 byte wrong.
      
      The fix/work-around for this problem is to store the pointer to the
      "MyISAM" string in a local integer variable and then use this integer
      variable to intialize the def_value member in the my_long_options
      array.
     @ sql/mysqld.cc
        Due to a bug in Sun Studio compiler when generating 32 bit code the 
        initialization of the def_value member of the default-storage-engine entry 
        in my_long_options only got 4 bytes allocated instead of 8 bytes. 
        The compiler bug was triggered by casting a 32 bit pointer to a 64 bit 
        integer value in the initialization code for my_long_options. To avoid 
        triggering the compiler bug this casting from 32 bit 
        pointer to 64 bit integer is moved to a local variable.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

http://lists.mysql.com/commits/96882

2966 Olav Sandstaa 2010-01-14
Fix for Bug#50221 Server refuses to start from non-default installdir on Solaris 32-bit

When starting mysqld it did not recognize most of the options given on
the command line when it was compiled for 32-bit Solaris using Sun
Studio compiler. The cause for this was that most of the entries in
the my_long_options array contained "garbage" data. The garbage data
was caused by a compiler bug. When initilizing the def_value member
for the "default-storage-engine" entry it was initialized like this:

(longlong)"MyISAM"

i.e. casting a 32 bit pointer to a 64 bit integer value. Due to the
compiler bug only 4 bytes was allocated (instead of 8 bytes). This
caused everything following this entry to be stored at a location that
was 4 byte wrong.

The fix/work-around for this problem is initialize the def_value
for default-storage-engine in my_long_options to 0 and instead
initialize the default_storage_engine variable to "MyISAM" in
init_common_variables().
@ sql/mysqld.cc
Due to a bug in Sun Studio compiler when generating 32 bit code the
initialization of the def_value member of the default-storage-engine entry
in my_long_options only got 4 bytes allocated instead of 8 bytes.
The compiler bug was triggered by casting a 32 bit pointer to a 64 bit
integer value in the initialization code for my_long_options. To avoid
triggering the compiler bug the intialization of the def_value in
my_long_options is set to 0 and instead the default_storage_engine
is initialized to "MyISAM" in init_common_variables().

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/96895

3826 Olav Sandstaa	2010-01-14 [merge]
      Merge of fix for Bug#50221

Pushed to:

mysql-next-mr-bugfixing: olav@sun.com-20100114123942-ie1pbuub6qg6ll9n
mysql-6.0-codebase-bugfixing: olav@sun.com-20100114131756-inc330r600raciyn

Thanks to this bugfix, Random Query Generator tests are back in business on "sol10 sparc32 debug_max" platform in Pushbuild. Thank you, Olav!

Pushed into 6.0.14-alpha (revid:alik@sun.com-20100121083501-but9pj2g3zmu10md) (version source revid:alik@sun.com-20100119194323-gcog2uiox2b7wsln) (merge vers: 6.0.14-alpha) (pib:16)

Pushed into mysql-next-mr (revid:luis.soares@sun.com-20100120170149-oa424hmew2rmwsr4) (version source revid:alik@sun.com-20100119194310-sq5yza06ddhwt0di) (pib:16)

Noted in 6.0.14 changelog.

mysqld did not recognize most of the options given on the command
line when it was compiled for 32-bit Solaris using the Sun Studio
compiler. 

Setting report to NDI pending push to Celosia.

Pushed into 5.5.3-m3 (revid:alik@sun.com-20100306103849-hha31z2enhh7jwt3) (version source revid:vvaintroub@mysql.com-20100122115033-udonw413fe2nal5p) (merge vers: 5.5.99-m3) (pib:16)

Noted in 5.5.3 changelog.

Correction: No changelog entry needed.