MySQL Bugs: #1553: not able to run mysql in lynxos-powerpc

Bug #1553	not able to run mysql in lynxos-powerpc
Submitted:	15 Oct 2003 2:42	Modified:	27 Sep 2008 9:22
Reporter:	Raja Subbiah	Email Updates:
Status:	Won't fix	Impact on me:	None
Category:	MySQL Server: Compiling	Severity:	S1 (Critical)
Version:	mysql-4.0.15a (Source distribution)	OS:	Any (LynxOS PowerPC)
Assigned to:	Lenz Grimmer	CPU Architecture:	Any

Description:
I tried to install mysql on PowerPC-LynxOS machine, and got into the
following issues. If some body can help me resolve this, it would be of
great help.

How to repeat:
First I like to list the changes I did to compile mysql for PowerPC-LynxOS
using cross compilers in sparc-sunos.

Configure script changes
-------------------------
1) Changed ps options in lines which find out whether a particular process
is running or not. In LynxOS it is ps -ax. In LynxOS scripts are shown as
bash process, hence ps -ax | grep process_name will not fetch one for
configure since it's process name is marked as bash.

2) In LynxOS for obtaining thread related functions you have to
use -mthreads options, there is no thread library.  Hence added check
for -mthreads in configure script when checking for thread related features.

3) I was using cross compiler environment, configure script was trying to
execute some binary produced for the host platform which failed, so I
started with config outputs generated for a similar environment(LynxOS/x86)
and tweaked it for the host.

Makefile/Code changes
---------------------
1) I was using cross compiler environment, hence gen_lex_hash and
conf_to_src compiled for the host platform didn't ran on the build
environment, so I have generate these binary for build environment and
modify Makefile to use them.

Also removed -O3 flag in sql/Makefile since it was reporting undefined
symbols with that option, In the following link,
http://gcc.gnu.org/ml/gcc-prs/2002-11/msg01111.html
they have adviced to remove that option for compiling it

2) vi_mode.c - Added the following
#  include <my_global.h>

3) my_pthread.h - LynxOS was offering a nonposix version of sigwait, so I
had to modify the my_sigwait macro def in
my_pthread.h to the below
....
#ifndef HAVE_NONPOSIX_SIGWAIT
#define my_sigwait(A,B) *B = sigwait((A),NULL)
#else
....

4)system include file : sem.h
Commented out mutex_exit macro in sem.h since it collides with mutex_exit
function of innobase

Some info about compilation :

Release:       mysql-4.0.15a (Source distribution)
C compiler:    2.95.3
C++ compiler:  2.95.3
Environment:
        <machine, os, target, libraries (multiple lines)>
System: LynxOS Decimus  0.7.0.0 PowerPC

GCC: Using builtin specs.
gcc version 2.9-gnupro-98r2
Compilation info: CC='gcc'  CFLAGS=''  CXX='g++'  CXXFLAGS=''  LDFLAGS=''
ASFLAGS=''
LIBC:
Configure command: ./configure '--prefix=/users/spraja/usr/local/mysql'
'--host=powerpc-unknown-lynxos'

Then things went fine, and I got the mysql bins. When I run it on
powerpc-lynxos,
faced the following issues/problems. This is where I require help

===============================================================
Decimus:/users/spraja/usr/local/mysql# bin/mysqladmin version
bin/mysqladmin: connect to server at 'localhost' failed
error: 'Lost connection to MySQL server during query'
==============================================================

So I tried telnet connection to 3306 port

==============================================================
Decimus:/users/spraja/usr/local/mysql# telnet Decimus 3306
Trying 172.16.254.93 ...
Connected to Decimus.
Escape character is '^]'.
(
4.0.15a\KBH$xQn  <-- I pressed enter key
                 <-- I pressed enter key pressed

Bad handshakeConnection closed by foreign host.
==============================================================

Log files content at time where

==============================================================
Log files (hostname.err)
031015 06:28:27  mysqld started
InnoDB: The first specified data file ./ibdata1 did not exist:
InnoDB: a new database to be created!
031015  6:28:28  InnoDB: Setting file ./ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
031015  6:28:29  InnoDB: Log file ./ib_logfile0 did not exist: new to be
created
InnoDB: Setting log file ./ib_logfile0 size to 5 MB
InnoDB: Database physically writes the file full: wait...
031015  6:28:30  InnoDB: Log file ./ib_logfile1 did not exist: new to be
created
InnoDB: Setting log file ./ib_logfile1 size to 5 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
031015  6:28:33  InnoDB: Started
/users/spraja/usr/local/mysql/libexec/mysqld: ready for connections.
Version: '4.0.15a'  socket: '/tmp/mysql.sock'  port: 3306

031015  6:29:16  Error on new connection socket: Socket operation on
non-socket
031015  6:29:26  Error on new connection socket: Socket operation on
non-socket
031015  6:31:39  Error on new connection socket: Socket operation on
non-socket
031015  6:32:08  Error on new connection socket: Socket operation on
non-socket

mysqld got signal 6;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose
the problem, but since we have already crashed, something is definitely
wrong
and this may fail.

key_buffer_size=16384
read_buffer_size=131072
max_used_connections=0
max_connections=100
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections =
19215 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

free: pid[91] : Double free of 0x215b8e68 allocated at
0x10374c44:0x10374a90:0x10378d98
==============================================================

I tried stopping the server, but in vain

==============================================================
Decimus:/users/spraja/usr/local/mysql# ./share/mysql/mysql.server stop
Killing mysqld with pid 108
Wait for mysqld to exit\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
.\c
 gave up waiting!
==============================================================

Looks like problem with unix sockets and/or compilation.

Can you please try connecting via TCP/IP and see if it gives same error?

mysql -h 127.0.0.1 -P 3306

(or whatever port mysqld is listening)

use 127.0.0.1 to make sure client connects via TCP/IP. I hope you are aware that -h localhosy instructs clients to connect via mysql.sock. With fully qualified domain names it can happen too depending on how they are resolved.

<< Tried tcp communications >>

Decimus:/mnt/dvp/spraja/usr/local/mysql# bin/mysqladmin -h 172.16.254.93 -P 3306 version
bin/mysqladmin  Ver 8.40 Distrib 4.0.15a, for unknown-lynxos on powerpc
Copyright (C) 2000 MySQL AB & MySQL Finland AB & TCX DataKonsult AB
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL license

Server version          4.0.15a
Protocol version        10
Connection              172.16.254.93 via TCP/IP
TCP port                3306
Uptime:                 1 min 24 sec

Threads: 1  Questions: 1  Slow queries: 0  Opens: 5  Flush tables: 1  Open tables: 0  Queries per second avg: 0.012

<< But here I noticed that when I run mysqladmin mysqld is restarting (log attached below) >>

<< Tired mysqlshow >>
Decimus:/mnt/dvp/spraja/usr/local/mysql# bin/mysqlshow -h 172.16.254.93 -P 3306  
<< hanged >>

<< Tried mysql >>
Decimus:/mnt/dvp/spraja/usr/local/mysql# bin/mysql -h 172.16.254.93 -P 3306 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1 to server version: 4.0.15a

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> connect mysql
Connection id:    2
Current database: mysql

mysql> SELECT host,db,user FROM db
    -> ;
ERROR 2013: Lost connection to MySQL server during query
mysql> quit;
Bye

<< Again mysqld was restarted >>

<< Log hostname.err file signal 6 -- SIGABRT in LynxOS>>

mysqld got signal 6;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=16384
read_buffer_size=131072
max_used_connections=0
max_connections=100
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 19215 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

<< free: pid[78] : Double free of 0x215a9798 allocated at 0x10374c44:0x10374a90:0x10378d98  >>

Number of processes running now: 0
031015 12:53:36  mysqld restarted
031015 12:53:36  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 43892
InnoDB: Doing recovery: scanned up to log sequence number 0 43892
031015 12:53:37  InnoDB: Flushing modified pages from the buffer pool...
031015 12:53:37  InnoDB: Started
/users/spraja/usr/local/mysql/libexec/mysqld: ready for connections.
Version: '4.0.15a'  socket: '/tmp/mysql.sock'  port: 3306

I looked at the code and made the following changes
-- In libmysql/libmysql.c
 Set timeout to zero in my_connect function. 
 The code uses fcntl calls to make the socket non-blocking and blocking, this seems to have problems in lynxos [Only after this change I was able to see the version in mysqladmin via tcp/ip which I missed it saying before]
-- Edit sql/mysqld.cc
   Comment out call to getsockname and subsequent continue. 
   getsockname on unix sockets in lynxos is returning error and this causes mysqld to terminate the client connection.
-- Added full debug, in doing so I got un-defined references to lot of symbols, so I inserted the following lines to innobase/include/univ.i, innonbase/include/univold.i, innobase/include/univoldmysql.i

            #undef UNIV_NONINL
            #undef UNIV_INLINE
            #define UNIV_NONINL
            #define UNIV_INLINE

After this I basically did two builds 
    B1 - Innobase and Bdb full debug
    B2 - Bdb full debug

I have attached all the traces I got so while running them

B2 server
client.bdb_trace_admin_version - mysqladmin unix socket
client.bdb_trace_admin_version_tcp - mysqladmin tcp socket
client.bdb_trace_mysql - mysql unix 
client.bdb_trace_mysql_tcp - mysql tcp
client.bdb_trace_mysqlshow - mysqlshow unix
client.bdb_trace_mysqlshow_tcp - mysqlshow tcp
mysqld.trace.bdb - mysqld trace
clients.bdb_stdout - clients stdout
mysqld.bdb_stdout - mysqld stdout

B1 server
myqld.inno_bdb_stdout - mysqld stdout

This server did not start at all, It was facing issues (Innobase assertion failure or signal 11)

To sum up the problems
   - fcntl calls in my_connect
   - getsockname for unix sockets
   - vio_read error returned -1
   - freeing wrong aligned pointer
   - Hangs while running mysqlshow and mysql
   - Innobase assertion failure
   - mysqld getting signal 11

Trace files in zip format

Attachment: trace.zip (application/x-zip-compressed, text), 14.43 KiB.

Attaching again, first is one not good

Attachment: trace.zip (application/x-zip-compressed, text), 14.43 KiB.

trace

Attachment: client.bdb_trace_admin_version (text/plain), 9.06 KiB.

trace

Attachment: client.bdb_trace_admin_version_tcp (text/plain), 9.08 KiB.

Trace files in zip format

Attachment: client.bdb_trace_mysql (text/plain), 13.35 KiB.

Trace

Attachment: client.bdb_trace_mysql_tcp (text/plain), 13.09 KiB.

Trace

Attachment: client.bdb_trace_mysqlshow (text/plain), 3.59 KiB.

trace

Attachment: client.bdb_trace_mysqlshow_tcp (text/plain), 3.61 KiB.

trace

Attachment: clients.bdb_stdout (text/plain), 5.25 KiB.

trace

Attachment: myqld.inno_bdb_stdout (text/plain), 2.00 KiB.

trace

Attachment: mysqld.bdb_stdout (text/plain), 1.74 KiB.

trace

Attachment: mysqld.trace.bdb (text/plain), 11.05 KiB.

Sorry, I tried to add all trace files in a single zip, but it didn't work
So I'm attaching all of them seperately

I was able to solve most of the issues 
   - fcntl calls in my_connect -- Commented out, using blocking service
   - getsockname for unix sockets -- Commented out
   - vio_read error returned -1 -- not a problem, it is rescheduling the read
   - freeing wrong aligned pointer -- This is not causing problems, but ...
   - Hangs while running mysqlshow and mysql -- was due to corrupted pthread_setschedparam
   - Innobase assertion failure -- memory problems, decreased max_connections to 10
   - mysqld getting signal 11 -- memory problems, decreased max_connections to 10

Was able to solve some more
   - fcntl calls in my_connect -- poll was not implemented, library function call was there but the call returned "not implemented", Hence used the select alternative
   - freeing wrong aligned pointer -- Align size is 4 and not sizeof(double)
   - getsockname for unix sockets -- Looks like lynxos doesn't supports getsockname for unix sockets, comment them

Remaining issues (2 in number)
   1. Hangs while running mysqlshow and mysql - this happens in pthread_setschedparam. Not all calls to the function are failing, only the call during query dispatch gets hanged [Trace with printfs whenever pthread_setschedparam called]

Patriots# ./mysqld --debug=d,info,error,query,general,where:O,/tmp/mysqld.trace --user=mysql
[76/72] pthread_setschedparam(72,0x400000,9) -- <ret 0, err 2> Done
[76/72] pthread_setschedparam(60,0x400000,6) -- <ret 0, err 0> Done
[76/72] pthread_setschedparam(47,0x400000,6) -- <ret 0, err 0> Done
[76/72] pthread_setschedparam(52,0x400000,6) -- <ret 0, err 0> Done
[76/72] pthread_setschedparam(24,0x400000,6) -- <ret 0, err 0> Done
031027  4:50:34  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 49103
InnoDB: Doing recovery: scanned up to log sequence number 0 49103
InnoDB: Last MySQL binlog file position 0 1299, file name ./Patriots-bin.002
031027  4:50:34  InnoDB: Flushing modified pages from the buffer pool...
[76/72] pthread_setschedparam(15,0x400000,6) -- <ret 0, err 17> Done
[76/72] pthread_setschedparam(74,0x400000,6) -- <ret 0, err 17> Done
[76/72] pthread_setschedparam(46,0x400000,6) -- <ret 0, err 17> Done
031027  4:50:34  InnoDB: Started
./mysqld: ready for connections.
Version: '4.0.15a-debug-log'  socket: '/tmp/mysql.sock'  port: 3306
031027  4:50:53  Error on new connection socket: Socket operation on non-socket
[76/59] pthread_setschedparam(59,0x400000,6) -- [[ *** Hang *** ]]

   2. Innobase assertion failure. This happens only at start of mysqld after killing the running server with signal 9(SIGKILL). pthread_cond_init returns 16(EBUSY) which happens when pthread_cond_init is called for the second time on the same pthread condition.
Patriots# ./mysqld --debug=d,info,error,query,general,where:O,/tmp/mysqld.trace --user=mysql
[52/59] pthread_setschedparam(59,0x400000,9) -- <ret 0, err 2> Done
pthread_cond_init <ret = 16, err = 0,Error number not set>
031027  6:54:12  InnoDB: Assertion failure in thread 59 in file os0sync.c line 148
InnoDB: Failing assertion: 0 == i
InnoDB: We intentionally generate a memory trap.
InnoDB: Send a detailed bug report to mysql@lists.mysql.com
InnoDB: Thread 59 stopped in file os0sync.c line 148

Was able to solve the rem 2 issues
1. Hangs while running mysqlshow and mysql - this happens in
pthread_setschedparam. Not all calls to the function are failing, only the call
during query dispatch gets hanged -- This was actually due to invalid sigwait call. Now I'm using sigwait offered by mysql instead of using the one in the system
2. pthread_cond_init - was due to bad library

Now I'm able to run mysql and do simple query operation
Should do more testing

To Sum up the changes I did are

a. Edit config.guess to include the following lines

i*86:LynxOS:*)
echo i386-unknown-lynxos${UNAME_RELEASE}
exit 0 ;;
PowerPC:AxiOS:*)
echo powerpc-unknown-lynxos${UNAME_RELEASE}
exit 0 ;;

b. Edit configure
1. Portion of the script which tries to figure out the way for finding out whether a process is running or not. In Lynxos we use ps -ax and scripts will appear only with process name =bash

elif $PS -ax 2> /dev/null | grep bash > /dev/null
then
FIND_PROC="$PS -ax | grep mysqld | grep \" \$\$PID \" > /dev/null"

2. Portion of the script which tries to figure out thread library. Use
-mthreads here

if test "$with_posix_threads" = "no"
then
# This is for FreeBSD
LIBS="$ac_save_LIBS -mthreads"
TOOLS_LIBS="$ac_save_TOOLS_LIBS -mthreads" :13481
echo "$as_me:$LINENO: checking \"for pthread_create in -mthreads\"" >&5
echo $ECHO_N "checking \"for pthread_create in -mthreads\"... $ECHO_C" >&6;

c. Edit some of the sources
1. Edit config.h and includes/my_config.h.
Ensure that HAVE_FCONVERT and HAVE_POLL is not defined and SOCKET_SIZE_TYPE is defined to socklen_t
2. readline/vi_mode.c : Add line # include <my_global.h> in the begining after inclusion of config.h
3. Add -ldb -lbsd to libraries in client, sql and extra Makefile's. Change both the vars CLIENT_LIBS and LIBS
4. Edit include/my_pthread.h. lynxos provides non-posix sigwait. Need to change my_sigwait macro definition to
#define my_sigwait(A,B) mysql_sigwait(A,B)
Edit mysys/my_pthread.c
Remove the preprocessor conditions (2 #if ) which guard mysql sigwait implementation at line 41
Rename sigwait function to mysql_sigwait
5. Unfortunately some of the system provided macros collide with innobase functions.
Work around here is to
5.1 comment out these macros in system header files
Edit /usr/include/sem.h and comment mutex_enter, mutex_exit and mutex_create (OR)
5.2. Change all ref to mutex_enter, mutex_exit and mutex_create to innobase_mutex_enter, innobase_mutex_exit and innobase_mutex_create.
6. Lynxos install command doesn't supports wild card in source, need to change these.
This feature is used in sql/share and mysql-test makefiles.
Work around is to use install-sh provided by mysql instead of install command
7. When running with debug enabled, add the following lines to innobase/include/univ.i, innonbase/include/univold.i, innobase/include/univoldmysql.i
#undef UNIV_NONINL
#undef UNIV_INLINE
#define UNIV_NONINL
#define UNIV_INLINE
There is problem with inlining and compiler optimization.

d. Some more changes to make mysql run properly
Eventhough the above changes will suffice to build mysql, to make it run properly following are required
1. Edit sql/mysqld.cc
Comment out call to getsockname and subsequent continue.
getsockname on unix sockets is returning error and this causes mysqld to terminate the client connection.
2. Using priority is problem, so comment call to pthread_setschedparam in mysys/my_pthread.c:41
3. Comment mysys/my_thr_init.c:194, as this free is causing double free.
4. If you enable debug change include/my_global.h :650 to
#define ALIGN_SIZE(A) MY_ALIGN((A),4)

One more change is required to allow drop database
Edit  mysys/my_symlink.c and check for ENXIO error value in 
    if (((my_errno=errno) == EINVAL)
       || ((my_errno=errno) == ENXIO)) 
since in LynxOS it is returing ENXIO when link is not present

I have tested my mysql port and it seems reasonable.
Even though I haven't run any regression or full functional test, It seems okay for my little application.

Raja, thanks a lot for sharing your experiences and please excuse that I did 
not comment on this report yet. Unfortunately we currently don't have 
access to a LynxOS box, so it's not possible for me to apply and verify 
your modifications. Would it be possible for you to send me all your 
modifications as a single patch file (unified or context diff)? You can simply 
attach it to this report. This would make it easier to review your 
modifications and to decide, how we can add them without breaking existing 
code. Thanks in advance!

Command used to generate patch:
diff -r mysql-4.0.15a bkp/mysql-4.0.15a

mysql-4.0.15a - Modified code repository
bkp/mysql-4.0.15a - Original code repository

Note :
1. I'm using modified lynxOS kernel and libs. It should be close to the original, but not sure.

2. I used sed script to replace all occurrences of mutex_create, mutex_exit and mutex_enter functions. While doing so It also replaced occurrences of them in other words/places were this is not required. I ignored them since changing them manually is painful and I'm not a sed expert to make it work properly.

3. The Patches are not clean. It has lot of #if 0 and #if 1 which needs to be changed to something like #if LYNXOS

4. With lynxos-ppc it ran fine, with lynxos-x86 it is giving assertion failure some times (pthread_mutex_init fails) when it is killed and restarted, Debugging on this

Sorry for the delay, I was out for vaccation.

Patch for lynxos (diff file)

Attachment: mysql-4.0.15a-lynxos-patch (application/octet-stream, text), 113.62 KiB.

Patch Description :
sql_string.cc - LynxOS has fconvert but no floatingpoint.h 
config.guess - I have different version not the standard one
config.h.in - LynxOS has ushort but no uint
configure - install in LynxOS doesnot support wild-cards but present in system, should use ps -ax to find process, use -mthreads for thread lib, check for ushort,  LynxOS has a poll which will not work
my_global.h - ushort present no uint, Align size is 4
my_pthread.h and my_pthread.c - LynxOS has sigwait but a non standard one with different signature, used sigwait implementation of mysql, to avoid name collision renamed it to mysql_sigwait
innobase - mutex_create, mutex_exit and mutex_enter name changes, already found in system, avoided inlining, causes problems when compiled with debug enabled
field.cc - LynxOS has fconvert but no floatingpoint.h. no sfconvert but has fconvert.
my_symlink.c - LynxOS returns ENXIO If link not found
mysqld.cc - getsockname is broken

I am changing the status to "deferred" until we have access to a LynxOS 
system, so we can verify and test this properly.

Also had problems with thread alarming. threads which asked for SIGUSR1 signal din't have corresponding handlers. In LynxOS flavour signal handlers are on a thread basis. So added SIGUSR1 handler to all threads (my_thread_init and thr_alarm function)

We have no plans to include LynxOS into the list of supported platforms. This may change in future.