MySQL Bugs: #4408: Static binary built with ICC fails to substitute as mysql user

Bug #4408	Static binary built with ICC fails to substitute as mysql user
Submitted:	5 Jul 2004 10:34	Modified:	7 Oct 2004 11:12
Reporter:	Alexander Keremidarski	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Compiling	Severity:	S1 (Critical)
Version:	4.0.20	OS:	Linux (Linux)
Assigned to:	Lenz Grimmer	CPU Architecture:	Any

Description:
mysql-standard-4.0.20-pc-linux-i686-icc.tar.gz

fails to start with an error:

040705 11:33:50  mysqld started
Fatal error: Can't change to run as user 'mysql' ;  Please check that the user exists!
040705 11:33:50  Aborting
                                                                                
040705 11:33:50  /root/mysql-standard-4.0.20-pc-linux-i686-icc/bin/mysqld: Shutdown Complete
                                                                                
040705 11:33:50  mysqld ended

The problem seems to be happen with static build only.

How to repeat:
Install mysql-standard-4.0.20-pc-linux-i686-icc.tar.gz and run mysqld_safe as root.

I also included this in another bug but it wasn't the main point.

Seeing now if I can repeat it.

Salle: do you still have this problem? I can't repeat it with the binary you said.
Does mysql user exist on your system (I guess yes) ?

Could not repeat it with the specified binary, neither on my machine nor on another.

doing some tests with Salle (specific to FC2 apparently)

I tested it on three of our Linux build hosts (running SuSE Linux 7.3, 8.2 and RHAS3) and was not
able to reproduce it :(

Not caused by icc. Using a MySQL binary built on the same machine as the icc one, but with gcc, leads to the same error. Probably related to this compiler's/linker's error message:
mysqld.o(.text+0x12a8): In function `set_user':
/home/mysql_src/mysql-4.0/sql/mysqld.cc:1081: Using 'initgroups' in statically linked applications requires at
+runtime the shared libraries from the glibc version used for linking
mysqld.o(.text+0x1193):/home/mysql_src/mysql-4.0/sql/mysqld.cc:1045: Using 'getpwnam' in statically linked
+applications requires at runtime the shared libraries from the glibc version used for linking

Yes, the problem lies here:
mysqld.o(.text+0x12a8): In function `set_user':
/home/mysql_src/mysql-4.0/sql/mysqld.cc:1081: Using 'initgroups' in statically
linked applications requires at runtime the shared libraries from the glibc version used for linking

It's not really possible anymore to make a really static binary. Since some recent version, glibc by default does not include libnss and libresolv; when you link statically, those 2 libraries are not included; resulting binary is dynamic for those 2 libraries: it needs them at run time. The reason for this is that NSS is something which by design can work differently on different machines.
A solution is to link with a glibc configured with --enable-static-nss, but this is discouraged:
http://www.delorie.com/gnu/docs/glibc/libc_690.html
and the glibc FAQ which addresses this issue: http://www.haskell.org/pipermail/haskell-cafe/2004-May/006176.html
So the only sound solution is making our binaries all dynamic, not static anymore. With the drawback that we cannot use a patched glibc (which brings again all the problems with LinuxThreads' stack size - but NPTL is gaining acceptance).
I'm raising to "Critical" as it was seen on a common Linux (Fedora Core 2) and is making the binary just unusable for unexperimented users. We need to discuss the solution.

Originaly reported on RedHat AS 3

Solving this bug properly requires some internal discussion on how to proceed with newer
Linux systems that run on glibc-2.3 - there are various options that need to evaluated.
Therfore I leave this bug open until we found a solution that works.

BUG#4265 has been marked as a duplicate of this one.

BUG#4215 has been marked as a duplicate of this one.

Hello,
I am using a glibc 2.3.2, Kernel 2.6.8.1, Debian-Woody and have the same problem, when starting the icc-binary of mysql.
I'm not sure whether or not this is a solution, but I can start mysql the following way:

su -c "bin/mysqld_safe &" mysql

I can't tell you, if there are other problems, running mysql this way, since I haven't really tested the intallation right now. 

I also had to replace the "resolveip"-program with the one, that comes with the gcc-compiled mysql-version. The icc-"resolveip"-program doesn't resolve anything.

Best regards,
Marcus

I have experience this same problem with the static icc build on RedHat AS 3. Installing the gcc build worked just find. My system has libresolv.so.2 and various libnss* files in /lib but they are all gcc libraries and not icc. Would it be possible to distribute the needed icc libs or provide a pointer to where they can be found?

BUG#5301 has been marked as a duplicate of this one.

Beginning with MySQL 4.1.4, the Linux binary compiled with the Intel icc compiler will
be linked dynamically against glibc-2.3 instead of being statically linked. This should resolve
all issues that this binary has regarding changing the user ID and other NSS related problems.

The downside of this is that one needs to install some Intel compiler related shared libraries,
namely libcprts.so.5, libcxa.so.5 and libunwind.so.5

These need to be installed into a place where the dynamic linker can find them, e.g. /usr/local/lib.

Future versions of MySQL compiled with icc will include these libs as part of the binary package 
- for now, they can be obtained as a separate package from here:

ftp://ftp.mysql.com/pub/mysql/download/icc-8.0-libs.tar.gz
MD5 cecksum: 0970bbfb9fb076dfe995b3176bf90fad

as far as I understand 4.0.21-icc isn't built statically either

Here's a trace demonstrating this behaviour:

[root@/]#  /opt/mysql-standard-4.0.20-pc-linux-i686-icc/bin/mysqld
--user=mysql  --basedir=/opt/mysql --datadir=/mysql/mysql 
--pid-file=/var/run/mysqld/mysqld.pid --skip-locking --port=3306
--socket=/mysql/mysql/mysql.sock
Fatal error: Can't change to run as user 'mysql' ;  Please check that
the user exists!
040908  9:31:05  Aborting
                                                                                
040908  9:31:05 
/opt/mysql-standard-4.0.20-pc-linux-i686-icc/bin/mysqld: Shutdown
Complete
                                                                                

[root@djali /]# strace
/opt/mysql-standard-4.0.20-pc-linux-i686-icc/bin/mysqld --user=mysql 
--basedir=/opt/mysql --datadir=/mysql/mysql 
--pid-file=/var/run/mysqld/mysqld.pid --skip-locking --port=3306
--socket=/mysql/mysql/mysql.sock
execve("/opt/mysql-standard-4.0.20-pc-linux-i686-icc/bin/mysqld",
["/opt/mysql-standard-4.0.20-pc-linux-i686-icc/bin/mysqld",
"--user=mysql", "--basedir=/opt/mysql", "--datadir=/mysql/mysql",
"--pid-file=/var/run/mysqld/mysqld.pid", "--skip-locking",
"--port=3306", "--socket=/mysql/mysql/mysql.sock"], [/* 29 vars */]) = 0
uname({sys="Linux", node="djali", ...}) = 0
fcntl64(0, F_GETFD)                     = 0
fcntl64(1, F_GETFD)                     = 0
fcntl64(2, F_GETFD)                     = 0
geteuid32()                             = 0
getuid32()                              = 0
getegid32()                             = 0
getgid32()                              = 0
getrlimit(RLIMIT_STACK, {rlim_cur=10240*1024, rlim_max=RLIM_INFINITY}) =
0
setrlimit(RLIMIT_STACK, {rlim_cur=2044*1024, rlim_max=RLIM_INFINITY}) =
0
getpid()                                = 4628
rt_sigaction(SIGRTMIN, {0x8254c10, [], SA_RESTORER, 0x8273ae8}, NULL, 8)
= 0
rt_sigaction(SIGRT_1, {0x8254c40, [], SA_RESTORER, 0x8273ae8}, NULL, 8)
= 0
rt_sigaction(SIGRT_2, {0x8254cf0, [], SA_RESTORER, 0x8273ae8}, NULL, 8)
= 0
rt_sigprocmask(SIG_BLOCK, [RTMIN], NULL, 8) = 0
_sysctl({{CTL_KERN, KERN_VERSION}, 2, 0xbfffe0a0, 31, (nil), 0}) = 0
brk(0)                                  = 0x84a5000
brk(0x84a6000)                          = 0x84a6000
rt_sigaction(SIGILL, {0x8257750, ~[], SA_RESTORER, 0x8273ae8},
{SIG_DFL}, 8) = 0rt_sigprocmask(SIG_BLOCK, NULL, [RTMIN], 8) = 0
rt_sigaction(SIGILL, {SIG_DFL}, NULL, 8) = 0
open("/etc/localtime", O_RDONLY)        = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=1017, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0xb7dfe000
read(3, "TZif\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\4\0\0\0\4\0"...,
4096) = 1017
close(3)                                = 0
munmap(0xb7dfe000, 4096)                = 0
time(NULL)                              = 1094662043
uname({sys="Linux", node="djali", ...}) = 0
stat64("/etc/my.cnf", {st_mode=S_IFREG|0644, st_size=351, ...}) = 0
open("/etc/my.cnf", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=351, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0xb7dfe000
read(3, "[client]\nsocket\t\t= /mysql/mysql/"..., 4096) = 351
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0xb7dfe000, 4096)                = 0
stat64("/usr/local/mysql/data/my.cnf", 0xbfffd028) = -1 ENOENT (No such
file or directory)
stat64("/root/.my.cnf", 0xbfffd028)     = -1 ENOENT (No such file or
directory)
lstat64("/opt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/opt/mysql", {st_mode=S_IFLNK|0777, st_size=35, ...}) = 0
readlink("/opt/mysql", "mysql-standard-4.0.20-pc-linux-i686", 4096) = 35
lstat64("/opt/mysql-standard-4.0.20-pc-linux-i686",
{st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
rt_sigaction(SIGALRM, {0x8257750, [], SA_RESTORER, 0x8273ae8},
{SIG_DFL}, 8) = 0rt_sigprocmask(SIG_UNBLOCK, [ALRM], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigaction(SIGSEGV, {0x8257750, [], SA_RESTORER|SA_NOMASK|SA_ONESHOT,
0x8273ae8}, NULL, 8) = 0
rt_sigaction(SIGABRT, {0x8257750, [], SA_RESTORER|SA_NOMASK|SA_ONESHOT,
0x8273ae8}, NULL, 8) = 0
rt_sigaction(SIGBUS, {0x8257750, [], SA_RESTORER|SA_NOMASK|SA_ONESHOT,
0x8273ae8}, NULL, 8) = 0
rt_sigaction(SIGILL, {0x8257750, [], SA_RESTORER|SA_NOMASK|SA_ONESHOT,
0x8273ae8}, NULL, 8) = 0
rt_sigaction(SIGFPE, {0x8257750, [], SA_RESTORER|SA_NOMASK|SA_ONESHOT,
0x8273ae8}, NULL, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGTERM, {0x8257750, [], SA_RESTORER, 0x8273ae8}, NULL, 8)
= 0
rt_sigaction(SIGHUP, {0x8257750, [], SA_RESTORER, 0x8273ae8}, NULL, 8) =
0
rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT PIPE TERM TSTP], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT PIPE TERM TSTP RTMIN], NULL,
8) = 0
open("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/Index", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=621, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0xb7dfe000
read(3, "# sql/share/charsets/Index\n#\n# T"..., 4096) = 621
brk(0x84a7000)                          = 0x84a7000
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0xb7dfe000, 4096)                = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/dec8.conf", {st_mode=S_IFREG|0644, st_size=4384, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/dos.conf", {st_mode=S_IFREG|0644, st_size=4383, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/german1.conf", {st_mode=S_IFREG|0644, st_size=4387, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/hp8.conf", {st_mode=S_IFREG|0644, st_size=4383, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/koi8_ru.conf", {st_mode=S_IFREG|0644, st_size=4387, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/latin2.conf", {st_mode=S_IFREG|0644, st_size=4386, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/swe7.conf", {st_mode=S_IFREG|0644, st_size=4384, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/usa7.conf", {st_mode=S_IFREG|0644, st_size=4384, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/cp1251.conf", {st_mode=S_IFREG|0644, st_size=4386, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/danish.conf", {st_mode=S_IFREG|0644, st_size=4386, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/hebrew.conf", {st_mode=S_IFREG|0644, st_size=4386, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/win1251.conf", {st_mode=S_IFREG|0644, st_size=4618, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/estonia.conf", {st_mode=S_IFREG|0644, st_size=4400, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/hungarian.conf", {st_mode=S_IFREG|0644, st_size=4389, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/koi8_ukr.conf", {st_mode=S_IFREG|0644, st_size=4388, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/win1251ukr.conf", {st_mode=S_IFREG|0644, st_size=4471, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/greek.conf", {st_mode=S_IFREG|0644, st_size=4385, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/win1250.conf", {st_mode=S_IFREG|0644, st_size=4400, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/croat.conf", {st_mode=S_IFREG|0644, st_size=4385, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/cp1257.conf", {st_mode=S_IFREG|0644, st_size=4399, ...}) = 0
stat64("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/charsets/latin5.conf", {st_mode=S_IFREG|0644, st_size=4545, ...}) = 0
unlink("/mysql/mysql/djali.LOWER-TEST") = -1 ENOENT (No such file or
directory)
open("/mysql/mysql/djali.lower-test", O_RDWR|O_CREAT|O_LARGEFILE, 0666)
= 3
close(3)                                = 0
stat64("/mysql/mysql/djali.LOWER-TEST", 0xbfffe1e0) = -1 ENOENT (No such
file or directory)
unlink("/mysql/mysql/djali.lower-test") = 0
sched_setscheduler(4628, SCHED_OTHER, { 9 }) = -1 EINVAL (Invalid
argument)
sched_get_priority_max(SCHED_OTHER)     = 0
sched_get_priority_min(SCHED_OTHER)     = 0
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024}) = 0
getpid()                                = 4628
time(NULL)                              = 1094662043
open("/opt/mysql-standard-4.0.20-pc-linux-i686/share/mysql/english/errmsg.sys", O_RDONLY|O_LARGEFILE) = 3
read(3, "\376\376\2\1\1\0R/\361\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 32)
= 32
brk(0x84ab000)                          = 0x84ab000
read(3, "\0\0\10\0\20\0\23\0\27\0>\0f\0\222\0\302\0\367\0002\1l"...,
482) = 482
read(3, "hashchk\0isamchk\0NO\0YES\0Can\'t cre"..., 12114) = 12114
close(3)                                = 0
umask(06)                               = 022
chdir("/mysql/mysql/")                  = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
bind(3, {sa_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr("0.0.0.0")}, 16) = 0
listen(3, 50)                           = 0
geteuid32()                             = 0
socket(PF_FILE, SOCK_STREAM, 0)         = 4
connect(4, {sa_family=AF_FILE, path="/var/run/.nscd_socket"}, 110) = -1
ENOENT (No such file or directory)
close(4)                                = 0
open("/etc/nsswitch.conf", O_RDONLY)    = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=1686, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0xb7dfe000
read(4, "#\n# /etc/nsswitch.conf\n#\n# An ex"..., 4096) = 1686
brk(0x84ac000)                          = 0x84ac000
read(4, "", 4096)                       = 0
close(4)                                = 0
munmap(0xb7dfe000, 4096)                = 0
open("/etc/ld.so.cache", O_RDONLY)      = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=84709, ...}) = 0
old_mmap(NULL, 84709, PROT_READ, MAP_PRIVATE, 4, 0) = 0xb7dea000
close(4)                                = 0
open("/lib/libnss_files.so.2", O_RDONLY) = 4
read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20\35\0"...,
1024) = 1024
fstat64(4, {st_mode=S_IFREG|0755, st_size=51920, ...}) = 0
old_mmap(NULL, 46716, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) =
0xb7dde000
mprotect(0xb7de9000, 1660, PROT_NONE)   = 0
old_mmap(0xb7de9000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED,
4, 0xa000) = 0xb7de9000
close(4)                                = 0
open("/lib/tls/libc.so.6", O_RDONLY)    = 4
read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200X\1"...,
1024) = 1024
fstat64(4, {st_mode=S_IFREG|0755, st_size=1568924, ...}) = 0
close(4)                                = 0
munmap(0xb7dea000, 84709)               = 0
munmap(0xb7dde000, 46716)               = 0
write(2, "Fatal error: Can\'t change to run"..., 87Fatal error: Can't
change to run as user 'mysql' ;  Please check that the user exists!
) = 87
time(NULL)                              = 1094662043
write(2, "040908  9:47:23  ", 17040908  9:47:23  )       = 17
write(2, "Aborting\n", 9Aborting
)               = 9
write(2, "\n", 1
)                       = 1
time(NULL)                              = 1094662043
write(2, "040908  9:47:23  ", 17040908  9:47:23  )       = 17
write(2, "/opt/mysql-standard-4.0.20-pc-li"...,
75/opt/mysql-standard-4.0.20-pc-linux-i686-icc/bin/mysqld: Shutdown
Complete
) = 75
write(2, "\n", 1
)                       = 1
unlink("/var/run/mysqld/mysqld.pid")    = -1 ENOENT (No such file or
directory)
exit_group(1)                           = ?
[root@/]#

FYI, I think this is related. I'm using intel binaries of 4.0.21 (mysql-standard-4.0.21-pc-linux-i686-icc-glibc23.tar.gz)...

After reading through the previous notes, I managed to locate the missing intel libraries and got everything complied nicely. However, when attempting to build the mysql extension for PHP I got some unresolved symbols that indicate yet another missing intel library (libirc, I'm told)...

The main symbols missing were "__intel_fast_memcpy" and "_intel_proc_init". I will look further into this...

I have now updated the package that includes the Intel libraries and moved it to a more prominent place - they should appear on http://dev.mysql.com/downloads/os-linux.html soon. Once the mirrors have caugth up, you should be able to download the package from the following locations:

http://dev.mysql.com/Downloads/Linux/intel-icc8-libs-8.0-0.i386.rpm
http://dev.mysql.com/Downloads/Linux/intel-icc8-libs-8.0.tar.gz

MD5 checksum of the tarball should be: ed2693bf9e067865a30495f42d415eb4
The RPM has been GPG signed.