| Bug #64570 | mysql client and server binaries are not dynamically linked to libmysqlclient | ||
|---|---|---|---|
| Submitted: | 6 Mar 2012 21:42 | Modified: | 15 Oct 2012 14:18 |
| Reporter: | Clint Byrum | Email Updates: | |
| Status: | Won't fix | Impact on me: | |
| Category: | MySQL Server: Packaging | Severity: | S3 (Non-critical) |
| Version: | 5.5.21 | OS: | Linux |
| Assigned to: | CPU Architecture: | Any | |
| Tags: | Contribution | ||
[6 Mar 2012 21:42]
Clint Byrum
[6 Mar 2012 21:43]
Clint Byrum
patch to compile client programs dynamically
Attachment: mysql-5.5-compile-clientprograms-dynamically.patch (text/x-patch), 3.21 KiB.
[7 Mar 2012 5:06]
Valeriy Kravchuk
Thank you for the problem report and patch contributed.
[9 Mar 2012 9:28]
Davi Arnaut
The client library that the server uses is not the same as the one intended for being shared. Some of the clients also peek into the API internals, which might lead to ABI problems if linked to the shared library.
[10 Mar 2012 14:13]
Vladislav Vaintroub
What Davi says is correct. If clients were programmed differently and did not use bunch of internal undocumented functions of mysys, dbug components - all the stuff inside the static library that is not the official API, then one could consider linking clients with shared lib.
[10 Mar 2012 14:31]
Vladislav Vaintroub
Also, looking at the patch All clients are now linked with
SET(LIBS libmysql strings mysys ${SSL_LIBRARIES})
and this means that every client now has 2 copies of mysys and strings- one inside the shared library and one linked directly to client program. I do not think mysys or strings were written to account for such use (they got static data inside libraries, which now would differ in each copy). Besides linking with 2 more static libraries sort of defies the purpose of saving space.
Out of curiosity, how much space do you save with this patch (if you compare to link with static library/without -rdynamic flag)?
[12 Mar 2012 16:06]
Hartmut Holzgraefe
> If clients were programmed differently and did not use bunch of internal > undocumented functions of mysys, dbug components - all the stuff inside > the static library that is not the official API, then one could consider > linking clients with shared lib." I fail to understand how this suddenly starts to become an issue with 5.5? The commandline tools in 5.1 and before were always built against libmysqlclient.so by default (when using plain "./configure; make"), not .a, by default and as far as i can tell distribution packages were always built that way, too (i only checked with Ubuntu, Fedora and OpenSuse so far).
[12 Mar 2012 21:07]
Vladislav Vaintroub
Something becomes an issue if someone complains. Nobody had complained or noticed that within the last 2 years, so perhaps it is not as much of an issue. Now, if one want a shared library to be the very same as static then there is a very simple specific hack for Linux (assuming static lib is PIC compiled): When linking shared library, include static library as the whole and use -rdynamic to link --Wl,-whole-archive /path/to/mysql-static-lib --Wl,-no-whole-archive -rdynamic That's it basically, you can link the result with every utility, and it will link just fine. This exports all symbols, just as in the bad autotools past, and it is also as dangerous. People who complain now that MySQL utilities are statically linked, will complain in the future about changed ABI, whenever an obscure non-API function used by command line client (e.g bmove_upp) is renamed or removed. Other distros (Fedora, Mageia) have understood the ABI compatibility and symbol pollution issues are now building MySQL rsp MariaDB 5.5 with a linker script that restricts visibility of what libmysqlclient.so export only to the official API symbols. They do not try to squeeze every bit of the disk space and are linking clients statically. I believe they are doing it right.
[14 Mar 2012 14:38]
Axel Schwenke
Let me see if I get this right: 1. in the past we exported every symbol from libmysqlclient.so (this was certainly bad) 2. we changed this, to export only API funktions (good) 3. our own clients use non-API functions and thus don't work with the new dynamic lib. There is a word for this: it's a bug. 4. we are too lazy to fix the clients (or extend the API). So we take the short path and link the cliesnt with the static lib. I also have a word for this: disgusting.
[14 Mar 2012 15:34]
Davi Arnaut
It also significantly reduces the ability to fix issues, specially in stable releases, associated with the client tools and libraries that are being linked dynamically.
[16 Mar 2012 19:56]
Clint Byrum
The per-binary savings is relatively large, note that in 5.1 we shipped some that appear to be statically linked too, but others that were not: clint@clint-MacBookPro:~/pkg/mysql-5.5/bzr$ dpkg -c mysql-client-5.1_5.1.58-1ubuntu5_amd64.deb|grep bin drwxr-xr-x root/root 0 2012-02-29 11:05 ./usr/sbin/ -rwxr-xr-x root/root 2209384 2012-02-29 11:05 ./usr/sbin/mysqlmanager drwxr-xr-x root/root 0 2012-02-29 11:05 ./usr/bin/ -rwxr-xr-x root/root 1226 2012-02-29 11:04 ./usr/bin/mysql_fix_extensions -rwxr-xr-x root/root 3245 2012-02-29 11:04 ./usr/bin/mysql_find_rows -rwxr-xr-x root/root 8869656 2012-02-29 11:05 ./usr/bin/mysql_client_test_embedded -rwxr-xr-x root/root 8596760 2012-02-29 11:05 ./usr/bin/mysqltest_embedded -rwxr-xr-x root/root 30936 2012-02-29 11:05 ./usr/bin/mysqlimport -rwxr-xr-x root/root 109904 2012-02-29 11:05 ./usr/bin/mysqldump -rwxr-xr-x root/root 1431464 2012-02-29 11:05 ./usr/bin/perror -rwxr-xr-x root/root 39016 2012-02-29 11:05 ./usr/bin/mysqlreport -rwxr-xr-x root/root 1743016 2012-02-29 11:05 ./usr/bin/myisam_ftdump -rwxr-xr-x root/root 400192 2012-02-29 11:05 ./usr/bin/innotop -rwxr-xr-x root/root 7351 2012-02-29 11:04 ./usr/bin/mysqldumpslow -rwxr-xr-x root/root 14192 2012-02-29 11:05 ./usr/bin/innochecksum -rwxr-xr-x root/root 173608 2012-02-29 11:05 ./usr/bin/mysql_waitpid -rwxr-xr-x root/root 38640 2012-02-29 11:05 ./usr/bin/mysqladmin -rwxr-xr-x root/root 52672 2012-02-29 11:05 ./usr/bin/mysqlslap -rwxr-xr-x root/root 29552 2012-02-29 11:05 ./usr/bin/mysqlshow -rwxr-xr-x root/root 12070 2012-02-29 11:04 ./usr/bin/mysqlbug -rwxr-xr-x root/root 111537 2012-02-29 11:04 ./usr/bin/mysqlaccess -rwxr-xr-x root/root 462496 2012-02-29 11:05 ./usr/bin/mysql_client_test lrwxrwxrwx root/root 0 2012-02-29 11:05 ./usr/bin/mysqlrepair -> mysqlcheck lrwxrwxrwx root/root 0 2012-02-29 11:05 ./usr/bin/mysqloptimize -> mysqlcheck lrwxrwxrwx root/root 0 2012-02-29 11:05 ./usr/bin/mysqlanalyze -> mysqlcheck clint@clint-MacBookPro:~/pkg/mysql-5.5/bzr$ dpkg -c mysql-client-core-5.1_5.1.58-1ubuntu5_amd64.deb |grep bin drwxr-xr-x root/root 0 2012-02-29 11:05 ./usr/bin/ -rwxr-xr-x root/root 187576 2012-02-29 11:05 ./usr/bin/mysql -rwxr-xr-x root/root 35552 2012-02-29 11:05 ./usr/bin/mysqlcheck Here is 5.5.20 with all statically linked binaries: clint@clint-MacBookPro:~/pkg/mysql-5.5/bzr$ dpkg -c mysql-client-5.5_5.5.20-0ubuntu3_amd64.deb |grep bin drwxr-xr-x root/root 0 2012-02-17 10:27 ./usr/bin/ -rwxr-xr-x root/root 3377880 2012-02-17 10:27 ./usr/bin/mysqlimport -rwxr-xr-x root/root 3395512 2012-02-17 10:27 ./usr/bin/mysqlslap -rwxr-xr-x root/root 111560 2012-02-17 10:05 ./usr/bin/mysqlaccess -rwxr-xr-x root/root 3797304 2012-02-17 10:27 ./usr/bin/mysql_client_test -rwxr-xr-x root/root 3448536 2012-02-17 10:27 ./usr/bin/mysqldump -rwxr-xr-x root/root 2885208 2012-02-17 10:27 ./usr/bin/mysql_plugin -rwxr-xr-x root/root 400192 2012-02-17 10:26 ./usr/bin/innotop -rwxr-xr-x root/root 3385528 2012-02-17 10:27 ./usr/bin/mysqladmin -rwxr-xr-x root/root 7402 2012-02-17 10:05 ./usr/bin/mysqldumpslow -rwxr-xr-x root/root 2872104 2012-02-17 10:27 ./usr/bin/mysql_waitpid -rwxr-xr-x root/root 39016 2012-02-17 10:26 ./usr/bin/mysqlreport -rwxr-xr-x root/root 10096 2012-02-17 10:27 ./usr/bin/innochecksum -rwxr-xr-x root/root 11023 2012-02-17 10:05 ./usr/bin/mysqlbug -rwxr-xr-x root/root 3172520 2012-02-17 10:27 ./usr/bin/myisam_ftdump -rwxr-xr-x root/root 3315 2012-02-17 10:05 ./usr/bin/mysql_find_rows -rwxr-xr-x root/root 1261 2012-02-17 10:05 ./usr/bin/mysql_fix_extensions -rwxr-xr-x root/root 3376536 2012-02-17 10:27 ./usr/bin/mysqlshow lrwxrwxrwx root/root 0 2012-02-17 10:27 ./usr/bin/mysqloptimize -> mysqlcheck lrwxrwxrwx root/root 0 2012-02-17 10:27 ./usr/bin/mysqlrepair -> mysqlcheck lrwxrwxrwx root/root 0 2012-02-17 10:27 ./usr/bin/mysqlanalyze -> mysqlcheck clint@clint-MacBookPro:~/pkg/mysql-5.5/bzr$ dpkg -c mysql-client-core-5.5_5.5.20-0ubuntu3_amd64.deb |grep bin drwxr-xr-x root/root 0 2012-02-17 10:27 ./usr/bin/ -rwxr-xr-x root/root 3382520 2012-02-17 10:27 ./usr/bin/mysqlcheck -rwxr-xr-x root/root 3521696 2012-02-17 10:27 ./usr/bin/mysql And with dynamically linked binaries: clint@clint-MacBookPro:~/pkg/mysql-5.5/bzr$ dpkg -c mysql-client-5.5_5.5.21-0ubuntu1~ppa1_amd64.deb |grep bin drwxr-xr-x root/root 0 2012-03-08 18:27 ./usr/bin/ -rwxr-xr-x root/root 111560 2012-03-08 15:58 ./usr/bin/mysqlaccess -rwxr-xr-x root/root 1261 2012-03-08 15:58 ./usr/bin/mysql_fix_extensions -rwxr-xr-x root/root 29776 2012-03-08 18:27 ./usr/bin/mysqlshow -rwxr-xr-x root/root 3802808 2012-03-08 18:27 ./usr/bin/mysql_client_test -rwxr-xr-x root/root 48768 2012-03-08 18:27 ./usr/bin/mysqlslap -rwxr-xr-x root/root 2881704 2012-03-08 18:27 ./usr/bin/mysql_waitpid -rwxr-xr-x root/root 31120 2012-03-08 18:27 ./usr/bin/mysqlimport -rwxr-xr-x root/root 3315 2012-03-08 15:58 ./usr/bin/mysql_find_rows -rwxr-xr-x root/root 39016 2012-03-08 18:27 ./usr/bin/mysqlreport -rwxr-xr-x root/root 3178024 2012-03-08 18:27 ./usr/bin/myisam_ftdump -rwxr-xr-x root/root 10096 2012-03-08 18:27 ./usr/bin/innochecksum -rwxr-xr-x root/root 23824 2012-03-08 18:27 ./usr/bin/mysql_plugin -rwxr-xr-x root/root 38784 2012-03-08 18:27 ./usr/bin/mysqladmin -rwxr-xr-x root/root 7402 2012-03-08 15:58 ./usr/bin/mysqldumpslow -rwxr-xr-x root/root 105872 2012-03-08 18:27 ./usr/bin/mysqldump -rwxr-xr-x root/root 400192 2012-03-08 18:27 ./usr/bin/innotop -rwxr-xr-x root/root 11028 2012-03-08 15:58 ./usr/bin/mysqlbug lrwxrwxrwx root/root 0 2012-03-08 18:27 ./usr/bin/mysqlrepair -> mysqlcheck lrwxrwxrwx root/root 0 2012-03-08 18:27 ./usr/bin/mysqlanalyze -> mysqlcheck lrwxrwxrwx root/root 0 2012-03-08 18:27 ./usr/bin/mysqloptimize -> mysqlcheck And the packages themselves shrink as a result (mysql-client-5.1 also shipped the _embedded binaries, which is why it is still large): -rw-rw-r-- 1 clint clint 9.0M Feb 29 11:33 mysql-client-5.1_5.1.58-1ubuntu5_amd64.deb -rw-rw-r-- 1 clint clint 8.0M Feb 17 10:28 mysql-client-5.5_5.5.20-0ubuntu3_amd64.deb -rw-rw-r-- 1 clint clint 3.0M Mar 8 18:31 mysql-client-5.5_5.5.21-0ubuntu1~ppa1_amd64.deb -rw-rw-r-- 1 clint clint 98K Feb 29 11:33 mysql-client-core-5.1_5.1.58-1ubuntu5_amd64.deb -rw-rw-r-- 1 clint clint 1.9M Feb 17 10:28 mysql-client-core-5.5_5.5.20-0ubuntu3_amd64.deb -rw-rw-r-- 1 clint clint 162K Mar 8 18:31 mysql-client-core-5.5_5.5.21-0ubuntu1~ppa1_amd64.deb I haven't fixed all of the linking, as you can see.. but the main concern is the core mysql client. I'm totally willing to accept that these binaries will just have to be statically linked and end up much bigger because of private API usage. However, I share the frustration at the lack of discipline in allowing this sort of practice in the client, as it brings into question why the client can cross API bounds but things built out of tree cannot. I am not as concerned with the management aspects, since any time we have to update libmysqlclient, we have to rebuild everything anyway, so its actually no additional management burden from the Debian/Ubuntu packaging point of view. It would be helpful if you could respond with whether this is going to be fixable in MySQL or not soon, so I can know whether or not it is safe to ship or revert this patch for Ubuntu 12.04.
[17 Mar 2012 0:27]
Vladislav Vaintroub
@Clint, I can't talk about Oracle's current plans in this area, I'm not working there anymore, but I also think it would be great to see cleaned up and shared-linkable clients one day. Maybe someone else who's still there can comment on idea or effort. As for your current packages, perhaps you can still save some bits by not packaging mysql_client_test - it only contains only bunch of unit tests for the client API, nothing else.
[26 Mar 2012 21:24]
Clint Byrum
Absent any clarification on whether or not this is safe, I'm going to go ahead and remove the posted patch from the pending upload to Ubuntu 12.04. Our testing has not revealed any problems, but I do see where we are simply exporting symbols that should not be exported and then depending on them in the client, which could lead to problems of library skew during subsequent rebuilds of the mysql package.
