Bug #55647 CMake build rebuilds everything twice for embedded
Submitted: 30 Jul 2010 12:09 Modified: 20 Nov 2010 17:32
Reporter: Konstantin Osipov (OCA) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Compiling Severity:S3 (Non-critical)
Version: OS:Any
Assigned to: Vladislav Vaintroub CPU Architecture:Any

[30 Jul 2010 12:09] Konstantin Osipov
Description:
As discussed with Wlad, CMake rebuilds every object file twice with and without position independent code, if the build includes the embedded library.
According to him, this is done to have a fast mysqld binary, built without PIC, and yet reliable embedded server library, that can be linked into a shared library, and thus built with PIC.

This is a regression to autotools build, and significantly slows down CMake build. In particular I'm not looking forward to double build of NDB
when it's added to CMake builds.

How to repeat:
Build with CMake. Observe:
[ 28%] Building CXX object storage/federated/CMakeFiles/federated.dir/ha_federated.cc.o
[ 28%] Building CXX object storage/federated/CMakeFiles/federated_embedded.dir/ha_federated.cc.o

Suggested fix:
Do not keep an option that effectively doubles build time ON by default.
Provide a way to build the embedded server with --with-pic, as suggested in Bug#39228, only to those who need it (release builds), and spare the developers from wasting their time staring at overheating laptops.
[30 Jul 2010 12:45] Konstantin Osipov
Sorry, I gave an incorrect bug id. The correct link is Bug#39288.
[30 Jul 2010 14:32] MySQL Verification Team
Thank you for the bug report.
[31 Jul 2010 17:40] Vladislav Vaintroub
Kostja,

First, it is not everything that gets rebuilt , and build times are not doubling.

Second, embedded is not on by default and can easily be switched on and off without recompiling everything

cmake . -DWITH_EMBEDDED_SERVER=1 
switches on embedded (e.g after the normal build)
cmake . -DWITH_EMBEDDED_SERVER=0
switches it  off again

You do not have to rebuild the full thing from scratch, there is seldom a necessity to do so (e.g normally you can just bzr pull && make in your build directory)

But, back to the statement that build times are doubling. This bug report mantions performance, but does not mention numbers, so I'll provide some benchmarking here.

If you want to measure the overhead of recompiling for embedded (excluding sql/ stuff), you can do e.g like this:

0) branch the repository
1) mkdir bld && cd bld
2) cmake .. -DWITH_ALL=1 -DWITH_DEBUG=1 # no embedded, all engines statically
3) make -j4                          # build everything w/o embedded
4) cmake .. -DWITH_EMBEDDED_SERVER=1 # enable embedded
5) make -j4 sql_embedded             # sql/ stuff gets recompiled
6) make -j4 mysqlserver                  # libmysqld.a is built
7) make                              # some embedded test utilities are made

What you are interested in, is the step 6) and how it compares performance-wise to to the rest of the build. So here is my report, made on my Linux VM (I gave it 2 cores, 1GB RAM, should be  no better than any laptop around)

step 2 . initial configuration :1:01.85
step 3.  make all              :3:41.38
step 4.  set embedded option   :0:04.92
step 5.  compile embedded sql/ :1:39.61
step 6.  rest of libmysqld.a   :1:00.01
step 7.  make embedded test prg:0:12.61

We're interested in step 6 and how it compares to the rest. The whole build takes ca 460 seconds (there is some overhead in multiple execution of "make" and checking dependencies), of which step 6 (the step you think should be called "recompile everything") takes 60 seconds, or about 13%.

Now, lets check if -DWITH_PIC=1 option can significantly reduce the build times (which , given 13% of the overhead does not sound likely). Doing the same procedure as described above with the difference that -DWITH_PIC =1 is added to cmake options on the step 2.

step 2. initial configuration :1:04.95
step 3. make all              :3:44.46
step 4. set embedded option   :0:04.79
step 5. compile embedded sql/ :1:42.27
step 6. rest of libmysqld.a   :0:27.38
step 7. make embedded test prg:0:13.12

The full build takes  now 435 seconds instead of 460, which in fact mostly comes from the step6. An improvement, but hardly a significant one (460/435 ≃ 1.05747), it the full improvement is about 5.7%.

For non-debug build, it looks like this :
NO PIC: 

2. initial configuration :1:01.44
3. make all              :5:09.52
4. set embedded option   :0:04.29
5. compile embedded sql/ :2:15.87
6. rest of libmysqld     :1:23.41
7. make embedded test prg:0:17.13

PIC:
2. initial configuration :1:04.41
3. make all              :5:07.73
4. set embedded option   :0:04.32
5. compile embedded sql/ :2:16.31
6. rest of libmysqld     :0:32.12
7. make embedded test prg:0:17.22

The different between the build is 611sec(non-PIC) vs 560(PIC), or 9%, a bit more that in Debug build, but if you're looking for build performance, you would do Debug anyway.

Note also, that on systems that are do not have PIC (Windows and OSX) there is not PIC recompilation. 

It is nowhere near doubling the build time,and given the build time less than 8 minutes on second-class VM, I think the overall speed is pretty impressive. Ti would be more impressive if stuff in sql/* would compile faster (it is the most performance critical component of the build). 

I invite you to measure on your own machine and provide the results here, lets see if they disagree.

Since autotools build was mentioned as something cmake can learn from performance-wise, I think it is also prudent to provide compile numbers for autotools here. On the same machine, debug build takes 12minutes 7 seconds, or 36-44% slower than cmake build.
[4 Aug 2010 0:05] Davi Arnaut
cmake: 2m8.461s autotools: 2m24.882s, both from ccache.

> Since autotools build was mentioned as something cmake can learn from performance-wise.

None said so in this bug report. Alas, the problem at hand is the rebuild. The cmake build could be much faster...
[4 Aug 2010 0:09] Davi Arnaut
Its a bit silly to even compare the performance because they would be about the same if the same things are built. Bottom line, be it with autotools or cmake, it shouldn't be rebuilding unless strictly necessary.
[4 Aug 2010 0:20] Vladislav Vaintroub
IT is not "a bit silly" to measure times, since the report mentions "effectively doubling" times, without any measuring.
[4 Aug 2010 0:24] Vladislav Vaintroub
The report also says "significantly slows down", and there should be a measurement for what  "significatly" means.
[4 Aug 2010 0:46] Davi Arnaut
> "Do not keep an option that effectively doubles build time ON by default."
> ".. and significantly slows down CMake build"

cmake without embedded rebuild: ~ 1m58.295s. Does not doubles, but It's significant.
[4 Aug 2010 0:57] Davi Arnaut
Hand waving a part and repeating again to make the point across, there is no point in rebuilding unless strictly necessary.
[4 Aug 2010 1:31] Vladislav Vaintroub
>cmake without embedded rebuild: ~ 1m58.295s. Does not doubles, but It's significant.

I do not know what you have expected. I have expected more, 10 seconds sounds minimal , or 7.8% of your build. 

Remember the biggest part of embedded build is compiling sql_embedded (sql/* stuff). It is present in autotools, and in cmake as well, and noone said it is unnecessary.  Same goes for mysqltest_embedded... 

The real cmake overhead when built without pic ("make mysqlserver" after "make sql_embedded" in my first comment) could be something like 4 seconds on your machine.
[4 Aug 2010 7:39] Konstantin Osipov
Vlad, I can confirm that rebuilding sql/ for embedded is unnecessary, and should not be done. The point I'm trying to make with this bug report is that going forward our build system needs to be making the situation better, not worse.
The times on my computer actually show that cmake build is faster:

HAVE_CMAKE=no ./BUILD/compile-pentium-valgrind-max-no-ndb  2548.48s user 324.65s system 169% cpu 28:11.35 total
./BUILD/compile-pentium-valgrind-max-no-ndb  2272.13s user 274.93s system 178% cpu 23:46.60 total

But it doesn't mean that the request is invalid, as, with CMake build:
- no symlinks are used to build libmysql and there is no duplication with libmysql_r
- dependency tracking is better, and thus innobase, say, is built in parallel with MyISAM, thus CPU utilization is higher.
- NDB is completely excluded, including ha_ndbcluster_binlog.cc and ha_ndbcluster.cc

Let's see what happens when the request is satisfied (adding --with-pic to compile flags):
./BUILD/compile-pentium-valgrind-max-no-ndb  2076.36s user 247.64s system 179% cpu 21:36.59 total
+10% improvement.
[4 Aug 2010 7:43] Konstantin Osipov
Vlad,
in this bug report I suggest to, by default, build the embedded library without PIC, rather than add --with-pic.
Plus, I suggest to not recompile MyISAM, MyISAMMRG, HEAP and others for embedded, but only their ha_* object files, like in autoconf.
10+% improvement it brings is significant enough in my view, but it's also good because it removes -DEMBEDDED_SERVER from many source files which don't need it and shouldn't rely on this define.
[4 Aug 2010 10:08] Vladislav Vaintroub
Kostja,
>I can confirm that rebuilding sql/ for embedded is unnecessary,
and should not be done

I cannot agree, after grepping for EMBEDDED_LIBRARY and NO_EMBEDDED_ACCESS_CHECKS in sql/*, and in include/*. Several hundreds of occurences, including header files. Core structures like THD are dependend on EMBEDDED_LIBRARY.
[4 Aug 2010 22:34] Vladislav Vaintroub
kostja,
I did not mean that your request is invalid. My point in measuring and explaining was to correct statements made in the original description . In particular

- embedded is not the default option, so by default nothing is recompiled
- embedded is easy to switch on and switch off, temporarily, between the builds
(builds should not be "from scratch" anyway)
- the recompilation net overhead is not 100%, it is at most 10%.
- you already have an option (WITH_PIC) to avoid much of that recompilation, though this option is OFF by default.
- On the PIC-immune systems (either always PIC, like OSX , or never PIC, like Windows) the recompilation does not happen.

If you reread the original description, the overhead sounds much more serious than  it is.

You're suggesting not to have non-PIC libmysqld.a by default, and I disagree, for 2 reasons:

1) We currently only not provide a static library for embedded. Out users/customers however want it to be useful and linkable to both executables and shared libraries (the single prominent user of libmysqld.a , Amarok player wants it to be usable in shared lib environment). 

Using single static lib in shared lib or executable can only be done on PIC-sensitive systems if every object in static library is compiled with PIC ,adn this is what I do. 

This will also avoids the trickery and the mess that is currently done by Linux distributors (lookup google  for "libmysqld PIC", there is a lot of interesting discussions and hacking around).

2) As a side effect of PIC compilation, it is now extremely easy to provide a shared library libmysqld.so when we'll be asked about it (we were asked several times already). It would not take a second to build a shared lib from PIC 
static lib.

Note that the above is acceptable since libmysqld.a does not have extreme performance requirements. It does not even have threads, performance is out of question, and PIC compilation which excludes a single general purpose register from allocation by compiler backend and thus makes code maybe .05% slower won't be a great deal here.

Summarizing the above, I conclude that how it is done currently, is done almost right, but perhaps I should have been nicer to people trying to build fast in Debug environments. Hence, my current plan is to switch PIC on by default in Debug builds (trading potential say 0.05% execution speedup in favor of 5-10% build speedup). 

I do not plan to switch this ON (by default) in Release environments, since the goal of fast build is hardly achievable in presence of optimizing compiler backend, and additional 10% build time does not make a big difference (compiler optimization does have an overhead of 40-50% or more at build time).

And, if someone wants something very unusual (faster build with embedded with Release settings), and is willing to trade the 0.05% execution speed  for 10% faster build,  he can set WITH_PIC.

What you say about recompiling full engines with -DEMBEDDED_LIBRARY instead of part of it (ha_*), is valid. I should think a bit more about this, this is not done with autotools-like ugly hack, too unportable. 

And CMake is not generally very suitable for low-level hacks like this(recompile this file  with those flags and replace the object file in the static library).

A portable solution is to split single plugin into 2 libraries with 
- embedded-dependent part which will be recompiled with -DEMBEDDED_LIBRARY  for libmysqld

- embedded-independent part (which due to possible PIC requirements, can be recompiled for libmysqld, but never uses -DEMBEDDED_LIBRARY)
[6 Oct 2010 11:41] Konstantin Osipov
I think Wlad doesn't want to fix it.
[6 Oct 2010 13:10] Vladislav Vaintroub
Kostja,
Thanks for reminding. and lets keep it open or a while. I did already have 2 patches to fixing this (a simple one and a complex one), that I will commit  shortly.
[6 Oct 2010 13:17] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/120131

3094 Vladislav Vaintroub	2010-10-06
      Bug#55647:CMake build rebuilds some libraries twice for embedded
      
      A  simple version of the patch : add WITH_PIC to debug compilation.
      
      Engines that are not dependent on internal stuff (like THD structure) 
      will not be recompiled for embedded in debug mode.
      
      Examples of such engines are : archive, federated, innodb. Engines
      with dependencies on internals (myisam,heap,myisammrg) will still be 
      fully recompiled.
[6 Oct 2010 13:41] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/120135

3094 Vladislav Vaintroub	2010-10-06
      Bug#55647:CMake build rebuilds some libraries twice for embedded
      
      A  complex  version of the patch :
      1. add WITH_PIC to debug compilation.
      
      Engines that are not dependent on internal stuff (like THD structure) 
      will not be recompiled for embedded in debug mode.
      Examples of such engines are : archive, federated, innodb.
      
      2. Engines with dependencies on internals (myisam,heap,myisammrg) 
      will still be partially recompiled. static libraries will be split into 2 parts
      one part depends on WITH_EMBEDDED_SERVER definition, another one
      does not.
      
      So for example instead of big single libmyisam.a there will be 2 libraries
      in debug compilation :
      - libmyisam.a (small one, only consisting of ha_myisam.cc which depends on WITH_EMBEDDED_SERVER)
      - libmyisam_core.a (everything else from myisam)
      If WITH_EMBEDDED is set, then there will be  libmyisam_embedded.a (consisting 
      of a single ha_myisam.cc compiled with -DWITH_EMBEDDED_SERVER)
      
      A mysqld binary will link with libmyisam.a and libmyisam_core.a
      libmysqld.a will be constructed of libmyisam_embedded.a and libmyisam_core.a
      
      The net effect is that only parts of engines are recompiled.
      
      But for release compilation, where WITH_PIC normally is not be set, there will be
      no gain, as libmyisam_core.a will still be recompiled into libmyisam_core_embedded.a
      (just because of the -fPIC flag)
[6 Oct 2010 17:57] Vladislav Vaintroub
Myself, I prefer the simple fix http://lists.mysql.com/commits/120131 to 
complex fix http://lists.mysql.com/commits/120135, because 

1. The overhead of recompiling engines marked RECOMPILE_FOR_EMBEDDED is hardly measurable up to now (well, with NDB this might change, but NDB is not yet there)

2. Optimizing compile time here is just papering over real problems (one of them is that WITH_EMBEDDED_SERVER macro is inappropriately overused, and another one is something marked as plugin is spelunking in internal structures)

3. The already complicated CMake macro MYSQL_ADD_PLUGIN would get even more complicated with the "complex" fix.

Anyway, I let you guys decide which one you wish, and I push  the one you like more.
[20 Oct 2010 19:43] Konstantin Osipov
The patch is OK with me.
[21 Oct 2010 12:53] Vladislav Vaintroub
queued to 5.5-bugfixing, trunk-merge
[13 Nov 2010 16:25] Bugs System
Pushed into mysql-trunk 5.6.99-m5 (revid:alexander.nozdrin@oracle.com-20101113155825-czmva9kg4n31anmu) (version source revid:alexander.nozdrin@oracle.com-20101113152450-2zzcm50e7i4j35v7) (merge vers: 5.6.1-m4) (pib:21)
[13 Nov 2010 16:36] Bugs System
Pushed into mysql-next-mr (revid:alexander.nozdrin@oracle.com-20101113160336-atmtmfb3mzm4pz4i) (version source revid:vasil.dimov@oracle.com-20100629074804-359l9m9gniauxr94) (pib:21)
[20 Nov 2010 17:32] Paul Dubois
Noted in 5.5.8 changelog.

For CMake builds, some parts of the source were unnecessarily
compiled twice if the embedded server was built.
[16 Dec 2010 22:31] Bugs System
Pushed into mysql-5.5 5.5.9 (revid:jonathan.perkin@oracle.com-20101216101358-fyzr1epq95a3yett) (version source revid:jonathan.perkin@oracle.com-20101216101358-fyzr1epq95a3yett) (merge vers: 5.5.9) (pib:24)