Bug #78900 GIS files in sql/ take unacceptably many resources to compile
Submitted: 21 Oct 2015 13:52 Modified: 2 Nov 2015 16:36
Reporter: Georgi Kodinov Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: GIS Severity:S3 (Non-critical)
Version:5.7.10 OS:Any
Assigned to: CPU Architecture:Any

[21 Oct 2015 13:52] Georgi Kodinov
Description:
The following files: 
$ ls /media/sf_git/mysql-5.7/sql/item_geofunc*.cc
/media/sf_git/mysql-5.7/sql/item_geofunc_buffer.cc  /media/sf_git/mysql-5.7/sql/item_geofunc_relchecks_bgwrap.cc  /media/sf_git/mysql-5.7/sql/item_geofunc_setops.cc/media/sf_git/mysql-5.7/sql/item_geofunc.cc         /media/sf_git/mysql-5.7/sql/item_geofunc_relchecks.cc

when compiling, cause a 4GB virtual machine running Ubuntu 14.04 to start swapping as the c compilers eat up all of the available memory.
I get the following:
[terminal1]
Scanning dependencies of target sql
[ 75%] Building CXX object sql/CMakeFiles/sql.dir/item_geofunc.cc.o
[ 75%] [ 76%] [ 76%] [ 76%] Building CXX object sql/CMakeFiles/sql.dir/item_geofunc_buffer.cc.o
Building CXX object sql/CMakeFiles/sql.dir/item_geofunc_relchecks.cc.o
Building CXX object sql/CMakeFiles/sql.dir/item_geofunc_relchecks_bgwrap.cc.o
Building CXX object sql/CMakeFiles/sql.dir/item_geofunc_setops.cc.o

[terminal2]
$ while true; do ps --sort=rss -evt pts/8; sleep 1; done
  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
12240 pts/8    S+     0:00      0   110  4341   416  0.0 /bin/sh -c cd /home/gkodinov/work/mysql-5.7/sql && /usr/bin/c++   -DHAVE_CONFIG_H -DHAVE_OPENSSL -DHAVE_REPLICATI
12228 pts/8    S+     0:00      0   110  4341   480  0.0 /bin/sh -c cd /home/gkodinov/work/mysql-5.7/sql && /usr/bin/c++   -DHAVE_CONFIG_H -DHAVE_OPENSSL -DHAVE_REPLICATI
12250 pts/8    S+     0:00      0   110  4341   488  0.0 /bin/sh -c cd /home/gkodinov/work/mysql-5.7/sql && /usr/bin/c++   -DHAVE_CONFIG_H -DHAVE_OPENSSL -DHAVE_REPLICATI
11934 pts/8    S+     0:00      0   110  5229   492  0.0 /bin/sh -c cd /home/gkodinov/work/mysql-5.7 && make -f CMakeFiles/Makefile2 sql/all SELINUX_INIT=YES WITH_BOOST=/
12242 pts/8    S+     0:00      0   110  4341   492  0.0 /bin/sh -c cd /home/gkodinov/work/mysql-5.7/sql && /usr/bin/c++   -DHAVE_CONFIG_H -DHAVE_OPENSSL -DHAVE_REPLICATI
12244 pts/8    S+     0:00      0   110  4341   528  0.0 /bin/sh -c cd /home/gkodinov/work/mysql-5.7/sql && /usr/bin/c++   -DHAVE_CONFIG_H -DHAVE_OPENSSL -DHAVE_REPLICATI
12241 pts/8    S+     0:00      0   747 12684  1488  0.0 /usr/bin/c++ -DHAVE_CONFIG_H -DHAVE_OPENSSL -DHAVE_REPLICATION -DMYSQL_SERVER -D_FILE_OFFSET_BITS=64 -D_GNU_SOURC
12252 pts/8    S+     0:00      0   747 12684  1492  0.0 /usr/bin/c++ -DHAVE_CONFIG_H -DHAVE_OPENSSL -DHAVE_REPLICATION -DMYSQL_SERVER -D_FILE_OFFSET_BITS=64 -D_GNU_SOURC
12246 pts/8    S+     0:00      0   747 12684  1504  0.0 /usr/bin/c++ -DHAVE_CONFIG_H -DHAVE_OPENSSL -DHAVE_REPLICATION -DMYSQL_SERVER -D_FILE_OFFSET_BITS=64 -D_GNU_SOURC
12230 pts/8    S+     0:00      0   747 12684  1524  0.0 /usr/bin/c++ -DHAVE_CONFIG_H -DHAVE_OPENSSL -DHAVE_REPLICATION -DMYSQL_SERVER -D_FILE_OFFSET_BITS=64 -D_GNU_SOURC
12245 pts/8    S+     0:00      0   747 12684  1576  0.0 /usr/bin/c++ -DHAVE_CONFIG_H -DHAVE_OPENSSL -DHAVE_REPLICATION -DMYSQL_SERVER -D_FILE_OFFSET_BITS=64 -D_GNU_SOURC
12221 pts/8    S+     0:00      0   157 17326  1700  0.0 make -f sql/CMakeFiles/sql.dir/build.make sql/CMakeFiles/sql.dir/build SELINUX_INIT=YES WITH_BOOST=/media/sf_boos
11929 pts/8    S+     0:00      0   157 13398  1748  0.0 make -j8 XDG_VTNR=7 LC_PAPER=bg_BG.UTF-8 LC_ADDRESS=bg_BG.UTF-8 XDG_SESSION_ID=c1 XDG_GREETER_DATA_DIR=/var/lib/l
11935 pts/8    S+     0:00      0   157 13258  1756  0.0 make -f CMakeFiles/Makefile2 sql/all LESSOPEN=| /usr/bin/lesspipe %s GNOME_KEYRING_PID= no_proxy=localhost,127.0.
 2001 pts/8    Ss     0:00      1   955 27440  3092  0.0 bash XDG_SEAT=seat0 XDG_SESSION_ID=c1 LC_IDENTIFICATION=bg_BG.UTF-8 SESSION_MANAGER=local/vboxdesktop:@/tmp/.ICE-
12249 pts/8    R+     0:18   1409 14183 891108 668572 16.5 /usr/lib/gcc/x86_64-linux-gnu/4.8/cc1plus -quiet -I /home/gkodinov/work/mysql-5.7/include -I /media/sf_git/mysq
12251 pts/8    D+     0:14   1980 14183 889708 669672 16.5 /usr/lib/gcc/x86_64-linux-gnu/4.8/cc1plus -quiet -I /home/gkodinov/work/mysql-5.7/include -I /media/sf_git/mysq
12248 pts/8    D+     0:17   1588 14183 915352 694960 17.1 /usr/lib/gcc/x86_64-linux-gnu/4.8/cc1plus -quiet -I /home/gkodinov/work/mysql-5.7/include -I /media/sf_git/mysq
12253 pts/8    R+     0:17   1631 14183 945608 760996 18.8 /usr/lib/gcc/x86_64-linux-gnu/4.8/cc1plus -quiet -I /home/gkodinov/work/mysql-5.7/include -I /media/sf_git/mysq
12235 pts/8    R+     0:17   1488 14183 989868 767220 18.9 /usr/lib/gcc/x86_64-linux-gnu/4.8/cc1plus -quiet -I /home/gkodinov/work/mysql-5.7/include -I /media/sf_git/mysq

How to repeat:
On a 4GB ram virtual box:
1. Do a clean build
2. touch sql/item_geofunc*.cc
3. make -j8
4. In another terminal run "while true; do ps --sort=rss -evt pts/8; sleep 1; done", where pts/8 must be replaced with the terminal where the make command runs.
5. Observe how cc1plus processes consume more than the available memory !
Note that no other cc1plus process for any other file ever goes above 5% of available memory.

Suggested fix:
Split these files in such chunks that, when compiled individually, don't consume more than the average for the other C++ files in sql/
[2 Nov 2015 16:36] Paul DuBois
Noted in 5.7.10, 5.8.0 changelogs.

Some of the source files for spatial functions in the sql directory
took excessive compile time and required too much compiler memory
allocation.