MySQL Bugs: #18461: UDF - Loading Sharable Library Bug (dlopen)

Bug #18461	UDF - Loading Sharable Library Bug (dlopen)
Submitted:	23 Mar 2006 15:48	Modified:	12 Apr 2006 15:27
Reporter:	Frederick Aubert	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server	Severity:	S2 (Serious)
Version:	5.0.18-max	OS:	MacOS (MacOS 10.4.5)
Assigned to:	Magnus Blåudd	CPU Architecture:	Any

Description:
I haven't been able to complete the last step of deploying UDF functions of my own on the latest deployement of MySQL Server 5 Max running on the latest MacOS X 10.4 version. After completing successfully a test of my own loading dynamically one function of the set of functions required by an UDF, I suspect my problem to be a bug rather than a bad usage of sharable libraries.

Thanks for considering my bug report, and in any case thanks for the work MySQL community do to constantly improve a very nice product,

Frederick Aubert

How to repeat:
I have written and compiled (or tried to compile) my udfs into dylibs (equivalent to so on MacOS X systems) in a number of ways:

1) using g++ with -bundle (worked)
export MACOSX_DEPLOYMENT_TARGET="10.4"
g++ -I /usr/local/mysql/include -c  udf_tools.cc
g++ -bundle -undefined dynamic_lookup -o libudf_tools.dylib udf_tools.o
-> I placed the resulting libudf_tools.dylib in /usr/lib

2) using g++ with -dynamiclib (worked)
g++ -I /usr/local/mysql/include -c  udf_tools.cc
g++ -dynamiclib udf_tools.o -o udf_tools.dylib
-> I placed the resulting libudf_tools.dylib in /usr/lib
I even tried the following shortcut,
g++ -I /usr/local/mysql/include -dynamiclib udf_tools.cc -o udf_tools.dylib

3) using XCode, BSD Dynamic Library Project Type (also worked)
-> I placed the resulting (built) libudf_tools.dylib in /usr/lib

4) using g++ and libtool (failed)
libtool -dynamic -o udf_tools.dylib udf_tools.o
ld: for architecture ppc
ld: Undefined symbols:
__Znwm
___gxx_personality_v0
_free
_memcpy
libtool: internal link edit command failed

Here is the step having the real problem, running, in mysql,
CREATE AGGREGATE FUNCTION pgdc RETURNS INTEGER SONAME "libudf_tools.dylib"
#1126 - Can't open shared library 'libudf_tools.dylib' (errno: 9 dlopen(libudf_tools.dylib, 2): no suitable image found.  Did find: /us) 

I also tried to ensure that it was not a problem with my system, specifically with the dlopen or dlsym, so I wrote the following testing c command line program,

#include <stdio.h>
#include <dlfcn.h>

int main() {
	void	*dl, *sym;
	
	dl = dlopen( "udf_tools.dylib", 2 );

	if( dl == NULL ) { // Error
		printf( "dlopen error: %s\n", dlerror() );
	}
	else {
		sym = dlsym( dl, "pgdc_add" );
	
		if( sym == NULL ) { // Error
			printf( "dlsym error: %s\n", dlerror() );
		}
		else {
			printf( "dlsym: loaded pgdc_add\n" );
		}
	}

	return 0;
}

The output, significantly enough, was:
dlsym: loaded pgdc_add

There is a typo in the c testing program, one should read:
dl = dlopen( "libudf_tools.dylib", 2 );
rather than
dl = dlopen( "udf_tools.dylib", 2 );

In short there is no problem with the name of the dynamic library

In latest 5.0 we have added a test case in the mysql-test directory called udf.test. It will try to load the "udf_example" module that is created in the sql/ directory.

Could you please try to get that test case going?

I will give it a try on our side.,

Tested to run the udf.test with latest MySQL version 5.0.21 and it managed to load the udf_example.so sucessfully.

Please have a look at "udf_example" in sql/ directory and the udf.test in mysql-test directory for examples.

Thanks for having a look at my problem...

As per your suggestion, I tried to build the udf_example.dylib from udf_example.cc using the compilation method #2 (#1 failed due to an undefined _my_charset_latin1 symbol). I moved the compiled dynamic lib to /usr/lib. I restarted the server (not sure whether it is required but it cannot hurt). When I tried to tell mysql that I have a new function, CREATE FUNCTION metaphon RETURNS STRING SONAME "libudf_example.dylib", I met the same "#1126: no suitable image found" error. 

I would guess it has not do with the way udf_example.cc or my own udf_tools.cc are written since I would expect a #1127 error if any of the required functions were to be missing or not following udf standards. In my own opinion it has rather to do with effective loading of dynamic lib in mysql  max server. Maybe it is worth mentioning I used mysql version for MacOS 10.4 64 bits since I am running it on a G5 hardware. 

I'll stay up for further testing and information,

Frederick Aubert

PS: One side note, 5.0.21 is not yet accessible to me, do you think it might operate differently from the version I am currently running?

Please grab the latest version of MySQL 5.0 from 
http://downloads.mysql.com/snapshots.php
Compile it "--with-debug" and have a look in sql_udf.cc, this is where the dlopen call is

sql_udf.cc:442 >>
    DBUG_PRINT("info", ("Calling dlopen, udf->dl: %s", udf->dl));
    if (!(dl = dlopen(udf->dl, RTLD_NOW)))
    {
      DBUG_PRINT("error",("dlopen of %s failed, error: %d (%s)",
			  udf->dl,errno,dlerror()));
      my_error(ER_CANT_OPEN_LIBRARY, MYF(0),
                      udf->dl, errno, dlerror());
                                              ^^^Here the error message from dlopen is retrieved
                                              unfortunately it  get truncated when the MySQL error message 
                                              is formatted.
      goto err;
    }
<<

$> cd mysql-test
edit t/disabled.def and remove the line the says "udf ...."
 to enable the udf.test
$>./mysql-test-run.pl --do-test=udf --debug
This will run the udf.test and make a trace file in var/log/master.trace which will have information from the DBUG_PRINT calls in the code. That will help us to analyze the problem.

Hope this works out for you.

I certainly wrong although I am not getting it, I grabbed the latest development tree with bitkeeper, set my compiler flags, ran configure with custom values for prefix/tcp-port/unix-socket options and with debug option. As soon as I am running 'make', I am hit by a yacc error when generating the mysql parser:

sql_yacc.yy:xxxx: type clash (`' `num') on default action
make[2]: *** [sql_yacc.cc] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

Please excuse me if my errors arre obvious mistakes of mine. I am not yet used to compile mysql mysql. I will look that compiler problem on the web and see if I get an answer unless you have some nice suggestion in mind. I will then proceed with the test suite...

Here are the info on the system that you requested:

*** hardware ***
Machine: PowerMacintosh G5 2x2Ghz
Uname -a: Darwin *** 8.5.0 Darwin Kernel Version 8.5.0: Sun Jan 22 10:38:46 PST 2006; root:xnu-792.6.61.obj~1/RELEASE_PPC Power Macintosh powerpc

*** software ***
GCC --version: powerpc-apple-darwin8-gcc-4.0.1 (GCC) 4.0.1 (Apple Computer, Inc. build 5247)
Make --version: GNU Make 3.80
Bison --version: GNU Bison version 1.28

Hope that helps ;)

Thank you for your bug report. This issue has been committed to our
source repository of that product and will be incorporated into the
next release.

If necessary, you can access the source repository and build the latest
available version, including the bugfix, yourself. More information 
about accessing the source trees is available at
    http://www.mysql.com/doc/en/Installing_source_tree.html

Additional info:

Tested udf's sucessfully on the following machine.

magnus@xserve-a:~/magnus/mysql-5.0.21-standard/mysql-test> uname -a
Darwin xserve-a.mysql.com 8.2.1 Darwin Kernel Version 8.2.1: Fri Jun 24 23:31:10 PDT 2005; root:xnu-792.3.2.obj~1/RELEASE_PPC Power Macintosh powerpc

magnus@xserve-a:~/magnus/mysql-5.0.21-standard/mysql-test> gcc -v
Using built-in specs.
Target: powerpc-apple-darwin8
Configured with: /private/var/tmp/gcc/gcc-5026.obj~19/src/configure --disable-checking --prefix=/usr --mandir=/share/man --enable-languages=c,objc,c++,obj-c++ --program-transform-name=/^[cg][^+.-]*$/s/$/-4.0/ --with-gxx-include-dir=/include/gcc/darwin/4.0/c++ --build=powerpc-apple-darwin8 --host=powerpc-apple-darwin8 --target=powerpc-apple-darwin8
Thread model: posix
gcc version 4.0.0 (Apple Computer, Inc. build 5026)

mysqldev@xserve-a:~/magnus/mysql-5.0.21-standard/mysql-test> ./mysql-test-run.pl --do-test=udf
Skipping ndbcluster, mysqld not compiled with ndbcluster
Skipping SSL, mysqld not compiled with SSL
Using MTR_BUILD_THREAD = 0
Using MASTER_MYPORT    = 9306
Using MASTER_MYPORT1   = 9307
Using SLAVE_MYPORT     = 9308
Using SLAVE_MYPORT1    = 9309
Using SLAVE_MYPORT2    = 9310
Using NDBCLUSTER_PORT  = 9350
Using IM_PORT          = 9310
Using IM_MYSQLD1_PORT  = 9312
Using IM_MYSQLD2_PORT  = 9314
Killing Possible Leftover Processes
Removing Stale Files
Installing Master Databases
Installing Master Databases
Installing Slave Databases
Installing Slave Databases
Installing Slave Databases
Creating IM password file (/users/mysqldev/magnus/mysql-5.0.21-standard/mysql-test/var/im.passwd)
Installing Im_mysqld_1 Databases
Installing Im_mysqld_2 Databases
=======================================================
Finding  Tests in the 'main' suite
Starting Tests in the 'main' suite

TEST                           RESULT
-------------------------------------------------------

udf                            [ pass ]
-------------------------------------------------------
Ending Tests
Shutting-down MySQL daemon

Master(s) shutdow
n finished
Slave(s) shutdown finished
All 1 tests were successful.

The udf.test will try to load the "udf_example.so" libray built from the sql/udf_example.cc by the  Makefile in that same directory.

So have a look at that Makefile.am and you will see the following lines that tells how to build a loadable library. Notice the --module switch.
sql/Makefile.am >>
# For testing of udf_example.so
noinst_LTLIBRARIES= udf_example.la
udf_example_la_SOURCES= udf_example.cc
udf_example_la_LDFLAGS= -module -rpath $(pkglibdir)
<<

You will need to edit the mysql-test/t/disabled.def file and remove the line that says "udf" otherwise mysql-test-run.pl will skip the test. (This is because there has been some problems on other platforms.)

MySQL version 5.0.20 is now our download sites so pick one from there if you find it difficult to build it. http://dev.mysql.com/downloads/mysql/5.0.html

There are also source downloads and they should contain a prebuilt sql_yacc.cc so you don't have to mess around with bison.'

Best luck.