| Bug #74310 | Crash in authentication_pam | ||
|---|---|---|---|
| Submitted: | 10 Oct 2014 9:33 | Modified: | 25 Jan 2016 15:36 |
| Reporter: | Daniël van Eeden (OCA) | Email Updates: | |
| Status: | Won't fix | Impact on me: | |
| Category: | MySQL Server: Pluggable Authentication | Severity: | S2 (Serious) |
| Version: | 5.6.14, 5.6.21 | OS: | Any |
| Assigned to: | Georgi Kodinov | CPU Architecture: | Any |
| Tags: | authentiction, crash, PAM | ||
[10 Oct 2014 9:39]
Daniël van Eeden
entering auth_pam_server entering auth_pam_next_token auth_pam_next_token:reading at [], sep=[,] auth_pam_next_token:state=PRESPACE, ptr=[], out=[] auth_pam_next_token:state=IDENT, ptr=[], out=[] auth_pam_next_token:state=AFTERSPACE, ptr=[], out=[] auth_pam_next_token:state=DELIMITER, ptr=[], out=[] auth_pam_next_token:state=DONE, ptr=[], out=[] leaving auth_pam_next_token on /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/plugin/pam-authentication-plugin/src/parser.c:195 auth_pam_server:password XXXXXXXXXX received auth_pam_server:pam_start rc=0 auth_pam_server:pam_set_item(PAM_RUSER,myuser) rc=0 auth_pam_server:pam_set_item(PAM_RHOST,localhost) rc=0 entering auth_pam_server_conv auth_pam_server_conv:PAM_PROMPT_ECHO_OFF [Password: ] received leaving auth_pam_server_conv on /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/plugin/pam-authentication-plugin/src/authenticatio n_pam.c:269 Segmentation fault (core dumped) 141010 11:01:16 mysqld_safe Number of processes running now: 0 141010 11:01:16 mysqld_safe mysqld restarted
[10 Oct 2014 10:06]
Daniël van Eeden
A bit more debug info after installing libpam0g-dbgsym
(gdb) bt
#0 0x00007f6268152b38 in pam_sm_authenticate () from /lib/security/pam_ecryptfs.so
#1 0x00007f6270081dff in _pam_dispatch_aux (use_cached_chain=<optimized out>, resumed=<optimized out>, h=<optimized out>, flags=32768,
pamh=0x7f6260011b70) at pam_dispatch.c:110
#2 _pam_dispatch (pamh=pamh@entry=0x7f6260011b70, flags=flags@entry=32768, choice=choice@entry=1) at pam_dispatch.c:395
#3 0x00007f627008164d in pam_authenticate (pamh=0x7f6260011b70, flags=32768) at pam_auth.c:34
[17 Nov 2014 8:29]
Georgi Kodinov
Hello Daniël, This looks like a PAM back-end module crashing. It may IMHO be because of some missing calls (or wrong call sequence) on behalf of the client (MySQL's PAM authentication plugin) or because of PAM under-configuration. Since I don't have your PAM setup I need to ask you to do some diagnosis. Can you please try to make any other PAM client program authenticate against the PAM configuration that you're using for MySQL ? In case that works can you also attach a sequence of the PAM calls to the bug ? Ideally I'd like to compare it with the sequence produced by MySQL's PAM plugin. I'd appreciate if you can include some data on the PAM function parameters too (obfuscated will do as well, as long as I can compare to the data passed by the MySQL's PAM plugin)
[23 Nov 2014 16:52]
Daniël van Eeden
Retested with Ubuntu 14.10 with MySQL Enterprise 5.6.21
$ dpkg -l ecryptfs-utils libpam-modules
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=============================-===================-===================-================================================================
ii ecryptfs-utils 104-0ubuntu1 amd64 ecryptfs cryptographic filesystem (utilities)
ii libpam-modules:amd64 1.1.8-3ubuntu4 amd64 Pluggable Authentication Modules for PAM
The PAM config:
dveeden@daniel-thinkpad:~$ egrep -v '^($|#)' /etc/pam.d/other
@include common-auth
@include common-account
@include common-password
@include common-session
dveeden@daniel-thinkpad:~$ egrep -hv '^($|#)' /etc/pam.d/common-{auth,account,password,session}
auth [success=1 default=ignore] pam_unix.so nullok_secure
auth requisite pam_deny.so
auth required pam_permit.so
auth optional pam_ecryptfs.so unwrap
auth optional pam_cap.so
account [success=1 new_authtok_reqd=done default=ignore] pam_unix.so
account requisite pam_deny.so
account required pam_permit.so
password [success=1 default=ignore] pam_unix.so obscure sha512
password requisite pam_deny.so
password required pam_permit.so
password optional pam_gnome_keyring.so
password optional pam_ecryptfs.so
session [default=1] pam_permit.so
session requisite pam_deny.so
session required pam_permit.so
session optional pam_umask.so
session required pam_unix.so
session optional pam_systemd.so
session optional pam_ecryptfs.so unwrap
session optional pam_ck_connector.so nox11
I copied this to /etc/pam.d/mysql:
egrep -hv '^($|#)' /etc/pam.d/common-{auth,account,password,session} | sudo tee /etc/pam.d/mysql
I had to do a chmod 644 on /etc/shadow as the user who runs the sandbox is not the user who is being authenticated. This step is not needed if they're the same.
Now it fails:
$ ./my sql -e "uninstall plugin authentication_pam;drop user 'myuser';"; ./restart; ./my sql -e "install plugin authentication_pam soname 'authentication_pam.so';create user 'myuser' identified with authentication_pam as 'mysql';"; ./my sql -u myuser --enable-cleartext-plugin -ptest
. sandbox server started
Warning: Using a password on the command line interface can be insecure.
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading final connect information', system error: 95
It doesn't crash when I uncomment this line:
"auth optional pam_ecryptfs.so unwrap"
I tested it with a user why uses eCryptfs (adduser --encrypt-home myuser) and a user who doesn't use an ecrypted homedir.
$ file /lib/security/pam_ecryptfs.so /lib/x86_64-linux-gnu/security/pam_unix.so
/lib/security/pam_ecryptfs.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=de113e2b000e33c8c98a7a0061c207e431d4ba18, stripped
/lib/x86_64-linux-gnu/security/pam_unix.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=46841a7e61dba3a1665f8ff8bcb943d33391f8cc, stripped
[23 Nov 2014 19:42]
Daniël van Eeden
I installed some more debug libs (had to enable the ddebs repo for that):
dpkg -l libpam-modules libpam-modules-dbgsym libpam0g-dbgsym libpam0g ecryptfs-utils ecryptfs-utils-dbg
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=============================-===================-===================-================================================================
ii ecryptfs-utils 104-0ubuntu1 amd64 ecryptfs cryptographic filesystem (utilities)
ii ecryptfs-utils-dbg 104-0ubuntu1 amd64 ecryptfs cryptographic filesystem (utilities; debug)
ii libpam-modules:amd64 1.1.8-3ubuntu4 amd64 Pluggable Authentication Modules for PAM
ii libpam-modules-dbgsym:amd64 1.1.8-3ubuntu4 amd64 debug symbols for package libpam-modules
ii libpam0g:amd64 1.1.8-3ubuntu4 amd64 Pluggable Authentication Modules library
ii libpam0g-dbgsym:amd64 1.1.8-3ubuntu4 amd64 debug symbols for package libpam0g
A more complete backtrace:
#0 pam_sm_authenticate (pamh=0x7f8e10007f70, flags=<optimized out>, argc=1, argv=0x7f8e1000c2f0) at pam_ecryptfs.c:114
#1 0x00007f8e33fcddcf in _pam_dispatch_aux (use_cached_chain=<optimized out>, resumed=<optimized out>, h=<optimized out>, flags=32768,
pamh=0x7f8e10007f70) at pam_dispatch.c:110
#2 _pam_dispatch (pamh=pamh@entry=0x7f8e10007f70, flags=flags@entry=32768, choice=choice@entry=1) at pam_dispatch.c:395
#3 0x00007f8e33fcd65d in pam_authenticate (pamh=0x7f8e10007f70, flags=32768) at pam_auth.c:34
#4 0x00007f8e341dc41d in auth_pam_server (vio=<optimized out>, info=0x7f8e25e653b8)
at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/plugin/pam-authentication-plugin/src/authentication_pam.c:360
#5 0x00000000006b55f4 in do_auth_once (thd=0x30879f0, auth_plugin_name=0x7f8e25e65870, mpvio=0x7f8e25e653a0)
at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/sql/sql_acl.cc:11109
#6 0x00000000006c1965 in acl_authenticate (thd=0x30879f0, com_change_user_pkt_len=0)
at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/sql/sql_acl.cc:11267
#7 0x00000000006f2ed9 in check_connection (thd=<optimized out>)
at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/sql/sql_connect.cc:677
#8 login_connection (thd=0x30879f0) at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/sql/sql_connect.cc:746
#9 0x00000000006f3104 in thd_prepare_connection (thd=0x30879f0)
at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/sql/sql_connect.cc:906
#10 0x00000000006f3993 in do_handle_one_connection (thd_arg=0x30879f0)
at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/sql/sql_connect.cc:975
#11 0x00000000006f3a95 in handle_one_connection (arg=0x30879f0)
at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/sql/sql_connect.cc:898
#12 0x00000000009b1cc6 in pfs_spawn_thread (arg=<optimized out>)
at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/storage/perfschema/pfs.cc:1860
#13 0x00007f8e3aed80a5 in start_thread (arg=0x7f8e25e66700) at pthread_create.c:309
#14 0x00007f8e39b9284d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
On pam_ecryptfs.c:114 there is this line:
rc = pam_get_user(pamh, &username, NULL);
[19 Nov 2015 17:35]
Georgi Kodinov
Thanks for the debugging. Can you also please verify that other PAM clients would work against your PAM path ?
[19 Nov 2015 19:37]
Daniël van Eeden
>Can you also please verify that other PAM clients would work against your PAM path ? Yes, I did do that and they were working fine.
[30 Nov 2015 16:21]
Georgi Kodinov
Daniel, Great ! That's some good news. Can you please consider sharing the PAM library call sequence for the successful attempt (without the sensitive data of course) ? Maybe it's just that the mysql plugin is not calling PAM in the right order for this particular back-end (or in general) somehow ?
[1 Dec 2015 15:05]
Daniël van Eeden
> Can you please consider sharing the PAM library call sequence for the > successful attempt (without the sensitive data of course) ? The machine was upgraded and replaced since I hit this bug, so it is not possible for me to try and reproduce it now.
[17 Dec 2015 9:39]
Sergei Glushchenko
We have encountered very similar crash with Percona Server (link: https://bugs.launchpad.net/percona-server/+bug/1521481) It turned out to be stack overflow. Posting analysis here, maybe it will help. As we can see from the stack trace mysqld crashes in pam_ecryptfs.so: #0 pam_sm_authenticate (pamh=0x7f5be4017cd0, flags=<optimized out>, argc=1, argv=0x7f5be4022e30) at pam_ecryptfs.c:142 Lets take a look at pam_sm_authenticate: PAM_EXTERN int pam_sm_authenticate(pam_handle_t *pamh, int flags, int argc, const char **argv) { uid_t uid = 0, oeuid = 0; long ngroups_max = sysconf(_SC_NGROUPS_MAX); gid_t gid = 0, oegid = 0, groups[ngroups_max+1]; int ngids = 0; char *homedir = NULL; const char *username; char *passphrase = NULL; char salt[ECRYPTFS_SALT_SIZE]; char salt_hex[ECRYPTFS_SALT_SIZE_HEX]; char *auth_tok_sig = NULL; char *private_mnt = NULL; pid_t child_pid, tmp_pid; long rc; rc = pam_get_user(pamh, &username, NULL); ... especially lines long ngroups_max = sysconf(_SC_NGROUPS_MAX); gid_t gid = 0, oegid = 0, groups[ngroups_max+1]; sysconf(_SC_NGROUPS_MAX) is 65536, sizeof(gid_t) is 4. So, 262144 bytes are allocated on stack. Now mysql> select @@thread_stack; +----------------+ | @@thread_stack | +----------------+ | 262144 | +----------------+ 1 row in set (0.01 sec) Default mysql stack size in not enough to use by pam_ecryptfs. Workaround is to increase MySQL stack size by setting --thread-stack variable.
[13 Jan 2016 6:47]
Roel Van de Paar
Based on input here + on https://bugs.launchpad.net/percona-server/+bug/1521481 can this please be made verified?
[25 Jan 2016 15:36]
Georgi Kodinov
Given Daniel's feedback I'm just going to accept Sergei's explanation (thank you !) and close the bug for now. Daniel, should it start crashing again (and increasing the stack size won't help) please re-open the bug. Roel, I'd really like to do one bug at a time. And the URL you provided looks like a different problem. So please open a new bug for it.
Description: On Ubuntu 14.04.1 LTS with eCryptfs for encrypted homedirs the PAM authentication plugin crashes. Tested with 5.6.14 and 5.6.21 (enterprise generic tar version) $ mysql -u myuser --enable-cleartext-plugin -p Enter password: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading final connect information', system error: 95 Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f6268152b38 in pam_sm_authenticate () from /lib/security/pam_ecryptfs.so (gdb) bt #0 0x00007f6268152b38 in pam_sm_authenticate () from /lib/security/pam_ecryptfs.so #1 0x00007f6270081dff in ?? () from /lib/x86_64-linux-gnu/libpam.so.0 #2 0x00007f627008164d in pam_authenticate () from /lib/x86_64-linux-gnu/libpam.so.0 #3 0x00007f6271eb741d in auth_pam_server (vio=<optimized out>, info=0x7f62720f93b8) at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/plugin/pam-authentication-plugin/src/authentication_pam.c:360 #4 0x00000000006b55f4 in do_auth_once (thd=0x29c90c0, auth_plugin_name=0x7f62720f9870, mpvio=0x7f62720f93a0) at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/sql/sql_acl.cc:11109 #5 0x00000000006c1965 in acl_authenticate (thd=0x29c90c0, com_change_user_pkt_len=0) at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/sql/sql_acl.cc:11267 #6 0x00000000006f2ed9 in check_connection (thd=<optimized out>) at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/sql/sql_connect.cc:677 #7 login_connection (thd=0x29c90c0) at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/sql/sql_connect.cc:746 #8 0x00000000006f3104 in thd_prepare_connection (thd=0x29c90c0) at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/sql/sql_connect.cc:906 #9 0x00000000006f3993 in do_handle_one_connection (thd_arg=0x29c90c0) at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/sql/sql_connect.cc:975 #10 0x00000000006f3a95 in handle_one_connection (arg=0x29c90c0) at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/sql/sql_connect.cc:898 #11 0x00000000009b1cc6 in pfs_spawn_thread (arg=<optimized out>) at /export/home/pb2/build/sb_0-13191046-1410443906.25/mysqlcom-pro-5.6.21/storage/perfschema/pfs.cc:1860 #12 0x00007f6286b28182 in start_thread (arg=0x7f62720fa700) at pthread_create.c:312 #13 0x00007f62857edfbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 How to repeat: Use mysql sandbox with 5.6.21. export AUTHENTICATION_PAM_LOG=1 ulimit -c unlimited add core-file to the config (my.sandbox.cnf) start the server install plugin authentication_pam soname 'authentication_pam.so'; create user 'myuser' identified with authentication_pam; the try to login with this uses with correct credentials. expected result: authenticated session actual result: crash This only happens when the plugin is loaded after startup.