MySQL Bugs: #82574: mysqld gets sig 11 starting up after attempt to downgrade from 5.7 to 5.6

Bug #82574	mysqld gets sig 11 starting up after attempt to downgrade from 5.7 to 5.6
Submitted:	13 Aug 2016 17:36	Modified:	13 Aug 2016 22:01
Reporter:	Mike Steigerwald	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server	Severity:	S3 (Non-critical)
Version:	5.6	OS:	Ubuntu (16.04)
Assigned to:		CPU Architecture:	Any
Tags:	mysqld, Signal 11, ubuntu

Description:
I upgraded Ubuntu from 14.04 to 16.04 without realizing Drupal and CiviCRM don't get along well with MySQL 5.7. :-( I'm now trying to replace 5.7 with 5.6, but it's not going well. The steps I've tried ...

1. Backed up all of my db.
2. Removed /var/lib/mysql/debian-5.7.flag
3. Installed sudo apt install mysql-client-5.6
4. Installed sudo apt install mysql-server-5.6
5. Unmasked the mysql.service
6. Removed /var/lib/mysql/ibdata1 and /var/lib/mysql/ib_log*
7. Changed ownership of /etc/mysql/my.cnf to mysql.
8. Added skip-grant-tables to /etc/mysql/mysql.conf.d/mysqld.cnf
9. Configured  apparmor to only complain about  /usr/sbin/mysqld

Those steps seem to have removed all errors I can find, but I still get the stack dump below. I'd be grateful for any advice.

Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: 16:41:57 UTC - mysqld got signal 11 ;
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: This could be because you hit a bug. It is also possible that this binary
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: or one of the libraries it was linked against is corrupt, improperly built,
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: or misconfigured. This error can also be caused by malfunctioning hardware.
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: We will try our best to scrape up some info that will hopefully help
Aug 13 11:41:57 EliteBook-6930p audit[21949]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/mysqld" name="/proc/21949/ta
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: diagnose the problem, but since we have already crashed,
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: something is definitely wrong and this may fail.
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: 
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: key_buffer_size=8388608
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: read_buffer_size=131072
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: max_used_connections=0
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: max_threads=151
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: thread_count=0
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: connection_count=0
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: It is possible that mysqld could use up to
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68239 K  bytes o
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: Hope that's ok; if not, decrease some variables in the equation.
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: 
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: Thread pointer: 0x7ffe5cd9e040
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: Attempting backtrace. You can use the following information to find out
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: where mysqld died. If you see no messages after this, something went
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: terribly wrong...
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: stack_bottom = 7ffe5cd9dae0 thread_stack 0x40000
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: /usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x55d6e60be79c]
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: /usr/sbin/mysqld(handle_fatal_signal+0x3d2)[0x55d6e5e20522]
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x113d0)[0x7fc1468c63d0]
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: /usr/sbin/mysqld(_Z18ha_resolve_by_nameP3THDPK19st_mysql_lex_stringb+0x6d)[0x55d6e5d4f
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: /usr/sbin/mysqld(_Z14open_table_defP3THDP11TABLE_SHAREj+0x2903)[0x55d6e5f2d5b3]
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: /usr/sbin/mysqld(_Z15get_table_shareP3THDP10TABLE_LISTPKcjjPij+0x1b7)[0x55d6e5e532e7]
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: /usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP18Open_table_context+0xafb)[0x55d6e5
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: /usr/sbin/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy+0x889)[0x
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: /usr/sbin/mysqld(_Z20open_and_lock_tablesP3THDP10TABLE_LISTbjP19Prelocking_strategy+0x
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: /usr/sbin/mysqld(+0x344959)[0x55d6e5d15959]
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: /usr/sbin/mysqld(_Z11plugin_initPiPPci+0x87a)[0x55d6e5eba22a]
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0xa8f)[0x55d6e5d497cf]
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fc145e81830]
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: /usr/sbin/mysqld(+0x36bcfd)[0x55d6e5d3ccfd]
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: 
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: Trying to get some variables.
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: Some pointers may be invalid and cause the dump to abort.
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: Query (0): Connection ID (thread ID): 0
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: Status: NOT_KILLED
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: 
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Aug 13 11:41:57 EliteBook-6930p mysqld[21950]: information that should help you find out what is causing the crash.

How to repeat:
 2040  sudo service mysql start
 2041  journalctl -xe

Downgrade from 5.7 to 5.6 cannot be done so simply.  System tables are not compatible.  I'd suggest examining these docs:

http://dev.mysql.com/doc/refman/5.7/en/downgrading-to-previous-series.html
http://dev.mysql.com/doc/refman/5.7/en/downgrading.html

But it's probably easier to just mysqldump your data from 5.7 and start with a fresh new 5.6 then import it again?

Shane,

Thanks for the response. I wasn't clear enough about my status. I had reached point where I was trying to do exactly what you suggest: start with a fresh new 5.6 then import it again. 

I guess I hadn't *completely* removed 5.7 on the 1st try. I got it right on the 2nd attempt! I've got 5.6 running again.

Thanks again for the response.

User error