Bug #118654 | Assertion failure: page0cur.cc:1206 thread 139804775790336 | ||
---|---|---|---|
Submitted: | 14 Jul 7:22 | Modified: | 5 Aug 14:20 |
Reporter: | VIL North HUB VIL | Email Updates: | |
Status: | Can't repeat | Impact on me: | |
Category: | MySQL Server | Severity: | S2 (Serious) |
Version: | 8.0.30 | OS: | Red Hat (7.9) |
Assigned to: | MySQL Verification Team | CPU Architecture: | x86 (48 core) |
[14 Jul 7:22]
VIL North HUB VIL
[14 Jul 17:38]
MySQL Verification Team
Hi, Proper way to upgrade was to upgrade your 5.7.30 first to 5.7.44 and then from 5.7.44 to 8.0.42 Deprecated configuration parameters are not the reason for crash. I just tried rather large database upgrade directly from 5.7.30 to 8.0.42 and it worked ok even while not being proper way to do it. This is a crash because you had corrupted datadir. You wrote that you made backup before trying to upgrade. Did you mysqldump or you just copied datadir? Get that system up with 5.7.30 and old datadir if this is how you created backup and try to mysqldump so we can check if the original datadir is ok.
[14 Jul 18:22]
VIL North HUB VIL
Before upgrade we used util.checkForServerUpgrade() to check possible errors. We don't get any error using it. So we upgraded from 5.7.30 to 8.0.30 directly firstly on TND(test& development). After that we upgraded on production and it was working fine since a week. On crash day it was planned upgrade activity of application side they updated some service agents from their side that time MYSQL database get crashed. As we have daily backup policy and before activity we also took backup using MYSQLDUMP. in this scenario slave was fine even we confirmed it with mysqlcheck. Then we promoted slave as master and application agent upgrade acivity was done successful. From that slave (which prompted as master) we took backup and restored it on crashed MYSQL server with new datadir and now it working fine. is it bug or if datadir was corrupted what was the reason ? because its working fine since upgrade to 8.0.30. is appellation agent upgrade cause crash/corruption or else something?
[15 Jul 4:10]
VIL North HUB VIL
Here you can see direct upgrade from 5.7.24 to 8.0.12 on MYSQL official site: https://dev.mysql.com/blog-archive/inplace-upgrade-from-mysql-5-7-to-mysql-8-0/
[15 Jul 8:27]
MySQL Verification Team
Hi, > Here you can... Yes, in the time of that article those were the latest versions. When upgrading you should always upgrade to latest minor version before upgrading major version. 5.7.30 to 8.0.30 should work but I'm talking about proper procedure. You should upgrade to latest 5.7 first and then move to latest 8.0. I do not think this is the reason for your issue, it is just a proper procedure you should always follow. What I fail to understand is since you are upgrading to 8.0 why did you decide to go with 3 year old release? Why not upgrade to latest 8.0?! > Before upgrade we used util.checkForServerUpgrade() t It will check for config and datatype compatibility. It will not check if your datadir is ok or not. Also depending on the version of the shell it can not know about what latest versions are. > we also took backup using MYSQLDUMP if that dump was successful it is most likely that your datadir corruption happened between that dump and that crash. Due to how tested InnoDB is we can in 99% cases state that the corruption is caused by bad hardware. > is it bug or if datadir was corrupted what was the reason ? As I mentioned in our experience in 99% of these cases it is hw issue (from simple bitflip when not using ECC RAM to a problem with storage controller or storage device itself). You do run a 3 year old server with a lot of bugs fixed but none are as far as I know related to a problem you experienced. > is appellation agent upgrade cause crash/corruption or else something? I doubt any external software could of caused this There is still a possibility that this can be a bug but I need to be able to reproduce the problem. It does not seem to be data / query related as your slave (new master) is working ok. Please monitor the system and update the report if you notice anything problematic.
[15 Jul 9:29]
VIL North HUB VIL
After upgrade we checked all databases with mysqlcheck and for all tables it was OK. Error logs also okay that time. It worked for 1 week. Cloud era meta save in SCM database. They upgraded CM (cloudera mananger )from 7.6.7 to 7.11.3 and while that upgrade MYSQL database get crashed. As I mentioned earlier, its cloud application meta data, and version is suggested by that cloudera team. Mysql upgrade & CM service upgrade also tested on TND environment after that only we proceed for upgrade,
[15 Jul 19:48]
MySQL Verification Team
Hi, I doubt cloudera touches mysql datadir, it connects to mysql as a client so that should not make it crash. I have no clue why would they suggest release that old but you should use latest one as even if there is a bug in 8.0.30 there's nothing we can do about it but tell you to upgrade. We need a way to reproduce this in latest release.
[16 Jul 9:52]
VIL North HUB VIL
okay.
[4 Aug 18:03]
VIL North HUB VIL
But Why did this issue not occur in the Test environments?