2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367399845 2018-06-24 04:58:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367401143 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367401362 2018-06-24 04:58:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367406529 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367406748 2018-06-24 04:58:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367407454 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367407673 2018-06-24 04:58:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367408971 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367409190 2018-06-24 04:58:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367411632 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367411851 2018-06-24 04:58:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367413149 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367413368 2018-06-24 04:58:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367414666 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367414885 2018-06-24 04:58:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367417499 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367417718 2018-06-24 04:58:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367421748 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367421967 2018-06-24 04:58:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367422640 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367422859 2018-06-24 04:58:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367423532 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367423751 2018-06-24 04:58:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367426365 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367426584 2018-06-24 04:58:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367427882 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367428101 2018-06-24 04:58:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367430448 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367430667 2018-06-24 04:58:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367431965 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367432184 2018-06-24 04:58:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367433482 2018-06-24 04:58:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367433701 2018-06-24 04:58:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367436592 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367436811 2018-06-24 04:58:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367438109 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367438328 2018-06-24 04:58:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367441199 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367441418 2018-06-24 04:58:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367443374 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367443593 2018-06-24 04:58:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367445946 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367446165 2018-06-24 04:58:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367447463 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367447682 2018-06-24 04:58:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367448980 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367449199 2018-06-24 04:58:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367451515 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367451734 2018-06-24 04:58:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367454008 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367454227 2018-06-24 04:58:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367455525 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367455744 2018-06-24 04:58:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367457985 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367458204 2018-06-24 04:58:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367460160 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367460379 2018-06-24 04:58:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367463804 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367464023 2018-06-24 04:58:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367464762 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367464981 2018-06-24 04:58:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367466279 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367466498 2018-06-24 04:58:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367468803 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367469022 2018-06-24 04:58:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367472001 2018-06-24 04:58:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367472220 2018-06-24 04:58:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367474606 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367474825 2018-06-24 04:58:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367477152 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367477371 2018-06-24 04:58:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367478669 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367478888 2018-06-24 04:58:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367481150 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367481369 2018-06-24 04:58:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367483700 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367483919 2018-06-24 04:58:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367486193 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367486412 2018-06-24 04:58:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367488743 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367488962 2018-06-24 04:58:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367492087 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367492306 2018-06-24 04:58:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367493604 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367493823 2018-06-24 04:58:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367495121 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367495340 2018-06-24 04:58:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367496638 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367496857 2018-06-24 04:58:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367498155 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367498374 2018-06-24 04:58:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367500730 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367500949 2018-06-24 04:58:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367503785 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367504004 2018-06-24 04:58:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367505302 2018-06-24 04:58:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367505521 2018-06-24 04:58:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367561248 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367561467 2018-06-24 04:59:00 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367562765 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367562984 2018-06-24 04:59:00 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367568066 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367568285 2018-06-24 04:59:00 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367576434 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367576653 2018-06-24 04:59:00 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367577871 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367578090 2018-06-24 04:59:00 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367588579 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367588798 2018-06-24 04:59:00 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367592065 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367592284 2018-06-24 04:59:00 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367596135 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367596354 2018-06-24 04:59:00 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367599194 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367599413 2018-06-24 04:59:00 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367603730 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367603949 2018-06-24 04:59:00 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367607135 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367607354 2018-06-24 04:59:00 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367610022 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367610241 2018-06-24 04:59:00 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367613298 2018-06-24 04:59:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367613517 2018-06-24 04:59:00 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367616505 2018-06-24 04:59:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367616724 2018-06-24 04:59:01 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367619895 2018-06-24 04:59:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367620114 2018-06-24 04:59:01 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367622814 2018-06-24 04:59:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367623033 2018-06-24 04:59:01 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367625848 2018-06-24 04:59:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367626067 2018-06-24 04:59:01 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367631444 2018-06-24 04:59:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367631663 2018-06-24 04:59:01 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367637850 2018-06-24 04:59:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367638069 2018-06-24 04:59:01 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367639172 2018-06-24 04:59:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367639391 2018-06-24 04:59:01 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367641709 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367641928 2018-06-24 04:59:02 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367657949 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367658652 2018-06-24 04:59:02 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367660000 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367660219 2018-06-24 04:59:02 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367661517 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367661736 2018-06-24 04:59:02 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367663034 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367663253 2018-06-24 04:59:02 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367678558 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367679277 2018-06-24 04:59:02 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367695650 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367696390 2018-06-24 04:59:02 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367697738 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367697957 2018-06-24 04:59:02 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367699255 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367699474 2018-06-24 04:59:02 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367700772 2018-06-24 04:59:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367700991 2018-06-24 04:59:02 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367717895 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367718631 2018-06-24 04:59:03 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367734963 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367735723 2018-06-24 04:59:03 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367736808 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367737027 2018-06-24 04:59:03 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367739300 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367739519 2018-06-24 04:59:03 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367740817 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367741036 2018-06-24 04:59:03 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367755831 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367756512 2018-06-24 04:59:03 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367771348 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367772063 2018-06-24 04:59:03 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367786107 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367786753 2018-06-24 04:59:03 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367789098 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367789317 2018-06-24 04:59:03 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367791653 2018-06-24 04:59:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367791872 2018-06-24 04:59:03 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367813764 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367813983 2018-06-24 04:59:04 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367814656 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367814875 2018-06-24 04:59:04 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367825876 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367826579 2018-06-24 04:59:04 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367840953 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367841605 2018-06-24 04:59:04 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367843800 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367844019 2018-06-24 04:59:04 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367846248 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367846467 2018-06-24 04:59:04 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367848622 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367848841 2018-06-24 04:59:04 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367858864 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367859497 2018-06-24 04:59:04 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367870689 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367871364 2018-06-24 04:59:04 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367883009 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367883722 2018-06-24 04:59:04 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367893879 2018-06-24 04:59:04 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367894512 2018-06-24 04:59:04 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367905872 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367906557 2018-06-24 04:59:05 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367910166 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367910418 2018-06-24 04:59:05 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367911442 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367911739 2018-06-24 04:59:05 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367914092 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367914311 2018-06-24 04:59:05 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367916546 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367916765 2018-06-24 04:59:05 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367933332 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367934035 2018-06-24 04:59:05 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367949345 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367950016 2018-06-24 04:59:05 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367966606 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367967318 2018-06-24 04:59:05 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367969814 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367970033 2018-06-24 04:59:05 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367972371 2018-06-24 04:59:05 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367972590 2018-06-24 04:59:05 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367976064 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367976283 2018-06-24 04:59:06 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367976989 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367977208 2018-06-24 04:59:06 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367989106 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 367989870 2018-06-24 04:59:06 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368000081 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368000733 2018-06-24 04:59:06 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368010866 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368011522 2018-06-24 04:59:06 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368026427 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368027096 2018-06-24 04:59:06 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368038531 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368039230 2018-06-24 04:59:06 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368050989 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368051753 2018-06-24 04:59:06 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368067668 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368068385 2018-06-24 04:59:06 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368073610 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368073829 2018-06-24 04:59:06 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368086639 2018-06-24 04:59:06 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368087354 2018-06-24 04:59:06 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368097568 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368098211 2018-06-24 04:59:07 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368108602 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368109280 2018-06-24 04:59:07 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368110628 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368110847 2018-06-24 04:59:07 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368113113 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368113332 2018-06-24 04:59:07 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368127398 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368128069 2018-06-24 04:59:07 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368139339 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368140081 2018-06-24 04:59:07 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368151063 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368151780 2018-06-24 04:59:07 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368163404 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368164113 2018-06-24 04:59:07 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368171442 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368171661 2018-06-24 04:59:07 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368173091 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368173310 2018-06-24 04:59:07 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368175590 2018-06-24 04:59:07 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368175809 2018-06-24 04:59:07 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368192426 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368193105 2018-06-24 04:59:08 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368206738 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368207400 2018-06-24 04:59:08 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368222128 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368222796 2018-06-24 04:59:08 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368225081 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368225300 2018-06-24 04:59:08 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368227548 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368227767 2018-06-24 04:59:08 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368230722 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368230941 2018-06-24 04:59:08 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368232239 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368232458 2018-06-24 04:59:08 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368237673 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368238326 2018-06-24 04:59:08 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368239674 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368239893 2018-06-24 04:59:08 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368241191 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368241410 2018-06-24 04:59:08 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368242708 2018-06-24 04:59:08 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368242927 2018-06-24 04:59:08 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368248937 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368249698 2018-06-24 04:59:09 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368255008 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368255657 2018-06-24 04:59:09 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368261012 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368261687 2018-06-24 04:59:09 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368263035 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368263254 2018-06-24 04:59:09 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368264552 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368264771 2018-06-24 04:59:09 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368266069 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368266288 2018-06-24 04:59:09 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368272214 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368272975 2018-06-24 04:59:09 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368278912 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368279608 2018-06-24 04:59:09 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368284897 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368285537 2018-06-24 04:59:09 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368291186 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368291900 2018-06-24 04:59:09 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368293248 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368293467 2018-06-24 04:59:09 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368294765 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368294984 2018-06-24 04:59:09 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368300697 2018-06-24 04:59:09 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368301436 2018-06-24 04:59:09 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368307352 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368308058 2018-06-24 04:59:10 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368313773 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368314495 2018-06-24 04:59:10 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368320321 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368321060 2018-06-24 04:59:10 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368322408 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368322627 2018-06-24 04:59:10 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368324833 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368325052 2018-06-24 04:59:10 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368327307 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368327526 2018-06-24 04:59:10 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368333350 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368334093 2018-06-24 04:59:10 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368340099 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368340862 2018-06-24 04:59:10 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368346412 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368347096 2018-06-24 04:59:10 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368348444 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368348663 2018-06-24 04:59:10 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368350980 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368351199 2018-06-24 04:59:10 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368353415 2018-06-24 04:59:10 18821 [Note] Multi-threaded slave statistics: seconds elapsed = 123; events assigned = 15038465; worker queues filled over overrun level = 0; waited due a Worker queue full = 0; waited due the total size = 0; slept when Workers occupied = 179253 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368353634 2018-06-24 04:59:10 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368359125 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368359843 2018-06-24 04:59:10 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368361570 2018-06-24 04:59:10 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368361789 2018-06-24 04:59:10 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368364437 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368364656 2018-06-24 04:59:11 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368368156 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368368375 2018-06-24 04:59:11 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368372148 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368372367 2018-06-24 04:59:11 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368373271 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368373490 2018-06-24 04:59:11 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368375762 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368375981 2018-06-24 04:59:11 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368379440 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368379659 2018-06-24 04:59:11 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368381925 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368382144 2018-06-24 04:59:11 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368386535 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368386754 2018-06-24 04:59:11 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368387460 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368387679 2018-06-24 04:59:11 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368389908 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368390127 2018-06-24 04:59:11 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368391425 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368391644 2018-06-24 04:59:11 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368394458 2018-06-24 04:59:11 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368394677 2018-06-24 04:59:11 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368396982 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368397201 2018-06-24 04:59:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368398499 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368398718 2018-06-24 04:59:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368401532 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368401751 2018-06-24 04:59:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368404028 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368404247 2018-06-24 04:59:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368406557 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368406776 2018-06-24 04:59:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368409839 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368410058 2018-06-24 04:59:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368413667 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368413886 2018-06-24 04:59:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368414757 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368414976 2018-06-24 04:59:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368417286 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368417505 2018-06-24 04:59:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368421730 2018-06-24 04:59:12 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368421949 2018-06-24 04:59:12 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368424862 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368425081 2018-06-24 04:59:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368427391 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368427610 2018-06-24 04:59:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368431018 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368431237 2018-06-24 04:59:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368432535 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368432754 2018-06-24 04:59:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368434810 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368435029 2018-06-24 04:59:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368436327 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368436546 2018-06-24 04:59:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368437844 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368438063 2018-06-24 04:59:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368439361 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368439580 2018-06-24 04:59:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368440878 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368441097 2018-06-24 04:59:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368444539 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368444758 2018-06-24 04:59:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368445464 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368445683 2018-06-24 04:59:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368448019 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368448238 2018-06-24 04:59:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368450538 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368450757 2018-06-24 04:59:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368452055 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368452274 2018-06-24 04:59:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368453572 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368453791 2018-06-24 04:59:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368455089 2018-06-24 04:59:13 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368455308 2018-06-24 04:59:13 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368456606 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368456825 2018-06-24 04:59:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368458123 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368458342 2018-06-24 04:59:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368459640 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368459859 2018-06-24 04:59:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368462247 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368462466 2018-06-24 04:59:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368463764 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368463983 2018-06-24 04:59:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368466893 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368467112 2018-06-24 04:59:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368469942 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368470161 2018-06-24 04:59:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368472566 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368472785 2018-06-24 04:59:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368475150 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368475369 2018-06-24 04:59:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368477698 2018-06-24 04:59:14 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368477917 2018-06-24 04:59:14 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:19 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368488027 2018-06-24 04:59:19 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368492888 2018-06-24 04:59:19 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 04:59:21 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368495882 2018-06-24 04:59:21 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368500743 2018-06-24 04:59:21 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368573640 2018-06-24 05:00:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368574293 2018-06-24 05:00:00 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368592140 2018-06-24 05:00:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368592840 2018-06-24 05:00:00 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368609675 2018-06-24 05:00:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368610389 2018-06-24 05:00:00 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368630327 2018-06-24 05:00:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368631066 2018-06-24 05:00:00 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368649642 2018-06-24 05:00:00 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368650350 2018-06-24 05:00:01 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368664610 2018-06-24 05:00:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368665273 2018-06-24 05:00:01 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368681949 2018-06-24 05:00:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368682651 2018-06-24 05:00:01 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368697980 2018-06-24 05:00:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368698609 2018-06-24 05:00:01 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368714595 2018-06-24 05:00:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368715236 2018-06-24 05:00:01 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368730629 2018-06-24 05:00:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368731309 2018-06-24 05:00:01 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368743205 2018-06-24 05:00:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368743930 2018-06-24 05:00:01 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368755786 2018-06-24 05:00:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368756532 2018-06-24 05:00:01 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368773378 2018-06-24 05:00:01 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368774069 2018-06-24 05:00:01 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:01 18821 [Warning] Access denied for user 'root'@'localhost' (using password: YES) 2018-06-24 05:00:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368786616 2018-06-24 05:00:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368787382 2018-06-24 05:00:02 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368799643 2018-06-24 05:00:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368800330 2018-06-24 05:00:02 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368812584 2018-06-24 05:00:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368813326 2018-06-24 05:00:02 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368831653 2018-06-24 05:00:02 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368832415 2018-06-24 05:00:02 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368848111 2018-06-24 05:00:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368848804 2018-06-24 05:00:03 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368859606 2018-06-24 05:00:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368860327 2018-06-24 05:00:03 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368870580 2018-06-24 05:00:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368871232 2018-06-24 05:00:03 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368881749 2018-06-24 05:00:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368882407 2018-06-24 05:00:03 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368898328 2018-06-24 05:00:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368899034 2018-06-24 05:00:03 18821 [Note] Coordinator synchronized with Workers, waited entries: 0, cant_sync: 0 2018-06-24 05:00:03 18821 [Warning] Aborted connection 181562207 to db: 'qrtz' user: 'qrtz' host: '10.25.182.65' (Got an error reading communication packets) 2018-06-24 05:00:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368915009 2018-06-24 05:00:03 18821 [Note] Coordinator and workers enter synchronization procedure when scheduling event relay-log: ./mysql-relay-bin.002412 pos: 368915716 InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:04:07 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 50 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836154 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 243.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 242.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 186.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773089 Mutex spin waits 340040021, rounds 3475069783, OS waits 89943705 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 243 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 244 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 145101 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 31638 Database pages 4017557 Old database pages 1482921 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 4017557, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 22272 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 1339 Database pages 504724 Old database pages 186294 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504724, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 1342 Database pages 504858 Old database pages 186343 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504858, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 1339 Database pages 504819 Old database pages 186329 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504819, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 1343 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 1334 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 1330 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 1339 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ ===================================== 2018-06-24 05:04:27 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 14 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836154 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 263.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 262.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 206.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773089 Mutex spin waits 340040021, rounds 3475069783, OS waits 89943705 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 263 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 264 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 144829 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 31910 Database pages 4017557 Old database pages 1482921 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 4017557, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 22301 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 1378 Database pages 504724 Old database pages 186294 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504724, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 1378 Database pages 504858 Old database pages 186343 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504858, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 1372 Database pages 504819 Old database pages 186329 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504819, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 1378 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 1367 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 1364 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 1372 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 272.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 271.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:04:47 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836154 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 283.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 282.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 226.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773089 Mutex spin waits 340040021, rounds 3475069783, OS waits 89943705 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 283 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 284 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 144452 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 32287 Database pages 4017557 Old database pages 1482921 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 4017557, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 22349 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 1423 Database pages 504724 Old database pages 186294 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504724, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 1433 Database pages 504858 Old database pages 186343 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504858, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 1414 Database pages 504819 Old database pages 186329 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504819, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 1427 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 1413 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 1408 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 1420 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 303.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 302.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 246.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:05:27 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 40 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836154 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 323.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 322.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 266.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773089 Mutex spin waits 340040021, rounds 3475069783, OS waits 89943705 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 323 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 324 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 143922 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 32817 Database pages 4017557 Old database pages 1482921 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 4017557, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 22408 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 1490 Database pages 504724 Old database pages 186294 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504724, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 1501 Database pages 504858 Old database pages 186343 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504858, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 1481 Database pages 504819 Old database pages 186329 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504819, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 1491 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 1483 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 1481 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 1482 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 334.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 333.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 277.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:05:47 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836155 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 343.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 342.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 286.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 --Thread 139684315379456 has waited at buf0flu.cc line 1755 for 9.00 seconds the semaphore: Mutex at 0x1fb98a8 created file buf0buf.cc line 1334, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773089 Mutex spin waits 340040022, rounds 3475069813, OS waits 89943706 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 343 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 344 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 143580 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 54367 Database pages 3996349 Old database pages 1475132 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3996349, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 22455 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 22741 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 1539 Database pages 504858 Old database pages 186343 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504858, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 1520 Database pages 504819 Old database pages 186329 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504819, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 1526 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 1528 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 1533 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 1525 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ ===================================== 2018-06-24 05:06:06 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 19 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836158 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 362.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 361.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 305.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773091 Mutex spin waits 340040025, rounds 3475069903, OS waits 89943709 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 362 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 363 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 143326 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 54621 Database pages 3996349 Old database pages 1475132 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3996349, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 22487 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 22775 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 1563 Database pages 504858 Old database pages 186343 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504858, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 1553 Database pages 504819 Old database pages 186329 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504819, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 1561 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 1559 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 1565 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 1558 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 365.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 364.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 308.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:06:26 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836158 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 382.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 381.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 325.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773091 Mutex spin waits 340040025, rounds 3475069903, OS waits 89943709 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 382 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 383 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 142940 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 55007 Database pages 3996349 Old database pages 1475132 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3996349, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 22535 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 22815 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 1618 Database pages 504858 Old database pages 186343 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504858, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 1605 Database pages 504819 Old database pages 186329 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504819, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 1606 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 1612 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 1607 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 1609 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 396.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 395.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 339.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:06:46 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836158 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 402.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 401.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 345.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773091 Mutex spin waits 340040025, rounds 3475069903, OS waits 89943709 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 402 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 403 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 142521 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 55426 Database pages 3996349 Old database pages 1475132 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3996349, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 22587 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 22864 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 1674 Database pages 504858 Old database pages 186343 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504858, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 1653 Database pages 504819 Old database pages 186329 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504819, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 1661 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 1669 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 1652 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 1666 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ ===================================== 2018-06-24 05:07:06 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836158 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 422.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 421.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 365.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773091 Mutex spin waits 340040025, rounds 3475069903, OS waits 89943709 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 422 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 423 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 142225 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 55722 Database pages 3996349 Old database pages 1475132 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3996349, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 22622 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 22903 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 1707 Database pages 504858 Old database pages 186343 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504858, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 1685 Database pages 504819 Old database pages 186329 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504819, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 1703 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 1708 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 1690 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 1704 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 427.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 426.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 370.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:07:26 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836158 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 442.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 441.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 385.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773091 Mutex spin waits 340040025, rounds 3475069903, OS waits 89943709 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 442 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 443 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 141886 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 56061 Database pages 3996349 Old database pages 1475132 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3996349, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 22661 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 22947 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 1747 Database pages 504858 Old database pages 186343 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504858, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 1732 Database pages 504819 Old database pages 186329 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504819, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 1751 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 1745 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 1728 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 1750 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 458.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 457.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 401.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:07:46 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836159 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 462.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 461.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 405.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 --Thread 139684315379456 has waited at buf0flu.cc line 1755 for 10.00 seconds the semaphore: Mutex at 0x1fb9c48 created file buf0buf.cc line 1334, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773091 Mutex spin waits 340040026, rounds 3475069933, OS waits 89943710 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 462 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 463 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 141542 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 78060 Database pages 3974694 Old database pages 1467178 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3974694, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 22705 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 22986 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 23438 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 1780 Database pages 504819 Old database pages 186329 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504819, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 1792 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 1785 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 1778 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 1796 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ ===================================== 2018-06-24 05:08:06 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836162 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 482.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 481.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 425.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773093 Mutex spin waits 340040029, rounds 3475070023, OS waits 89943713 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 482 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 483 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 141259 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 78343 Database pages 3974694 Old database pages 1467178 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3974694, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 22738 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23018 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 23475 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 1822 Database pages 504819 Old database pages 186329 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504819, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 1829 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 1817 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 1811 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 1833 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 489.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 488.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 432.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:08:26 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836162 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 502.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 501.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 445.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773093 Mutex spin waits 340040029, rounds 3475070023, OS waits 89943713 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 502 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 503 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 140732 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 78870 Database pages 3974694 Old database pages 1467178 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3974694, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 22799 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23087 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 23543 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 1889 Database pages 504819 Old database pages 186329 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504819, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 1898 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 1881 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 1872 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 1901 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 520.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 519.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 463.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:08:46 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836162 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 522.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 521.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 465.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773093 Mutex spin waits 340040029, rounds 3475070023, OS waits 89943713 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 522 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 523 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 140313 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 79289 Database pages 3974694 Old database pages 1467178 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3974694, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 22848 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23143 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 23610 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 1942 Database pages 504819 Old database pages 186329 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504819, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 1947 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 1929 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 1917 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 1953 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ ===================================== 2018-06-24 05:09:06 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836162 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 542.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 541.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 485.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773093 Mutex spin waits 340040029, rounds 3475070023, OS waits 89943713 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 542 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 543 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 139929 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 79673 Database pages 3974694 Old database pages 1467178 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3974694, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 22896 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23193 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 23662 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 1990 Database pages 504819 Old database pages 186329 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504819, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 1994 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 1977 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 1959 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2002 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 551.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 550.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 494.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:09:26 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 13 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836162 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 562.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 561.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 505.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773093 Mutex spin waits 340040029, rounds 3475070023, OS waits 89943713 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 562 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 563 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 139410 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 80192 Database pages 3974694 Old database pages 1467178 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3974694, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 22967 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23263 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 23722 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 2047 Database pages 504819 Old database pages 186329 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504819, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 2061 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 2038 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2025 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2069 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 582.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 581.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 525.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:09:46 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836164 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 582.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 581.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 525.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773095 Mutex spin waits 340040031, rounds 3475070083, OS waits 89943715 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 582 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 583 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 139319 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 101409 Database pages 3953568 Old database pages 1459419 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3953568, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 22977 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23275 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 23737 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23184 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 2076 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 2049 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2034 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2077 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ 2018-06-24 05:10:01 18821 [Warning] Access denied for user 'root'@'localhost' (using password: YES) ===================================== 2018-06-24 05:10:06 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836164 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 602.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 601.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 545.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773095 Mutex spin waits 340040031, rounds 3475070083, OS waits 89943715 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 602 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 603 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 138729 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 101999 Database pages 3953568 Old database pages 1459419 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3953568, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 23047 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23350 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 23810 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23259 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 2155 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 2125 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2107 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2146 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 613.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 612.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 556.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:10:26 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836164 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 622.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 621.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 565.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773095 Mutex spin waits 340040031, rounds 3475070083, OS waits 89943715 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 622 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 623 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 138257 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 102471 Database pages 3953568 Old database pages 1459419 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3953568, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 23106 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23410 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 23873 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23317 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 2212 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 2181 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2166 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2206 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ ===================================== 2018-06-24 05:10:46 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836164 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 642.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 641.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 585.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773095 Mutex spin waits 340040031, rounds 3475070083, OS waits 89943715 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 642 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 643 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 137858 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 102870 Database pages 3953568 Old database pages 1459419 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3953568, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 23156 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23460 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 23921 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23369 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 2263 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 2231 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2216 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2254 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 644.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 643.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 587.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:11:06 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836164 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 662.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 661.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 605.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773095 Mutex spin waits 340040031, rounds 3475070083, OS waits 89943715 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 662 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 663 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 137377 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 103351 Database pages 3953568 Old database pages 1459419 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3953568, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 23221 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23522 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 23977 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23428 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 2321 Database pages 504787 Old database pages 186317 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504787, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 2292 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2280 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2310 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 675.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 674.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 618.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:11:26 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836165 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 682.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 681.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 625.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773096 Mutex spin waits 340040032, rounds 3475070113, OS waits 89943716 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 682 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 683 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 137119 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 125279 Database pages 3931898 Old database pages 1451460 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3931898, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 23256 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23552 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 24007 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23460 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 24024 Database pages 483117 Old database pages 178358 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483117, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 2326 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2310 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2344 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ ===================================== 2018-06-24 05:11:46 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836165 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 702.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 701.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 645.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773096 Mutex spin waits 340040032, rounds 3475070113, OS waits 89943716 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 702 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 703 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 136605 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 125793 Database pages 3931898 Old database pages 1451460 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3931898, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 23321 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23615 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 24071 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23526 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 24092 Database pages 483117 Old database pages 178358 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483117, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 2390 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2370 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2408 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 706.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 705.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 649.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:12:06 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836165 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 722.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 721.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 665.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773096 Mutex spin waits 340040032, rounds 3475070113, OS waits 89943716 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 722 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 723 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 136110 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 126288 Database pages 3931898 Old database pages 1451460 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3931898, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 23384 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23677 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 24138 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23591 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 24151 Database pages 483117 Old database pages 178358 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483117, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 2443 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2431 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2473 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 737.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 736.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 680.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:12:26 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836165 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 742.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 741.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 685.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773096 Mutex spin waits 340040032, rounds 3475070113, OS waits 89943716 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 742 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 743 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 135753 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 126645 Database pages 3931898 Old database pages 1451460 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3931898, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 23429 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23721 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 24183 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23637 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 24194 Database pages 483117 Old database pages 178358 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483117, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 2489 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2474 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2518 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ ===================================== 2018-06-24 05:12:46 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836165 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 762.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 761.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 705.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773096 Mutex spin waits 340040032, rounds 3475070113, OS waits 89943716 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 762 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 763 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 135541 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 126857 Database pages 3931898 Old database pages 1451460 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3931898, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 23455 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23747 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 24208 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23665 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 24221 Database pages 483117 Old database pages 178358 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483117, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 2516 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2502 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2543 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 768.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 767.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 711.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:13:06 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836165 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 782.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 781.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 725.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773096 Mutex spin waits 340040032, rounds 3475070113, OS waits 89943716 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 782 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 783 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 135248 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 127150 Database pages 3931898 Old database pages 1451460 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3931898, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 23491 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23781 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 24246 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23703 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 24257 Database pages 483117 Old database pages 178358 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483117, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 2553 Database pages 504846 Old database pages 186339 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504846, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2539 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2580 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 799.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 798.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 742.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:13:26 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836167 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 802.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 801.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 745.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 --Thread 139684315379456 has waited at buf0flu.cc line 1755 for 14.00 seconds the semaphore: Mutex at 0x1fba728 created file buf0buf.cc line 1334, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773097 Mutex spin waits 340040034, rounds 3475070173, OS waits 89943718 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 802 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 803 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 135225 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 830.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 829.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 773.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 Buffer pool size 4194296 Free buffers 147887 Database pages 3911184 Old database pages 1443853 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3911184, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 23494 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23784 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 24249 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23706 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 24260 Database pages 483117 Old database pages 178358 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483117, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 23270 Database pages 484132 Old database pages 178732 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 484132, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2542 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2582 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ ===================================== 2018-06-24 05:13:59 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 33 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836171 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 835.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 834.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 778.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773100 Mutex spin waits 340040038, rounds 3475070293, OS waits 89943722 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 835 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 836 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 135218 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 147894 Database pages 3911184 Old database pages 1443853 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3911184, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 23495 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23785 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 24250 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23707 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 24261 Database pages 483117 Old database pages 178358 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483117, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 23270 Database pages 484132 Old database pages 178732 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 484132, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2543 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2583 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ ===================================== 2018-06-24 05:14:19 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836171 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 855.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 854.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 798.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773100 Mutex spin waits 340040038, rounds 3475070293, OS waits 89943722 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 855 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 856 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 135136 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 147976 Database pages 3911184 Old database pages 1443853 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3911184, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 23504 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23796 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 24261 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23718 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 24270 Database pages 483117 Old database pages 178358 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483117, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 23280 Database pages 484132 Old database pages 178732 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 484132, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2554 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2593 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 861.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 860.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 804.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:14:39 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836171 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 875.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 874.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 818.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773100 Mutex spin waits 340040038, rounds 3475070293, OS waits 89943722 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 875 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 876 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 135041 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 148071 Database pages 3911184 Old database pages 1443853 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3911184, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 23516 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23807 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 24272 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23731 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 24283 Database pages 483117 Old database pages 178358 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483117, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 23293 Database pages 484132 Old database pages 178732 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 484132, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2564 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2605 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 892.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 891.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 835.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:14:59 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836171 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 895.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 894.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 838.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773100 Mutex spin waits 340040038, rounds 3475070293, OS waits 89943722 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 895 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 896 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 134947 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 148165 Database pages 3911184 Old database pages 1443853 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3911184, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 23528 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23819 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 24284 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23742 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 24292 Database pages 483117 Old database pages 178358 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483117, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 23306 Database pages 484132 Old database pages 178732 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 484132, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2577 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2617 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ ===================================== 2018-06-24 05:15:19 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836171 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 915.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 914.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 858.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773100 Mutex spin waits 340040038, rounds 3475070293, OS waits 89943722 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 915 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 916 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 134872 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 148240 Database pages 3911184 Old database pages 1443853 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3911184, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 23536 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23828 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 24293 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23752 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 24302 Database pages 483117 Old database pages 178358 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483117, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 23315 Database pages 484132 Old database pages 178732 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 484132, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2587 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2627 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 923.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 922.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 InnoDB: Warning: a long semaphore wait: --Thread 139684280211200 has waited at dict0dict.cc line 560 for 866.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2018-06-24 05:15:39 7f0acaffd700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 21669014 srv_active, 0 srv_shutdown, 10482952 srv_idle srv_master_thread log flush and writes: 32151957 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 211836171 --Thread 139684271425280 has waited at row0mysql.cc line 1739 for 935.00 seconds the semaphore: S-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684429113088 has waited at srv0srv.cc line 2006 for 934.00 seconds the semaphore: X-lock on RW-latch at 0x1396b00 created in file dict0dict.cc line 1066 a writer (thread id 139684265568000) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0ins.cc line 1791 Last time write locked in file /export/home/pb2/build/sb_0-18153926-1457020193.06/rpm/BUILD/mysql-5.6.30/mysql-5.6.30/storage/innobase/row/row0mysql.cc line 3287 --Thread 139684280211200 has waited at dict0dict.cc line 560 for 878.00 seconds the semaphore: Mutex at 0x6e057208 created file dict0dict.cc line 1057, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 252773100 Mutex spin waits 340040038, rounds 3475070293, OS waits 89943722 RW-shared spins 85594399, rounds 1832704264, OS waits 50840716 RW-excl spins 156813656, rounds 3111600796, OS waits 59509405 Spin rounds per wait: 10.22 mutex, 21.41 RW-shared, 19.84 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 4519807940 Purge done for trx's n:o < 4519807937 undo n:o < 0 state: running but idle History list length 1181 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 4519667348, not started MySQL thread id 181559348, OS thread handle 0x7f0ac7304700, query id 7616299080 10.25.182.31 tyqk_server cleaning up ---TRANSACTION 4519806692, not started MySQL thread id 181546711, OS thread handle 0x7f0ac7345700, query id 7616498243 Waiting for an event from Coordinator ---TRANSACTION 4515354295, not started MySQL thread id 181546710, OS thread handle 0x7f0ac7a20700, query id 0 Waiting for Slave Worker to release partition ---TRANSACTION 4519807939, ACTIVE 935 sec truncating table mysql tables in use 1, locked 1 MySQL thread id 181562206, OS thread handle 0x7f0ac707a700, query id 7616499499 10.25.182.65 qrtz System lock TRUNCATE table qrtz.ljbos_sitedata ---TRANSACTION 4519807937, ACTIVE 936 sec updating or deleting mysql tables in use 1, locked 1 8 lock struct(s), heap size 1184, 17 row lock(s), undo log entries 30 MySQL thread id 181546712, OS thread handle 0x7f0ac7610700, query id 7616499515 invalidating query cache entries (table) -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 577400206 OS file reads, 1535435390 OS file writes, 373055379 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 623661, seg size 623663, 82309514 merges merged operations: insert 145934649, delete mark 113740744, delete 8791044 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 151384511, node heap has 134780 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 6672125133604 Log flushed up to 6672125133604 Pages flushed up to 6672125133604 Last checkpoint at 6672124522908 0 pending log writes, 0 pending chkp writes 29327479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 70330089472; in additional pool allocated 0 Dictionary memory allocated 2488430 Buffer pool size 4194296 Free buffers 148332 Database pages 3911184 Old database pages 1443853 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 244661567, not young 14547884225 0.00 youngs/s, 0.00 non-youngs/s Pages read 576000267, created 138992943, written 1417535285 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3911184, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 524287 Free buffers 23545 Database pages 483883 Old database pages 178640 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30738453, not young 1826154622 0.00 youngs/s, 0.00 non-youngs/s Pages read 72235791, created 17378873, written 294166148 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483883, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 524287 Free buffers 23841 Database pages 483516 Old database pages 178505 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30501376, not young 1812026218 0.00 youngs/s, 0.00 non-youngs/s Pages read 71828456, created 17388485, written 147012635 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483516, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 524287 Free buffers 24303 Database pages 483203 Old database pages 178389 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30714834, not young 1827872958 0.00 youngs/s, 0.00 non-youngs/s Pages read 72206222, created 17399727, written 145729127 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483203, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 524287 Free buffers 23765 Database pages 483693 Old database pages 178570 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30684124, not young 1820683092 0.00 youngs/s, 0.00 non-youngs/s Pages read 72214383, created 17413662, written 176504284 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483693, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 524287 Free buffers 24314 Database pages 483117 Old database pages 178358 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30448866, not young 1818935202 0.00 youngs/s, 0.00 non-youngs/s Pages read 72106483, created 17409278, written 164154754 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 483117, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 524287 Free buffers 23328 Database pages 484132 Old database pages 178732 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30598157, not young 1811513833 0.00 youngs/s, 0.00 non-youngs/s Pages read 71860417, created 17378187, written 149922864 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 484132, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 524287 Free buffers 2597 Database pages 504831 Old database pages 186333 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30464683, not young 1811710232 0.00 youngs/s, 0.00 non-youngs/s Pages read 71349286, created 17234075, written 194944133 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504831, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 524287 Free buffers 2639 Database pages 504809 Old database pages 186326 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 30511074, not young 1818988068 0.00 youngs/s, 0.00 non-youngs/s Pages read 72199229, created 17390656, written 145101340 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 504809, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Main thread process no. 18821, id 139684429113088, state: enforcing dict cache limit Number of rows inserted 3234569564, updated 1380585889, deleted 453842807, read 71076663546 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Error: semaphore wait has lasted > 600 seconds InnoDB: We intentionally crash the server, because it appears to be hung. 2018-06-24 05:15:57 7f0acb7fe700 InnoDB: Assertion failure in thread 139684340557568 in file srv0srv.cc line 1756 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 21:15:58 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388608 read_buffer_size=2097152 max_used_connections=127 max_threads=1000 thread_count=20 connection_count=16 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 18452988 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x3b)[0x8e48eb] /usr/sbin/mysqld(handle_fatal_signal+0x491)[0x674141] /lib64/libpthread.so.0(+0xf100)[0x7f1b9cb64100] /lib64/libc.so.6(gsignal+0x37)[0x7f1b9b75f5f7] /lib64/libc.so.6(abort+0x148)[0x7f1b9b760ce8] /usr/sbin/mysqld[0x9cf6a7] /lib64/libpthread.so.0(+0x7dc5)[0x7f1b9cb5cdc5] /lib64/libc.so.6(clone+0x6d)[0x7f1b9b820ced] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 180624 05:16:18 mysqld_safe Number of processes running now: 0 180624 05:16:18 mysqld_safe mysqld restarted 2018-06-24 05:16:19 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2018-06-24 05:16:19 0 [Note] /usr/sbin/mysqld (mysqld 5.6.30-log) starting as process 20572 ... 2018-06-24 05:16:19 20572 [Note] Plugin 'FEDERATED' is disabled. 2018-06-24 05:16:19 20572 [ERROR] Function 'validate_password' already exists 2018-06-24 05:16:19 20572 [Warning] Couldn't load plugin named 'validate_password' with soname 'validate_password.so'. 2018-06-24 05:16:19 20572 [Note] InnoDB: Using atomics to ref count buffer pool pages 2018-06-24 05:16:19 20572 [Note] InnoDB: The InnoDB memory heap is disabled 2018-06-24 05:16:19 20572 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2018-06-24 05:16:19 20572 [Note] InnoDB: Memory barrier is not used 2018-06-24 05:16:19 20572 [Note] InnoDB: Compressed tables use zlib 1.2.3 2018-06-24 05:16:19 20572 [Note] InnoDB: Using Linux native AIO 2018-06-24 05:16:19 20572 [Note] InnoDB: Not using CPU crc32 instructions 2018-06-24 05:16:19 20572 [Note] InnoDB: Initializing buffer pool, size = 64.0G 2018-06-24 05:16:25 20572 [Note] InnoDB: Completed initialization of buffer pool 2018-06-24 05:16:26 20572 [Note] InnoDB: Highest supported file format is Barracuda. 2018-06-24 05:16:26 20572 [Note] InnoDB: Log scan progressed past the checkpoint lsn 6672124522908 2018-06-24 05:16:26 20572 [Note] InnoDB: Database was not shutdown normally! 2018-06-24 05:16:26 20572 [Note] InnoDB: Starting crash recovery. 2018-06-24 05:16:26 20572 [Note] InnoDB: Reading tablespace information from the .ibd files... 2018-06-24 05:16:28 20572 [Note] InnoDB: Restoring possible half-written data pages 2018-06-24 05:16:28 20572 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 6672125133604 InnoDB: 2 transaction(s) which must be rolled back or cleaned up InnoDB: in total 30 row operations to undo InnoDB: Trx id counter is 4519808256 2018-06-24 05:16:30 20572 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 367998019, file name mysql-bin.001130 2018-06-24 05:16:41 7efc3ab5b740 InnoDB: Rolling back trx with id 4519807939, 0 rows to undo 2018-06-24 05:16:41 20572 [Note] InnoDB: Rollback of trx with id 4519807939 completed 2018-06-24 05:16:41 20572 [Note] InnoDB: 128 rollback segment(s) are active. InnoDB: Starting in background the rollback of uncommitted transactions 2018-06-24 05:16:41 7eeb25f50700 InnoDB: Rolling back trx with id 4519807937, 30 rows to undo 2018-06-24 05:16:41 20572 [Note] InnoDB: Waiting for purge to start 2018-06-24 05:16:41 20572 [Note] InnoDB: 5.6.30 started; log sequence number 6672125133604 2018-06-24 05:16:41 20572 [Warning] Plugin validate_password reported: 'Dictionary file not specified' 2018-06-24 05:16:41 20572 [Note] Recovering after a crash using /data/log/bin/mysql-bin 2018-06-24 05:16:41 20572 [Note] InnoDB: Rollback of trx with id 4519807937 completed 2018-06-24 05:16:41 7eeb25f50700 InnoDB: Rollback of non-prepared transactions completed 2018-06-24 05:16:50 20572 [Note] Starting crash recovery... 2018-06-24 05:16:50 20572 [Note] Crash recovery finished. 2018-06-24 05:16:52 20572 [Note] Server hostname (bind-address): '*'; port: 3306 2018-06-24 05:16:52 20572 [Note] IPv6 is available. 2018-06-24 05:16:52 20572 [Note] - '::' resolves to '::'; 2018-06-24 05:16:52 20572 [Note] Server socket created on IP: '::'. 2018-06-24 05:16:52 20572 [Warning] 'proxies_priv' entry '@ root@localhost.localdomain' ignored in --skip-name-resolve mode. 2018-06-24 05:16:52 20572 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=mysql-relay-bin' to avoid this problem. 2018-06-24 05:17:00 20572 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. 2018-06-24 05:17:00 20572 [Note] Slave: MTS group recovery relay log info based on Worker-Id 1, group_relay_log_name ./mysql-relay-bin.002412, group_relay_log_pos 368899444 group_master_log_name mysql-bin.002222, group_master_log_pos 368899314 2018-06-24 05:17:00 20572 [Note] Slave: MTS group recovery relay log info group_master_log_name mysql-bin.002222, event_master_log_pos 368882308. 2018-06-24 05:17:00 20572 [Note] Slave: MTS group recovery relay log info group_master_log_name mysql-bin.002222, event_master_log_pos 368882687. 2018-06-24 05:17:00 20572 [Note] Slave: MTS group recovery relay log info group_master_log_name mysql-bin.002222, event_master_log_pos 368898935. 2018-06-24 05:17:00 20572 [Note] Slave: MTS group recovery relay log info group_master_log_name mysql-bin.002222, event_master_log_pos 368899314. 2018-06-24 05:17:00 20572 [Note] Slave I/O thread: connected to master 'repl@10.25.182.65:3306',replication started in log 'mysql-bin.002222' at position 381177493 2018-06-24 05:17:00 20572 [Warning] Slave SQL: If a crash happens this configuration does not guarantee that the relay log info will be consistent, Error_code: 0 2018-06-24 05:17:00 20572 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.002222' at position 368871512, relay log './mysql-relay-bin.002412' position: 368871642 2018-06-24 05:17:00 20572 [Note] Slave: MTS Recovery has completed at relay log ./mysql-relay-bin.002412, position 368899444 master log mysql-bin.002222, position 368899314. 2018-06-24 05:17:00 20572 [Note] Event Scheduler: Loaded 0 events 2018-06-24 05:17:00 20572 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.6.30-log' socket: '/var/run/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)