Comments on: Trivia: identify this replication failure

By: Ryan

Ryan — Wed, 28 Oct 2009 17:49:04 +0000

I agree with Shlomi’s example as the most likely cause. However I’ve also seen the same behaviour when a router was silently garbling packets — with enough frequency that some would pass through the crude TCP checksum and produce invalid SQL. Solution was the same though.

By: Daniel Schneller

Daniel Schneller — Wed, 28 Oct 2009 10:11:21 +0000

Writing binlogs and the actual table data is not atomic. With InnoDB, the binlog is only written after the transaction has been committed. When the server (is made to) crash when it’s writing the binlogs it might leave them unfinished, i. e. corrupt, while InnoDB rolls back the transaction.

The sync_binlog and innodb_support_xa setting should prevent this, however there can still be problems with too short binlogs.

To repair this there are several options. If there is only a small to moderate amount of data, using a fresh dump might be a solution. If this takes too long or is otherwise impractical, you need to find out up there where the slave executed correctly – the SHOW SLAVE STATUS command helps here. Then look at the master’s corresponding binlog and either set the skip_slave_counter appropriately (if the statement was executed correctly after the crash) or better reset the replication with CHANGE MASTER TO and point it to the new binlog the master would have started when it was brought back up again.

Anyway maybe a mk-table-checksum based verification would be advisable to see if you actually got it right

By: hartmut

hartmut — Wed, 28 Oct 2009 09:19:59 +0000

replication event checksums anyone?

By: Paul

Paul — Wed, 28 Oct 2009 08:53:58 +0000

Without knowing the exact error its not possible to make a real decision, however its highly likely the master DB has crashed and binlog has become corrupted. However the slave DB has become stuck reading to the end of the binlog file of the master with an “impossible position” type error.

Thus the quickest way to get the slave working again is to get the new master DB’s bin log name from column “Master _Log_file” using query;

SLAVE> SHOW SLAVE STATUS;

Then change replication position with;

SLAVE> STOP SLAVE; — to stop IO thread
SLAVE> CHANGE MASTER TO master_host=”, Master_user=”, Master_password = ”, Master_log_file = ”, Matser_log_pos=0;
SLAVE> START SLAVE;

This doesn’t ensure data integrity, but as mentioned about you need to know the nature of error to statt making decisions to ensure data integrity.

By: Shlomi Noach

Shlomi Noach — Wed, 28 Oct 2009 08:51:47 +0000

What’s wrong: the relay log file is corrupted. e.g. server restarted as it was being written.
Possible solution: check otu the master lgos file/position in “show slave status” (not the relay log one), and try “stop slave; change master to …; start slave”, thus reading from that same position again. The “change master to” will discard the existing relay log(s).