Comments on: Quest for Resilience: Multi-DC Masters http://openquery.com/blog/quest-resilience-multidc-masters About MySQL, Drizzle, MariaDB and more! Mon, 19 Mar 2012 14:26:12 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Jeremy Cole http://openquery.com/blog/quest-resilience-multidc-masters/comment-page-1#comment-2513 Jeremy Cole Wed, 19 May 2010 06:30:03 +0000 http://openquery.com/blog/?p=1241#comment-2513 Hey Arjen, At Yahoo!, where this was a pretty common requirement, we came up with a model that used a simple TCP proxy to forward traffic, similar (but slightly more heavy-handed) to using a floating/role IP and IP takeover. Since IP takeover doesn't work across routers, it can't be used, but this gets you similar behaviour. Regards, Jeremy Hey Arjen,

At Yahoo!, where this was a pretty common requirement, we came up with a model that used a simple TCP proxy to forward traffic, similar (but slightly more heavy-handed) to using a floating/role IP and IP takeover. Since IP takeover doesn’t work across routers, it can’t be used, but this gets you similar behaviour.

Regards,

Jeremy

]]> By: Anon http://openquery.com/blog/quest-resilience-multidc-masters/comment-page-1#comment-2481 Anon Fri, 14 May 2010 17:36:59 +0000 http://openquery.com/blog/?p=1241#comment-2481 Running replication between DCs for a couple years, I've only seen seconds_behind > 1s when relay log processing stopped. (error, etc.) If the APP is reasonably efficient, and its behavior is known/measured, WAN connections shouldn't be a problem - read only client, or read/write (MM) assuming proper planning and SLA. But of course that's theory. Example problem: Important site gets an unexpected traffic spike due to a special event. Its Wordpress implementation is horribly inefficient (lots of poorly written queries in plugins, templates, etc.) and the primary server is struggling - hurting hundreds of other sites. (of course this happens at the end of a long day, with zero notice) Solution: Move connections to the slave, accessed over WAN, abandoning consistency but spreading the load between two servers. This works great, for several days. New Problem: Possibly due to DC admin's throttling, packet loss between the two servers hits ~30% and major problems return for the inefficient site. Simple queries slow, and active connections pile up. New Solution: The spike in traffic has subsided, so moving the connections back to the primary server, dump from slave, truncate tables on both M+S, and restore to master. Yeah, it's probably time for some capacity planning and disaster recovery discussions. :) Running replication between DCs for a couple years, I’ve only seen seconds_behind > 1s when relay log processing stopped. (error, etc.)

If the APP is reasonably efficient, and its behavior is known/measured, WAN connections shouldn’t be a problem – read only client, or read/write (MM) assuming proper planning and SLA. But of course that’s theory.

Example problem: Important site gets an unexpected traffic spike due to a special event. Its WordPress implementation is horribly inefficient (lots of poorly written queries in plugins, templates, etc.) and the primary server is struggling – hurting hundreds of other sites. (of course this happens at the end of a long day, with zero notice)

Solution: Move connections to the slave, accessed over WAN, abandoning consistency but spreading the load between two servers. This works great, for several days.

New Problem: Possibly due to DC admin’s throttling, packet loss between the two servers hits ~30% and major problems return for the inefficient site. Simple queries slow, and active connections pile up.

New Solution: The spike in traffic has subsided, so moving the connections back to the primary server, dump from slave, truncate tables on both M+S, and restore to master.

Yeah, it’s probably time for some capacity planning and disaster recovery discussions. :)

]]> By: paul http://openquery.com/blog/quest-resilience-multidc-masters/comment-page-1#comment-2479 paul Fri, 14 May 2010 06:21:19 +0000 http://openquery.com/blog/?p=1241#comment-2479 We run a geographic setup using circular replication and DRBD combined. Each DC has two MySQL master servers using DRBD failover to handle local failure on the nodes. Then the servers in the DC are linked together using circular replication. We wrote our own scripts to handle the change over of Slave nodes feeding off its local Master because we have had this kind of setup long before tools like MMM became available. We have web/application servers in each DC so if one DC goes down then all customer traffic is diverted to the other DC using DNS failover and a low TTL. DNS failover is not perfect but from our testing it works well enough for Web for ~94% of our customers. We run a geographic setup using circular replication and DRBD combined.

Each DC has two MySQL master servers using DRBD failover to handle local failure on the nodes. Then the servers in the DC are linked together using circular replication.

We wrote our own scripts to handle the change over of Slave nodes feeding off its local Master because we have had this kind of setup long before tools like MMM became available.

We have web/application servers in each DC so if one DC goes down then all customer traffic is diverted to the other DC using DNS failover and a low TTL. DNS failover is not perfect but from our testing it works well enough for Web for ~94% of our customers.

]]> By: Gerry http://openquery.com/blog/quest-resilience-multidc-masters/comment-page-1#comment-2477 Gerry Fri, 14 May 2010 01:27:10 +0000 http://openquery.com/blog/?p=1241#comment-2477 We have master-master and regular setups across data centers. The master-master setup is only for quick fail over. We monitor slave status very closely and we have determined that under normal conditions, the setup works just fine with all the caveats and considerations. I would not recommended for high traffic since it will be almost guaranteed that the slave will fall behind for a few seconds. How much is "high" will depend on the link between data centers and the particular application profile. Reading from the slave across data centers has never been an issue for us. My $.02 G We have master-master and regular setups across data centers. The master-master setup is only for quick fail over. We monitor slave status very closely and we have determined that under normal conditions, the setup works just fine with all the caveats and considerations.

I would not recommended for high traffic since it will be almost guaranteed that the slave will fall behind for a few seconds. How much is “high” will depend on the link between data centers and the particular application profile.

Reading from the slave across data centers has never been an issue for us.

My $.02
G

]]>