Planet

While a scale-out solution has traditionally been popular for MySQL, it’s interesting to see what room we now have to scale up – cheap memory, fast storage, better power efficiency.  There certainly are a lot of options now – I’ve been meeting about a customer/week using Fusion-IO cards.  One interesting choice I’ve seen people make however, is buying an SSD when they still have a lot of pages read/second – I would have preferred to buy memory instead, and use the storage device for writes.

Here’s the benchmark I came up with to confirm if this is the case:

• Percona-XtraDB-9.1 release
• Sysbench OLTP workload with 80 million rows (about 18GB worth of data+indexes)
• XFS Filesystem mounted with nobarrier option.
• Tests run with:
  • RAID10 with BBU over 8 disks
  • Intel SSD X25-E 32GB
  • FusionIO 320GB MLC
• For each test, run with a buffer pool of between 2G and 22G (to test performance compared to memory fit).
• Hardware was our Dell 900 (specs here).

To start with, we have a test on the RAID10 storage to establish a baseline.  The Y axis is transactions/second (more is better), the X axis is the size of innodb_buffer_pool_size:

Let me point out three interesting characteristics about this benchmark:

• The A arrow is when data fits completely in the buffer pool (best performance). It’s important to point out that once you hit this point, a further increase in memory at all.
• The B arrow is where the data just started to exceed the size of the buffer pool.  This is the most painful point for many customers – because while memory decreased by only ~10% the performance dropped by 2.6 times!  In production this usually matches the description of “Last week everything was fine.. but it’s just getting slower and slower!”.  I would suggest that adding memory is by far the best thing to do here.
• The C arrow shows where data is approximately three times the buffer pool.  This is an interesting point to zoom in on – since you may not be able to justify the cost of the memory, but an SSD might be a good fit:

Where the C arrow was, in this graph a Fusion-IO card improves performance by about five times (or 2x with an Intel SSD).  To get the same improvement with memory, you would have needed to add 60% more memory -or- 260% more memory for a 5x improvement.  Imagine a situation where your C point is when you have 32GB of RAM and 100GB of data.  Than it gets interesting:

• Can you easily add another 32G RAM (are your memory slots already filled?)
• Does your budget allow to install SSD cards? (You may still need more than one, since they are all relatively small.  There are already appliances on the market which use 8 Intel SSD devices).
• Is a 2x or 5x improvement enough?  There are more wins to be had if you can afford to buy all the memory that is required.

The workload here is designed to keep as much of the data hot as possible, but I guess the main lesson here is not to underestimate the size of your “active set” of data.  For some people who just append data to some sort of logging table it may only need to be a small percentage – but in other cases it can be considerably higher.  If you don’t know what your working set is – ask us!

Important note: This graph and these results are valid only for sysbench uniform. In your particular workload the points B and C may be located in differently.

Raw results:

Entry posted by Vadim |
One comment

Add to: | | | |

We’re hiring. We are looking for the following qualifications:

1. Expert knowledge of MySQL. Not just “certified” — years of production experience with it. You need to know server internals, for example. You need to be able to do anything from optimizing difficult queries to moving high-volume services between data centers without interruption.
2. Expert knowledge of InnoDB. You should understand its inner workings well enough to answer questions about its internals from memory, such as “how does the insert buffer work?” or “how does MVCC work on secondary indexes?” You should also know why it has trouble on some workloads and how to solve that.
3. Expert knowledge of Linux systems administration. You need to know how to solve issues with filesystems, hardware, and networking. You need to be able to use tools such as gdb, strace, tcpdump, etc to solve weird problems.
4. Expert with Apache, memcached, and other technologies you’d typically find in a LAMP application. You need to be able to administer and troubleshoot a variety of popular open-source server software.
5. Proficient with Maatkit and MMM.
6. Proven success working in a distributed environment where e-mail, IRC and voice calls are your only interaction with clients, colleagues and managers on a daily basis. You must be a self-starter.

We need all of the above, but if you’re a little weaker in some areas that might be OK. Highly desirable skills include:

1. Non-core MySQL technologies: NDB Cluster, third-party storage engines and appliances, etc.
2. Technologies such as LVS, Nginx, Sphinx, and load balancers.
3. Other operating systems such FreeBSD and OpenSolaris.
4. Amazon’s cloud offerings.
5. Programming languages and platforms, especially frameworks such as Ruby On Rails and Django.

If you think you’re a good candidate, please fill out the contact form on our website. Thanks!

Entry posted by Baron Schwartz |
No comment

Add to: | | | |