[ 2011-July-19 17:24 ]

Summary: With producer-consumer tasks, the Linux scheduler can make bad decisions. Long version: I've been doing some performance debugging and recently ran into an interesting puzzle. I have been working on a MySQL proxy that reads client requests from the network, manipulates them in some way, and then sends them to the local MySQL server using a local Unix socket. To test this, I put the proxy and MySQL on one machine, and a bunch of simulated clients on another. I set the proxy to do nothing but pass the requests through, and configured the client requests so the data is read-only and in memory. The system looks something like this:

Without the proxy, MySQL uses all available CPU on the system (showing ~1600% CPU consumption in top, since this system has 16 virtual cores). With the proxy, we would expect that maybe the throughput is a bit lower, but we would still use all the CPU available on the server, divided between the proxy and MySQL. Strangely, that isn't what happened. Instead, we ended up using approximately half the CPU on the system, with MySQL using ~750%, and my proxy using ~30%. Adjusting the number of clients made no difference. So where is the bottleneck? I'll give you a hint: my proxy is single threaded, using non-blocking IO to pass data between the client and server connections. I was certainly puzzled about this for a few days, until I gave my proxy its own core, putting MySQL on all other cores (taskset is a wonderful tool). Suddenly I saw the performance I was expecting: The proxy was using nearly 100% CPU, and MySQL was using close to 1500% CPU: There was no idle CPU time left on the system. So why did this happen?

My first thought was that suddenly I was seeing better cache behaviour. However, it turns out that the real reason is that the Linux process scheduler was making bad decisions. With 100 clients, I had 100 MySQL threads, all ready to run. So the Linux scheduler says: "great, let's distribute CPU time evenly between these 100 MySQL threads and the proxy." And there is the problem: all 100 MySQL threads respond, but then suddenly there is no work for them to do and they are idle, because the proxy hasn't been running. The proxy needs to run in order to produce work for MySQL. So what really needs to happen is that the proxy task needs to have higher priority. In effect, I gave it higher priority by giving it a private core.

The lesson here: the Linux scheduler is not magic, and queues are also more complicated than you might think. I'm also not the first to run into this. Ariel Weisberg ran into this same issue with a logging thread inside a multi-threaded Java application. In his case, he solved it by using multiple threads. However, you can also use tricks like pinning tasks to cores or adjusting thread priority to try and fight this kind of performance weirdness.