)]}' { "commit": "2bd2c92cf07cc4a373bf316c75b78ac465fefd35", "tree": "0d8e6121b5f9d364a3fa90489c8f8fe32717b63c", "parents": [ "0dc8c730c98a06a4d927f8d08bd0dd6de973b8dd" ], "author": { "name": "Waiman Long", "email": "Waiman.Long@hp.com", "time": "Wed Apr 17 15:23:13 2013 -0400" }, "committer": { "name": "Ingo Molnar", "email": "mingo@kernel.org", "time": "Fri Apr 19 09:33:36 2013 +0200" }, "message": "mutex: Queue mutex spinners with MCS lock to reduce cacheline contention\n\nThe current mutex spinning code (with MUTEX_SPIN_ON_OWNER option\nturned on) allow multiple tasks to spin on a single mutex\nconcurrently. A potential problem with the current approach is\nthat when the mutex becomes available, all the spinning tasks\nwill try to acquire the mutex more or less simultaneously. As a\nresult, there will be a lot of cacheline bouncing especially on\nsystems with a large number of CPUs.\n\nThis patch tries to reduce this kind of contention by putting\nthe mutex spinners into a queue so that only the first one in\nthe queue will try to acquire the mutex. This will reduce\ncontention and allow all the tasks to move forward faster.\n\nThe queuing of mutex spinners is done using an MCS lock based\nimplementation which will further reduce contention on the mutex\ncacheline than a similar ticket spinlock based implementation.\nThis patch will add a new field into the mutex data structure\nfor holding the MCS lock. This expands the mutex size by 8 bytes\nfor 64-bit system and 4 bytes for 32-bit system. This overhead\nwill be avoid if the MUTEX_SPIN_ON_OWNER option is turned off.\n\nThe following table shows the jobs per minute (JPM) scalability\ndata on an 8-node 80-core Westmere box with a 3.7.10 kernel. The\nnumactl command is used to restrict the running of the fserver\nworkloads to 1/2/4/8 nodes with hyperthreading off.\n\n+-----------------+-----------+-----------+-------------+----------+\n| Configuration | Mean JPM | Mean JPM | Mean JPM | % Change |\n| | w/o patch | patch 1 | patches 1\u00262 | 1-\u003e1\u00262 |\n+-----------------+------------------------------------------------+\n| | User Range 1100 - 2000 |\n+-----------------+------------------------------------------------+\n| 8 nodes, HT off | 227972 | 227237 | 305043 | +34.2% |\n| 4 nodes, HT off | 393503 | 381558 | 394650 | +3.4% |\n| 2 nodes, HT off | 334957 | 325240 | 338853 | +4.2% |\n| 1 node , HT off | 198141 | 197972 | 198075 | +0.1% |\n+-----------------+------------------------------------------------+\n| | User Range 200 - 1000 |\n+-----------------+------------------------------------------------+\n| 8 nodes, HT off | 282325 | 312870 | 332185 | +6.2% |\n| 4 nodes, HT off | 390698 | 378279 | 393419 | +4.0% |\n| 2 nodes, HT off | 336986 | 326543 | 340260 | +4.2% |\n| 1 node , HT off | 197588 | 197622 | 197582 | 0.0% |\n+-----------------+-----------+-----------+-------------+----------+\n\nAt low user range 10-100, the JPM differences were within +/-1%.\nSo they are not that interesting.\n\nThe fserver workload uses mutex spinning extensively. With just\nthe mutex change in the first patch, there is no noticeable\nchange in performance. Rather, there is a slight drop in\nperformance. This mutex spinning patch more than recovers the\nlost performance and show a significant increase of +30% at high\nuser load with the full 8 nodes. Similar improvements were also\nseen in a 3.8 kernel.\n\nThe table below shows the %time spent by different kernel\nfunctions as reported by perf when running the fserver workload\nat 1500 users with all 8 nodes.\n\n+-----------------------+-----------+---------+-------------+\n| Function | % time | % time | % time |\n| | w/o patch | patch 1 | patches 1\u00262 |\n+-----------------------+-----------+---------+-------------+\n| __read_lock_failed | 34.96% | 34.91% | 29.14% |\n| __write_lock_failed | 10.14% | 10.68% | 7.51% |\n| mutex_spin_on_owner | 3.62% | 3.42% | 2.33% |\n| mspin_lock | N/A | N/A | 9.90% |\n| __mutex_lock_slowpath | 1.46% | 0.81% | 0.14% |\n| _raw_spin_lock | 2.25% | 2.50% | 1.10% |\n+-----------------------+-----------+---------+-------------+\n\nThe fserver workload for an 8-node system is dominated by the\ncontention in the read/write lock. Mutex contention also plays a\nrole. With the first patch only, mutex contention is down (as\nshown by the __mutex_lock_slowpath figure) which help a little\nbit. We saw only a few percents improvement with that.\n\nBy applying patch 2 as well, the single mutex_spin_on_owner\nfigure is now split out into an additional mspin_lock figure.\nThe time increases from 3.42% to 11.23%. It shows a great\nreduction in contention among the spinners leading to a 30%\nimprovement. The time ratio 9.9/2.33\u003d4.3 indicates that there\nare on average 4+ spinners waiting in the spin_lock loop for\neach spinner in the mutex_spin_on_owner loop. Contention in\nother locking functions also go down by quite a lot.\n\nThe table below shows the performance change of both patches 1 \u0026\n2 over patch 1 alone in other AIM7 workloads (at 8 nodes,\nhyperthreading off).\n\n+--------------+---------------+----------------+-----------------+\n| Workload | mean % change | mean % change | mean % change |\n| | 10-100 users | 200-1000 users | 1100-2000 users |\n+--------------+---------------+----------------+-----------------+\n| alltests | 0.0% | -0.8% | +0.6% |\n| five_sec | -0.3% | +0.8% | +0.8% |\n| high_systime | +0.4% | +2.4% | +2.1% |\n| new_fserver | +0.1% | +14.1% | +34.2% |\n| shared | -0.5% | -0.3% | -0.4% |\n| short | -1.7% | -9.8% | -8.3% |\n+--------------+---------------+----------------+-----------------+\n\nThe short workload is the only one that shows a decline in\nperformance probably due to the spinner locking and queuing\noverhead.\n\nSigned-off-by: Waiman Long \u003cWaiman.Long@hp.com\u003e\nReviewed-by: Davidlohr Bueso \u003cdavidlohr.bueso@hp.com\u003e\nAcked-by: Rik van Riel \u003criel@redhat.com\u003e\nCc: Linus Torvalds \u003ctorvalds@linux-foundation.org\u003e\nCc: Andrew Morton \u003cakpm@linux-foundation.org\u003e\nCc: Peter Zijlstra \u003ca.p.zijlstra@chello.nl\u003e\nCc: Thomas Gleixner \u003ctglx@linutronix.de\u003e\nCc: Chandramouleeswaran Aswin \u003caswin@hp.com\u003e\nCc: Norton Scott J \u003cscott.norton@hp.com\u003e\nCc: Paul E. McKenney \u003cpaulmck@linux.vnet.ibm.com\u003e\nCc: David Howells \u003cdhowells@redhat.com\u003e\nCc: Dave Jones \u003cdavej@redhat.com\u003e\nCc: Clark Williams \u003cwilliams@redhat.com\u003e\nCc: Peter Zijlstra \u003cpeterz@infradead.org\u003e\nLink: http://lkml.kernel.org/r/1366226594-5506-4-git-send-email-Waiman.Long@hp.com\nSigned-off-by: Ingo Molnar \u003cmingo@kernel.org\u003e\n", "tree_diff": [ { "type": "modify", "old_id": "9121595a8ebfe14a1d2330c79454f4b53a088669", "old_mode": 33188, "old_path": "include/linux/mutex.h", "new_id": "433da8a1a42642ce33596b536f2dcaad4427b3ac", "new_mode": 33188, "new_path": "include/linux/mutex.h" }, { "type": "modify", "old_id": "70ebd855d9e83bb7c6a08d9f89ade4dbef1fdd33", "old_mode": 33188, "old_path": "kernel/mutex.c", "new_id": "1dbd4210baef17a6ea28354c26737009077eefce", "new_mode": 33188, "new_path": "kernel/mutex.c" } ] }