)]}'
{
  "commit": "dfb4f09609827301740ef0a11b37530d190f1681",
  "tree": "eb4d13d8699cf01abada9f45e1670cc601fb4b00",
  "parents": [
    "484f51f820199ab3e0ef15d08f1b6be20f53bf39"
  ],
  "author": {
    "name": "Christoph Lameter",
    "email": "clameter@sgi.com",
    "time": "Tue Oct 16 01:26:05 2007 -0700"
  },
  "committer": {
    "name": "Linus Torvalds",
    "email": "torvalds@woody.linux-foundation.org",
    "time": "Tue Oct 16 09:43:01 2007 -0700"
  },
  "message": "SLUB: Avoid page struct cacheline bouncing due to remote frees to cpu slab\n\nA remote free may access the same page struct that also contains the lockless\nfreelist for the cpu slab. If objects have a short lifetime and are freed by\na different processor then remote frees back to the slab from which we are\ncurrently allocating are frequent. The cacheline with the page struct needs\nto be repeately acquired in exclusive mode by both the allocating thread and\nthe freeing thread. If this is frequent enough then performance will suffer\nbecause of cacheline bouncing.\n\nThis patchset puts the lockless_freelist pointer in its own cacheline. In\norder to make that happen we introduce a per cpu structure called\nkmem_cache_cpu.\n\nInstead of keeping an array of pointers to page structs we now keep an array\nto a per cpu structure that--among other things--contains the pointer to the\nlockless freelist. The freeing thread can then keep possession of exclusive\naccess to the page struct cacheline while the allocating thread keeps its\nexclusive access to the cacheline containing the per cpu structure.\n\nThis works as long as the allocating cpu is able to service its request\nfrom the lockless freelist. If the lockless freelist runs empty then the\nallocating thread needs to acquire exclusive access to the cacheline with\nthe page struct lock the slab.\n\nThe allocating thread will then check if new objects were freed to the per\ncpu slab. If so it will keep the slab as the cpu slab and continue with the\nrecently remote freed objects. So the allocating thread can take a series\nof just freed remote pages and dish them out again. Ideally allocations\ncould be just recycling objects in the same slab this way which will lead\nto an ideal allocation / remote free pattern.\n\nThe number of objects that can be handled in this way is limited by the\ncapacity of one slab. Increasing slab size via slub_min_objects/\nslub_max_order may increase the number of objects and therefore performance.\n\nIf the allocating thread runs out of objects and finds that no objects were\nput back by the remote processor then it will retrieve a new slab (from the\npartial lists or from the page allocator) and start with a whole\nnew set of objects while the remote thread may still be freeing objects to\nthe old cpu slab. This may then repeat until the new slab is also exhausted.\nIf remote freeing has freed objects in the earlier slab then that earlier\nslab will now be on the partial freelist and the allocating thread will\npick that slab next for allocation. So the loop is extended. However,\nboth threads need to take the list_lock to make the swizzling via\nthe partial list happen.\n\nIt is likely that this kind of scheme will keep the objects being passed\naround to a small set that can be kept in the cpu caches leading to increased\nperformance.\n\nMore code cleanups become possible:\n\n- Instead of passing a cpu we can now pass a kmem_cache_cpu structure around.\n  Allows reducing the number of parameters to various functions.\n- Can define a new node_match() function for NUMA to encapsulate locality\n  checks.\n\nEffect on allocations:\n\nCachelines touched before this patch:\n\n\tWrite:\tpage cache struct and first cacheline of object\n\nCachelines touched after this patch:\n\n\tWrite:\tkmem_cache_cpu cacheline and first cacheline of object\n\tRead: page cache struct (but see later patch that avoids touching\n\t\tthat cacheline)\n\nThe handling when the lockless alloc list runs empty gets to be a bit more\ncomplicated since another cacheline has now to be written to. But that is\nhalfway out of the hot path.\n\nEffect on freeing:\n\nCachelines touched before this patch:\n\n\tWrite: page_struct and first cacheline of object\n\nCachelines touched after this patch depending on how we free:\n\n  Write(to cpu_slab):\tkmem_cache_cpu struct and first cacheline of object\n  Write(to other):\tpage struct and first cacheline of object\n\n  Read(to cpu_slab):\tpage struct to id slab etc. (but see later patch that\n  \t\t\tavoids touching the page struct on free)\n  Read(to other):\tcpu local kmem_cache_cpu struct to verify its not\n  \t\t\tthe cpu slab.\n\nSummary:\n\nPro:\n\t- Distinct cachelines so that concurrent remote frees and local\n\t  allocs on a cpuslab can occur without cacheline bouncing.\n\t- Avoids potential bouncing cachelines because of neighboring\n\t  per cpu pointer updates in kmem_cache\u0027s cpu_slab structure since\n\t  it now grows to a cacheline (Therefore remove the comment\n\t  that talks about that concern).\n\nCons:\n\t- Freeing objects now requires the reading of one additional\n\t  cacheline. That can be mitigated for some cases by the following\n\t  patches but its not possible to completely eliminate these\n\t  references.\n\n\t- Memory usage grows slightly.\n\n\tThe size of each per cpu object is blown up from one word\n\t(pointing to the page_struct) to one cacheline with various data.\n\tSo this is NR_CPUS*NR_SLABS*L1_BYTES more memory use. Lets say\n\tNR_SLABS is 100 and a cache line size of 128 then we have just\n\tincreased SLAB metadata requirements by 12.8k per cpu.\n\t(Another later patch reduces these requirements)\n\nSigned-off-by: Christoph Lameter \u003cclameter@sgi.com\u003e\nSigned-off-by: Andrew Morton \u003cakpm@linux-foundation.org\u003e\nSigned-off-by: Linus Torvalds \u003ctorvalds@linux-foundation.org\u003e\n",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "3b361b2906bdb563b6a082b214ff8b9bd0cb1e73",
      "old_mode": 33188,
      "old_path": "include/linux/slub_def.h",
      "new_id": "0a7ae25f7e8de16f7186e6b71255d5cad3ba3354",
      "new_mode": 33188,
      "new_path": "include/linux/slub_def.h"
    },
    {
      "type": "modify",
      "old_id": "a90c4ffc95765e3c16965473d3b51b262ea91f0b",
      "old_mode": 33188,
      "old_path": "mm/slub.c",
      "new_id": "4b8037f14fce317168a52bc328fcd21f71aded87",
      "new_mode": 33188,
      "new_path": "mm/slub.c"
    }
  ]
}