)]}' { "commit": "71e3aac0724ffe8918992d76acfe3aad7d8724a5", "tree": "4ff96e1fc3e53bc9d25b859bf7e5bdbab8f1b25a", "parents": [ "5c3240d92e29ae7bfb9cb58a9b37e80ab40894ff" ], "author": { "name": "Andrea Arcangeli", "email": "aarcange@redhat.com", "time": "Thu Jan 13 15:46:52 2011 -0800" }, "committer": { "name": "Linus Torvalds", "email": "torvalds@linux-foundation.org", "time": "Thu Jan 13 17:32:42 2011 -0800" }, "message": "thp: transparent hugepage core\n\nLately I\u0027ve been working to make KVM use hugepages transparently without\nthe usual restrictions of hugetlbfs. Some of the restrictions I\u0027d like to\nsee removed:\n\n1) hugepages have to be swappable or the guest physical memory remains\n locked in RAM and can\u0027t be paged out to swap\n\n2) if a hugepage allocation fails, regular pages should be allocated\n instead and mixed in the same vma without any failure and without\n userland noticing\n\n3) if some task quits and more hugepages become available in the\n buddy, guest physical memory backed by regular pages should be\n relocated on hugepages automatically in regions under\n madvise(MADV_HUGEPAGE) (ideally event driven by waking up the\n kernel deamon if the order\u003dHPAGE_PMD_SHIFT-PAGE_SHIFT list becomes\n not null)\n\n4) avoidance of reservation and maximization of use of hugepages whenever\n possible. Reservation (needed to avoid runtime fatal faliures) may be ok for\n 1 machine with 1 database with 1 database cache with 1 database cache size\n known at boot time. It\u0027s definitely not feasible with a virtualization\n hypervisor usage like RHEV-H that runs an unknown number of virtual machines\n with an unknown size of each virtual machine with an unknown amount of\n pagecache that could be potentially useful in the host for guest not using\n O_DIRECT (aka cache\u003doff).\n\nhugepages in the virtualization hypervisor (and also in the guest!) are\nmuch more important than in a regular host not using virtualization,\nbecasue with NPT/EPT they decrease the tlb-miss cacheline accesses from 24\nto 19 in case only the hypervisor uses transparent hugepages, and they\ndecrease the tlb-miss cacheline accesses from 19 to 15 in case both the\nlinux hypervisor and the linux guest both uses this patch (though the\nguest will limit the addition speedup to anonymous regions only for\nnow...). Even more important is that the tlb miss handler is much slower\non a NPT/EPT guest than for a regular shadow paging or no-virtualization\nscenario. So maximizing the amount of virtual memory cached by the TLB\npays off significantly more with NPT/EPT than without (even if there would\nbe no significant speedup in the tlb-miss runtime).\n\nThe first (and more tedious) part of this work requires allowing the VM to\nhandle anonymous hugepages mixed with regular pages transparently on\nregular anonymous vmas. This is what this patch tries to achieve in the\nleast intrusive possible way. We want hugepages and hugetlb to be used in\na way so that all applications can benefit without changes (as usual we\nleverage the KVM virtualization design: by improving the Linux VM at\nlarge, KVM gets the performance boost too).\n\nThe most important design choice is: always fallback to 4k allocation if\nthe hugepage allocation fails! This is the _very_ opposite of some large\npagecache patches that failed with -EIO back then if a 64k (or similar)\nallocation failed...\n\nSecond important decision (to reduce the impact of the feature on the\nexisting pagetable handling code) is that at any time we can split an\nhugepage into 512 regular pages and it has to be done with an operation\nthat can\u0027t fail. This way the reliability of the swapping isn\u0027t decreased\n(no need to allocate memory when we are short on memory to swap) and it\u0027s\ntrivial to plug a split_huge_page* one-liner where needed without\npolluting the VM. Over time we can teach mprotect, mremap and friends to\nhandle pmd_trans_huge natively without calling split_huge_page*. The fact\nit can\u0027t fail isn\u0027t just for swap: if split_huge_page would return -ENOMEM\n(instead of the current void) we\u0027d need to rollback the mprotect from the\nmiddle of it (ideally including undoing the split_vma) which would be a\nbig change and in the very wrong direction (it\u0027d likely be simpler not to\ncall split_huge_page at all and to teach mprotect and friends to handle\nhugepages instead of rolling them back from the middle). In short the\nvery value of split_huge_page is that it can\u0027t fail.\n\nThe collapsing and madvise(MADV_HUGEPAGE) part will remain separated and\nincremental and it\u0027ll just be an \"harmless\" addition later if this initial\npart is agreed upon. It also should be noted that locking-wise replacing\nregular pages with hugepages is going to be very easy if compared to what\nI\u0027m doing below in split_huge_page, as it will only happen when\npage_count(page) matches page_mapcount(page) if we can take the PG_lock\nand mmap_sem in write mode. collapse_huge_page will be a \"best effort\"\nthat (unlike split_huge_page) can fail at the minimal sign of trouble and\nwe can try again later. collapse_huge_page will be similar to how KSM\nworks and the madvise(MADV_HUGEPAGE) will work similar to\nmadvise(MADV_MERGEABLE).\n\nThe default I like is that transparent hugepages are used at page fault\ntime. This can be changed with\n/sys/kernel/mm/transparent_hugepage/enabled. The control knob can be set\nto three values \"always\", \"madvise\", \"never\" which mean respectively that\nhugepages are always used, or only inside madvise(MADV_HUGEPAGE) regions,\nor never used. /sys/kernel/mm/transparent_hugepage/defrag instead\ncontrols if the hugepage allocation should defrag memory aggressively\n\"always\", only inside \"madvise\" regions, or \"never\".\n\nThe pmd_trans_splitting/pmd_trans_huge locking is very solid. The\nput_page (from get_user_page users that can\u0027t use mmu notifier like\nO_DIRECT) that runs against a __split_huge_page_refcount instead was a\npain to serialize in a way that would result always in a coherent page\ncount for both tail and head. I think my locking solution with a\ncompound_lock taken only after the page_first is valid and is still a\nPageHead should be safe but it surely needs review from SMP race point of\nview. In short there is no current existing way to serialize the O_DIRECT\nfinal put_page against split_huge_page_refcount so I had to invent a new\none (O_DIRECT loses knowledge on the mapping status by the time gup_fast\nreturns so...). And I didn\u0027t want to impact all gup/gup_fast users for\nnow, maybe if we change the gup interface substantially we can avoid this\nlocking, I admit I didn\u0027t think too much about it because changing the gup\nunpinning interface would be invasive.\n\nIf we ignored O_DIRECT we could stick to the existing compound refcounting\ncode, by simply adding a get_user_pages_fast_flags(foll_flags) where KVM\n(and any other mmu notifier user) would call it without FOLL_GET (and if\nFOLL_GET isn\u0027t set we\u0027d just BUG_ON if nobody registered itself in the\ncurrent task mmu notifier list yet). But O_DIRECT is fundamental for\ndecent performance of virtualized I/O on fast storage so we can\u0027t avoid it\nto solve the race of put_page against split_huge_page_refcount to achieve\na complete hugepage feature for KVM.\n\nSwap and oom works fine (well just like with regular pages ;). MMU\nnotifier is handled transparently too, with the exception of the young bit\non the pmd, that didn\u0027t have a range check but I think KVM will be fine\nbecause the whole point of hugepages is that EPT/NPT will also use a huge\npmd when they notice gup returns pages with PageCompound set, so they\nwon\u0027t care of a range and there\u0027s just the pmd young bit to check in that\ncase.\n\nNOTE: in some cases if the L2 cache is small, this may slowdown and waste\nmemory during COWs because 4M of memory are accessed in a single fault\ninstead of 8k (the payoff is that after COW the program can run faster).\nSo we might want to switch the copy_huge_page (and clear_huge_page too) to\nnot temporal stores. I also extensively researched ways to avoid this\ncache trashing with a full prefault logic that would cow in 8k/16k/32k/64k\nup to 1M (I can send those patches that fully implemented prefault) but I\nconcluded they\u0027re not worth it and they add an huge additional complexity\nand they remove all tlb benefits until the full hugepage has been faulted\nin, to save a little bit of memory and some cache during app startup, but\nthey still don\u0027t improve substantially the cache-trashing during startup\nif the prefault happens in \u003e4k chunks. One reason is that those 4k pte\nentries copied are still mapped on a perfectly cache-colored hugepage, so\nthe trashing is the worst one can generate in those copies (cow of 4k page\ncopies aren\u0027t so well colored so they trashes less, but again this results\nin software running faster after the page fault). Those prefault patches\nallowed things like a pte where post-cow pages were local 4k regular anon\npages and the not-yet-cowed pte entries were pointing in the middle of\nsome hugepage mapped read-only. If it doesn\u0027t payoff substantially with\ntodays hardware it will payoff even less in the future with larger l2\ncaches, and the prefault logic would blot the VM a lot. If one is\nemebdded transparent_hugepage can be disabled during boot with sysfs or\nwith the boot commandline parameter transparent_hugepage\u003d0 (or\ntransparent_hugepage\u003d2 to restrict hugepages inside madvise regions) that\nwill ensure not a single hugepage is allocated at boot time. It is simple\nenough to just disable transparent hugepage globally and let transparent\nhugepages be allocated selectively by applications in the MADV_HUGEPAGE\nregion (both at page fault time, and if enabled with the\ncollapse_huge_page too through the kernel daemon).\n\nThis patch supports only hugepages mapped in the pmd, archs that have\nsmaller hugepages will not fit in this patch alone. Also some archs like\npower have certain tlb limits that prevents mixing different page size in\nthe same regions so they will not fit in this framework that requires\n\"graceful fallback\" to basic PAGE_SIZE in case of physical memory\nfragmentation. hugetlbfs remains a perfect fit for those because its\nsoftware limits happen to match the hardware limits. hugetlbfs also\nremains a perfect fit for hugepage sizes like 1GByte that cannot be hoped\nto be found not fragmented after a certain system uptime and that would be\nvery expensive to defragment with relocation, so requiring reservation.\nhugetlbfs is the \"reservation way\", the point of transparent hugepages is\nnot to have any reservation at all and maximizing the use of cache and\nhugepages at all times automatically.\n\nSome performance result:\n\nvmx andrea # LD_PRELOAD\u003d/usr/lib64/libhugetlbfs.so HUGETLB_MORECORE\u003dyes HUGETLB_PATH\u003d/mnt/huge/ ./largep\nages3\nmemset page fault 1566023\nmemset tlb miss 453854\nmemset second tlb miss 453321\nrandom access tlb miss 41635\nrandom access second tlb miss 41658\nvmx andrea # LD_PRELOAD\u003d/usr/lib64/libhugetlbfs.so HUGETLB_MORECORE\u003dyes HUGETLB_PATH\u003d/mnt/huge/ ./largepages3\nmemset page fault 1566471\nmemset tlb miss 453375\nmemset second tlb miss 453320\nrandom access tlb miss 41636\nrandom access second tlb miss 41637\nvmx andrea # ./largepages3\nmemset page fault 1566642\nmemset tlb miss 453417\nmemset second tlb miss 453313\nrandom access tlb miss 41630\nrandom access second tlb miss 41647\nvmx andrea # ./largepages3\nmemset page fault 1566872\nmemset tlb miss 453418\nmemset second tlb miss 453315\nrandom access tlb miss 41618\nrandom access second tlb miss 41659\nvmx andrea # echo 0 \u003e /proc/sys/vm/transparent_hugepage\nvmx andrea # ./largepages3\nmemset page fault 2182476\nmemset tlb miss 460305\nmemset second tlb miss 460179\nrandom access tlb miss 44483\nrandom access second tlb miss 44186\nvmx andrea # ./largepages3\nmemset page fault 2182791\nmemset tlb miss 460742\nmemset second tlb miss 459962\nrandom access tlb miss 43981\nrandom access second tlb miss 43988\n\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n#include \u003cstdio.h\u003e\n#include \u003cstdlib.h\u003e\n#include \u003cstring.h\u003e\n#include \u003csys/time.h\u003e\n\n#define SIZE (3UL*1024*1024*1024)\n\nint main()\n{\n\tchar *p \u003d malloc(SIZE), *p2;\n\tstruct timeval before, after;\n\n\tgettimeofday(\u0026before, NULL);\n\tmemset(p, 0, SIZE);\n\tgettimeofday(\u0026after, NULL);\n\tprintf(\"memset page fault %Lu\\n\",\n\t (after.tv_sec-before.tv_sec)*1000000UL +\n\t after.tv_usec-before.tv_usec);\n\n\tgettimeofday(\u0026before, NULL);\n\tmemset(p, 0, SIZE);\n\tgettimeofday(\u0026after, NULL);\n\tprintf(\"memset tlb miss %Lu\\n\",\n\t (after.tv_sec-before.tv_sec)*1000000UL +\n\t after.tv_usec-before.tv_usec);\n\n\tgettimeofday(\u0026before, NULL);\n\tmemset(p, 0, SIZE);\n\tgettimeofday(\u0026after, NULL);\n\tprintf(\"memset second tlb miss %Lu\\n\",\n\t (after.tv_sec-before.tv_sec)*1000000UL +\n\t after.tv_usec-before.tv_usec);\n\n\tgettimeofday(\u0026before, NULL);\n\tfor (p2 \u003d p; p2 \u003c p+SIZE; p2 +\u003d 4096)\n\t\t*p2 \u003d 0;\n\tgettimeofday(\u0026after, NULL);\n\tprintf(\"random access tlb miss %Lu\\n\",\n\t (after.tv_sec-before.tv_sec)*1000000UL +\n\t after.tv_usec-before.tv_usec);\n\n\tgettimeofday(\u0026before, NULL);\n\tfor (p2 \u003d p; p2 \u003c p+SIZE; p2 +\u003d 4096)\n\t\t*p2 \u003d 0;\n\tgettimeofday(\u0026after, NULL);\n\tprintf(\"random access second tlb miss %Lu\\n\",\n\t (after.tv_sec-before.tv_sec)*1000000UL +\n\t after.tv_usec-before.tv_usec);\n\n\treturn 0;\n}\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nSigned-off-by: Andrea Arcangeli \u003caarcange@redhat.com\u003e\nAcked-by: Rik van Riel \u003criel@redhat.com\u003e\nSigned-off-by: Johannes Weiner \u003channes@cmpxchg.org\u003e\nSigned-off-by: Andrew Morton \u003cakpm@linux-foundation.org\u003e\nSigned-off-by: Linus Torvalds \u003ctorvalds@linux-foundation.org\u003e\n", "tree_diff": [ { "type": "modify", "old_id": "1fb61a74b2e1e300a09df16f7f1fc504dde95ec0", "old_mode": 33188, "old_path": "arch/x86/include/asm/pgtable_64.h", "new_id": "b2df039a411973290663fb11587f83993ea96acf", "new_mode": 33188, "new_path": "arch/x86/include/asm/pgtable_64.h" }, { "type": "modify", "old_id": "49d2356bb82d582a07da80c82e0c09f2e1ff1e50", "old_mode": 33188, "old_path": "include/linux/gfp.h", "new_id": "d95082cc6f4a44065eade40283184254ce609cd7", "new_mode": 33188, "new_path": "include/linux/gfp.h" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "9301824c7491ae72f9ac49826eda850ecc1804bd", "new_mode": 33188, "new_path": "include/linux/huge_mm.h" }, { "type": "modify", "old_id": "cc6ab1038f6fa038e9e72a39a92df7306bc7ad60", "old_mode": 33188, "old_path": "include/linux/mm.h", "new_id": "78adec4ba9f4466bfc3b0aa92a8455ff20ee8424", "new_mode": 33188, "new_path": "include/linux/mm.h" }, { "type": "modify", "old_id": "8835b877b8dbefdd57bf8ba4b52fcc25b47662ff", "old_mode": 33188, "old_path": "include/linux/mm_inline.h", "new_id": "650f31eabdb1ac3a1b24f5ee7f67008237d8e542", "new_mode": 33188, "new_path": "include/linux/mm_inline.h" }, { "type": "modify", "old_id": "907f1605926b14cf550927f58d10c685dc668297", "old_mode": 33188, "old_path": "include/linux/page-flags.h", "new_id": "4ca1241ef94e0796ee1c092eea25a5514c873821", "new_mode": 33188, "new_path": "include/linux/page-flags.h" }, { "type": "modify", "old_id": "bb83c0da207197122c9ef3988cf0b3d843404cd3", "old_mode": 33188, "old_path": "include/linux/rmap.h", "new_id": "e9fd04ca1e518f9f6eeb0d430443a64e667722e4", "new_mode": 33188, "new_path": "include/linux/rmap.h" }, { "type": "modify", "old_id": "eba53e71d2ccfa5f678cc8063dd55c0d56ef1dec", "old_mode": 33188, "old_path": "include/linux/swap.h", "new_id": "4d559325d919fb5fa194d6a7e61711ae52e01192", "new_mode": 33188, "new_path": "include/linux/swap.h" }, { "type": "modify", "old_id": "380772a9ccb8843227e93e4e46c015dc6c7c3491", "old_mode": 33188, "old_path": "mm/Makefile", "new_id": "2b1b575ae712c2f0f15976a23ec5e4dc53f0897b", "new_mode": 33188, "new_path": "mm/Makefile" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "0c1e8f939f7cffe05215b42da396cd00680e7aa8", "new_mode": 33188, "new_path": "mm/huge_memory.c" }, { "type": "modify", "old_id": "bd4f581f624a7059d304a6ac76982c6a48480524", "old_mode": 33188, "old_path": "mm/internal.h", "new_id": "69488205723d1666d3265c60279d9dfd6bcf67e0", "new_mode": 33188, "new_path": "mm/internal.h" }, { "type": "modify", "old_id": "60e1c68d821845f4bf75fe23e8d070ccc80eb360", "old_mode": 33188, "old_path": "mm/memory.c", "new_id": "c50a195041ecb1545a49732c724f4720c5f0bcb9", "new_mode": 33188, "new_path": "mm/memory.c" }, { "type": "modify", "old_id": "a3197a8a295b8ea515c6aa85c11512cb6d2bb06e", "old_mode": 33188, "old_path": "mm/rmap.c", "new_id": "e41375a6b029fe25743c680930c6c607f5a1b86c", "new_mode": 33188, "new_path": "mm/rmap.c" }, { "type": "modify", "old_id": "e0eeef940886084fb11297ed4580716a59425c27", "old_mode": 33188, "old_path": "mm/swap.c", "new_id": "c02f93611a849e6c79bdbae2e13ee6e4cff6eb37", "new_mode": 33188, "new_path": "mm/swap.c" } ] }