x86: add missed pgtable_pmd_page_ctor/dtor calls for preallocated pmds
x86: add missed pgtable_pmd_page_ctor/dtor calls for preallocated pmds

In split page table lock case, we embed spinlock_t into struct page.
For obvious reason, we don't want to increase size of struct page if
spinlock_t is too big, like with DEBUG_SPINLOCK or DEBUG_LOCK_ALLOC or
on -rt kernel.  So we disable split page table lock, if spinlock_t is
too big.

This patchset allows to allocate the lock dynamically if spinlock_t is
big.  In this page->ptl is used to store pointer to spinlock instead of
spinlock itself.  It costs additional cache line for indirect access,
but fix page fault scalability for multi-threaded applications.

LOCK_STAT depends on DEBUG_SPINLOCK, so on current kernel enabling
LOCK_STAT to analyse scalability issues breaks scalability.  ;)

The patchset mostly fixes this.  Results for ./thp_memscale -c 80 -b 512M
on 4-socket machine:

baseline, no CONFIG_LOCK_STAT:	9.115460703 seconds time elapsed
baseline, CONFIG_LOCK_STAT=y:	53.890567123 seconds time elapsed
patched, no CONFIG_LOCK_STAT:	8.852250368 seconds time elapsed
patched, CONFIG_LOCK_STAT=y:	11.069770759 seconds time elapsed

Patch count is scary, but most of them trivial. Overview:

 Patches 1-4	Few bug fixes. No dependencies to other patches.
		Probably should applied as soon as possible.

 Patch 5	Changes signature of pgtable_page_ctor(). We will use it
		for dynamic lock allocation, so it can fail.

 Patches 6-8	Add missing constructor/destructor calls on few archs.
		It's fixes NR_PAGETABLE accounting and prepare to use
		split ptl.

 Patches 9-33	Add pgtable_page_ctor() fail handling to all archs.

 Patches 34	Finally adds support of dynamically-allocated page->pte.
		Also contains documentation for split page table lock.

This patch (of 34):

I've missed that we preallocate few pmds on pgd_alloc() if X86_PAE
enabled.  Let's add missed constructor/destructor calls.

I haven't noticed it during testing since prep_new_page() clears
page->mapping and therefore page->ptl.  It's effectively equal to
spin_lock_init(&page->ptl).

Signed-off-by: Kirill A. Shutemov <eec72676ad9a62f2727dfed6d270d98c0d02adec@linux.intel.com>
Acked-by: Ingo Molnar <9dbbbf0688fedc85ad4da37637f1a64b8c718ee2@kernel.org>
Cc: "H. Peter Anvin" <8a453bad9912ffe59bc0f0b8abe03df9be19379e@zytor.com>
Cc: "James E.J. Bottomley" <78abc16a0671a4bb8f94993f5c9afbb002942fb1@parisc-linux.org>
Cc: "Kirill A. Shutemov" <eec72676ad9a62f2727dfed6d270d98c0d02adec@linux.intel.com>
Cc: Benjamin Herrenschmidt <a7089bb6e7e92505d88aaff006cbdd60cc9120b6@kernel.crashing.org>
Cc: Catalin Marinas <15ce75b290ebaf27c3f9fd73ab848685ed3d8261@arm.com>
Cc: Chen Liqin <0e9400de518c52242a076b36c42b197d35a2652e@sunplusct.com>
Cc: Chris Metcalf <074881f6f5da4d3b5278870ff234d1077ee622c0@tilera.com>
Cc: Chris Zankel <711c73f64afdce07b7e38039a96d2224209e9a6c@zankel.net>
Cc: Christoph Lameter <ef3ecccf258fa062c5c6521a4887d40541963af7@linux.com>
Cc: David Howells <ebac1d06c1688626821bb0e574a037a7a5354e49@redhat.com>
Cc: David S. Miller <fe08d3c717adf2ae63592e4c9aec6e3e404d8e3e@davemloft.net>
Cc: Fenghua Yu <94fe9a7ec8ff966c1952fe98e39b5ae3a7477695@intel.com>
Cc: Geert Uytterhoeven <0da414d9d963da4039c2a0525b1844228075aa58@linux-m68k.org>
Cc: Grant Likely <9069e6f5a2b566e2674a0ba1e2bf39c12c195fad@linaro.org>
Cc: Guan Xuetao <e3695cf74a76bd4528efcb62d7670a686738ecc1@mprc.pku.edu.cn>
Cc: Haavard Skinnemoen <de7418319212d7eab32260a76732c1f83e514852@gmail.com>
Cc: Hans-Christian Egtvedt <11ce19a6fc3bb47abce595a29a6ea4cfed5efa4c@samfundet.no>
Cc: Heiko Carstens <8dcf0f69152f32f23184f83357a3731522e56b9c@de.ibm.com>
Cc: Helge Deller <5b1e7895819cf7d6bc77091b09bbfd6d330a4e74@gmx.de>
Cc: Hirokazu Takata <cf11bbe1b6d3d076b415c503f9127c50075c83bd@linux-m32r.org>
Cc: Ivan Kokshaysky <ec437ab60af367f0fec0f6a3cd47f9cbad7b487f@jurassic.park.msu.ru>
Cc: James Hogan <547fa85bb3888a0222eed8ec39e36708605c6d53@imgtec.com>
Cc: Jeff Dike <f9b6309e85863eba5d28138b6a0f35841ce976d0@addtoit.com>
Cc: Jesper Nilsson <987a7dbc972893e93e5578401bedc3a0f2ccb5e3@axis.com>
Cc: Jonas Bonn <35a2c6fae61f8077aab61faa4019722abf05093c@southpole.se>
Cc: Koichi Yasutake <9e1743478df52b5f4b5518b23325bb811b7ea24b@jp.panasonic.com>
Cc: Lennox Wu <dd55f9cbb020f98a777870a31286e3419ae6ac5e@gmail.com>
Cc: Martin Schwidefsky <52616596d8f5df0d597e85ab515377f92f939c68@de.ibm.com>
Cc: Matt Turner <789b315743a28dd066a4ba1459c35951c291d8a6@gmail.com>
Cc: Max Filippov <8c5ea195856807f4f02a6e0749c87ea6b7375f38@gmail.com>
Cc: Michal Simek <3a191dd2be46e826fee926d532b136ed65cb5318@monstr.eu>
Cc: Mikael Starvik <6c4d7924b92058127b0499322f0d64063fb5d591@axis.com>
Cc: Paul Mackerras <19a0ba370c443ba08d20b5061586430ab449ee8c@samba.org>
Cc: Paul Mundt <38b52dbb5f0b63d149982b6c5de788ec93a89032@linux-sh.org>
Cc: Peter Zijlstra <3fddac958924aef220f202ca567388ddab3f14a8@infradead.org>
Cc: Ralf Baechle <92f48d309cda194c8eda36aa8f9ae28c488fa208@linux-mips.org>
Cc: Richard Henderson <aea4fd857e97ee7d33f60175ba9388a955b9d916@twiddle.net>
Cc: Richard Kuo <5c405e4535fa35126dd73e0f658fb1523e5f9f8e@codeaurora.org>
Cc: Richard Weinberger <320bca71fc381a4a025636043ca86e734e31cf8b@nod.at>
Cc: Rob Herring <aa00819e304141d872f2e4b1c4be422ea3abbfe1@calxeda.com>
Cc: Russell King <ba324ca7b1c77fc20bb970d5aff6eea9377918a5@arm.linux.org.uk>
Cc: Thomas Gleixner <00e4cf8f46a57000a44449bf9dd8cbbcc209fd2a@linutronix.de>
Cc: Tony Luck <e7984595ec0368ff920a7b3521dc7093683f6f26@intel.com>
Cc: Vineet Gupta <af669d95a4c0de051445d4cafdbb2c5c4a2838be@synopsys.com>
Cc: Will Deacon <b913f13ef92ac157955c954ccaaff7be065faba4@arm.com>
Signed-off-by: Andrew Morton <5c1e68b099950c134891f0b6e179498a8ebe9cf9@linux-foundation.org>
Signed-off-by: Linus Torvalds <69652caca27c8b940640ad396ab71f93cacec34f@linux-foundation.org>

KristFoundation/Programs,KristFoundation/Programs,KristFoundation/Programs,KristFoundation/Programs,KristFoundation/Programs,KristFoundation/Programs