x86-64, rwsem: Avoid store forwarding hazard in __downgrade_write
x86-64, rwsem: Avoid store forwarding hazard in __downgrade_write

The Intel Architecture Optimization Reference Manual states that a short
load that follows a long store to the same object will suffer a store
forwading penalty, particularly if the two accesses use different addresses.
Trivially, a long load that follows a short store will also suffer a penalty.

__downgrade_write() in rwsem incurs both penalties:  the increment operation
will not be able to reuse a recently-loaded rwsem value, and its result will
not be reused by any recently-following rwsem operation.

A comment in the code states that this is because 64-bit immediates are
special and expensive; but while they are slightly special (only a single
instruction allows them), they aren't expensive: a test shows that two loops,
one loading a 32-bit immediate and one loading a 64-bit immediate, both take
1.5 cycles per iteration.

Fix this by changing __downgrade_write to use the same add instruction on
i386 and on x86_64, so that it uses the same operand size as all the other
rwsem functions.

Signed-off-by: Avi Kivity <8f920f22884d6fea9df883843c4a8095a2e5ac6f@redhat.com>
LKML-Reference: <1266049992-17419-1-git-send-email-8f920f22884d6fea9df883843c4a8095a2e5ac6f@redhat.com>
Signed-off-by: H. Peter Anvin <8a453bad9912ffe59bc0f0b8abe03df9be19379e@zytor.com>

KristFoundation/Programs,TeamVee-Kanas/android_kernel_samsung_kanas,TeamVee-Kanas/android_kernel_samsung_kanas,KristFoundation/Programs,TeamVee-Kanas/android_kernel_samsung_kanas,TeamVee-Kanas/android_kernel_samsung_kanas,KristFoundation/Programs,KristFoundation/Programs,KristFoundation/Programs,KristFoundation/Programs,TeamVee-Kanas/android_kernel_samsung_kanas