• Peter Zijlstra (Intel)'s avatar
    locking/qspinlock: Optimize for smaller NR_CPUS · 69f9cae9
    Peter Zijlstra (Intel) authored
    When we allow for a max NR_CPUS < 2^14 we can optimize the pending
    wait-acquire and the xchg_tail() operations.
    By growing the pending bit to a byte, we reduce the tail to 16bit.
    This means we can use xchg16 for the tail part and do away with all
    the repeated compxchg() operations.
    This in turn allows us to unconditionally acquire; the locked state
    as observed by the wait loops cannot change. And because both locked
    and pending are now a full byte we can use simple stores for the
    state transition, obviating one atomic operation entirely.
    This optimization is needed to make the qspinlock achieve performance
    parity with ticket spinlock at light load.
    All this is horribly broken on Alpha pre EV56 (and any other arch that
    cannot do single-copy atomic byte stores).
    Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
    Signed-off-by: default avatarWaiman Long <Waiman.Long@hp.com>
    Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
    Cc: Borislav Petkov <bp@alien8.de>
    Cc: Daniel J Blueman <daniel@numascale.com>
    Cc: David Vrabel <david.vrabel@citrix.com>
    Cc: Douglas Hatch <doug.hatch@hp.com>
    Cc: H. Peter Anvin <hpa@zytor.com>
    Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Oleg Nesterov <oleg@redhat.com>
    Cc: Paolo Bonzini <paolo.bonzini@gmail.com>
    Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
    Cc: Rik van Riel <riel@redhat.com>
    Cc: Scott J Norton <scott.norton@hp.com>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: virtualization@lists.linux-foundation.org
    Cc: xen-devel@lists.xenproject.org
    Link: http://lkml.kernel.org/r/1429901803-29771-6-git-send-email-Waiman.Long@hp.com
    Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
qspinlock.c 9.69 KB