• Steven Rostedt's avatar
    ftrace: Remove memory barriers from NMI code when not needed · 0c54dd34
    Steven Rostedt authored
    The code in stop_machine that modifies the kernel text has a bit
    of logic to handle the case of NMIs. stop_machine does not prevent
    NMIs from executing, and if an NMI were to trigger on another CPU
    as the modifying CPU is changing the NMI text, a GPF could result.
    To prevent the GPF, the NMI calls ftrace_nmi_enter() which may
    modify the code first, then any other NMIs will just change the
    text to the same content which will do no harm. The code that
    stop_machine called must wait for NMIs to finish while it changes
    each location in the kernel. That code may also change the text
    to what the NMI changed it to. The key is that the text will never
    change content while another CPU is executing it.
    To make the above work, the call to ftrace_nmi_enter() must also
    do a smp_mb() as well as atomic_inc().  But for applications like
    perf that require a high number of NMIs for profiling, this can have
    a dramatic effect on the system. Not only is it doing a full memory
    barrier on both nmi_enter() as well as nmi_exit() it is also
    modifying a global variable with an atomic operation. This kills
    performance on large SMP machines.
    Since the memory barriers are only needed when ftrace is in the
    process of modifying the text (which is seldom), this patch
    adds a "modifying_code" variable that gets set before stop machine
    is executed and cleared afterwards.
    The NMIs will check this variable and store it in a per CPU
    "save_modifying_code" variable that it will use to check if it
    needs to do the memory barriers and atomic dec on NMI exit.
    Acked-by: default avatarPeter Zijlstra <peterz@infradead.org>
    Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
ftrace.c 12.6 KB