1. 17 Jan, 2014 1 commit
  2. 15 Jan, 2014 1 commit
    • Ming Lei's avatar
      lib/percpu_counter.c: fix __percpu_counter_add() · 74e72f89
      Ming Lei authored
      
      
      __percpu_counter_add() may be called in softirq/hardirq handler (such
      as, blk_mq_queue_exit() is typically called in hardirq/softirq handler),
      so we need to call this_cpu_add()(irq safe helper) to update percpu
      counter, otherwise counts may be lost.
      
      This fixes the problem that 'rmmod null_blk' hangs in blk_cleanup_queue()
      because of miscounting of request_queue->mq_usage_counter.
      
      This patch is the v1 of previous one of "lib/percpu_counter.c:
      disable local irq when updating percpu couter", and takes Andrew's
      approach which may be more efficient for ARCHs(x86, s390) that
      have optimized this_cpu_add().
      Signed-off-by: default avatarMing Lei <tom.leiming@gmail.com>
      Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
      Cc: Shaohua Li <shli@fusionio.com>
      Cc: Jens Axboe <axboe@kernel.dk>
      Cc: Fan Du <fan.du@windriver.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      74e72f89
  3. 25 Oct, 2013 1 commit
    • Shaohua Li's avatar
      percpu_counter: make APIs irq safe · 098faf58
      Shaohua Li authored
      
      
      In my usage, sometimes the percpu APIs are called with irq locked,
      sometimes not. lockdep complains there is potential deadlock. Let's
      always use percpucounter lock in irq safe way. There should be no
      performance penality, as all those are slow code path.
      
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarShaohua Li <shli@fusionio.com>
      Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
      098faf58
  4. 14 Jul, 2013 1 commit
    • Paul Gortmaker's avatar
      kernel: delete __cpuinit usage from all core kernel files · 0db0628d
      Paul Gortmaker authored
      The __cpuinit type of throwaway sections might have made sense
      some time ago when RAM was more constrained, but now the savings
      do not offset the cost and complications.  For example, the fix in
      commit 5e427ec2 ("x86: Fix bit corruption at CPU resume time")
      is a good example of the nasty type of bugs that can be created
      with improper use of the various __init prefixes.
      
      After a discussion on LKML[1] it was decided that cpuinit should go
      the way of devinit and be phased out.  Once all the users are gone,
      we can then finally remove the macros themselves from linux/init.h.
      
      This removes all the uses of the __cpuinit macros from C files in
      the core kernel directories (kernel, init, lib, mm, and include)
      that don't really have a specific maintainer.
      
      [1] https://lkml.org/lkml/2013/5/20/589
      
      Signed-off-by: default avatarPaul Gortmaker <paul.gortmaker@windriver.com>
      0db0628d
  5. 03 Jul, 2013 1 commit
  6. 31 Jul, 2012 1 commit
  7. 01 Nov, 2011 1 commit
  8. 13 Sep, 2011 1 commit
  9. 17 Dec, 2010 1 commit
    • Christoph Lameter's avatar
      percpucounter: Optimize __percpu_counter_add a bit through the use of this_cpu() options. · 819a72af
      Christoph Lameter authored
      
      
      The this_cpu_* options can be used to optimize __percpu_counter_add a bit. Avoids
      some address arithmetic and saves 12 bytes.
      
      Before:
      
      
      00000000000001d3 <__percpu_counter_add>:
       1d3:	55                   	push   %rbp
       1d4:	48 89 e5             	mov    %rsp,%rbp
       1d7:	41 55                	push   %r13
       1d9:	41 54                	push   %r12
       1db:	53                   	push   %rbx
       1dc:	48 89 fb             	mov    %rdi,%rbx
       1df:	48 83 ec 08          	sub    $0x8,%rsp
       1e3:	4c 8b 67 30          	mov    0x30(%rdi),%r12
       1e7:	65 4c 03 24 25 00 00 	add    %gs:0x0,%r12
       1ee:	00 00
       1f0:	4d 63 2c 24          	movslq (%r12),%r13
       1f4:	48 63 c2             	movslq %edx,%rax
       1f7:	49 01 f5             	add    %rsi,%r13
       1fa:	49 39 c5             	cmp    %rax,%r13
       1fd:	7d 0a                	jge    209 <__percpu_counter_add+0x36>
       1ff:	f7 da                	neg    %edx
       201:	48 63 d2             	movslq %edx,%rdx
       204:	49 39 d5             	cmp    %rdx,%r13
       207:	7f 1e                	jg     227 <__percpu_counter_add+0x54>
       209:	48 89 df             	mov    %rbx,%rdi
       20c:	e8 00 00 00 00       	callq  211 <__percpu_counter_add+0x3e>
       211:	4c 01 6b 18          	add    %r13,0x18(%rbx)
       215:	48 89 df             	mov    %rbx,%rdi
       218:	41 c7 04 24 00 00 00 	movl   $0x0,(%r12)
       21f:	00
       220:	e8 00 00 00 00       	callq  225 <__percpu_counter_add+0x52>
       225:	eb 04                	jmp    22b <__percpu_counter_add+0x58>
       227:	45 89 2c 24          	mov    %r13d,(%r12)
       22b:	5b                   	pop    %rbx
       22c:	5b                   	pop    %rbx
       22d:	41 5c                	pop    %r12
       22f:	41 5d                	pop    %r13
       231:	c9                   	leaveq
       232:	c3                   	retq
      
      
      After:
      
      00000000000001d3 <__percpu_counter_add>:
       1d3:	55                   	push   %rbp
       1d4:	48 63 ca             	movslq %edx,%rcx
       1d7:	48 89 e5             	mov    %rsp,%rbp
       1da:	41 54                	push   %r12
       1dc:	53                   	push   %rbx
       1dd:	48 89 fb             	mov    %rdi,%rbx
       1e0:	48 8b 47 30          	mov    0x30(%rdi),%rax
       1e4:	65 44 8b 20          	mov    %gs:(%rax),%r12d
       1e8:	4d 63 e4             	movslq %r12d,%r12
       1eb:	49 01 f4             	add    %rsi,%r12
       1ee:	49 39 cc             	cmp    %rcx,%r12
       1f1:	7d 0a                	jge    1fd <__percpu_counter_add+0x2a>
       1f3:	f7 da                	neg    %edx
       1f5:	48 63 d2             	movslq %edx,%rdx
       1f8:	49 39 d4             	cmp    %rdx,%r12
       1fb:	7f 21                	jg     21e <__percpu_counter_add+0x4b>
       1fd:	48 89 df             	mov    %rbx,%rdi
       200:	e8 00 00 00 00       	callq  205 <__percpu_counter_add+0x32>
       205:	4c 01 63 18          	add    %r12,0x18(%rbx)
       209:	48 8b 43 30          	mov    0x30(%rbx),%rax
       20d:	48 89 df             	mov    %rbx,%rdi
       210:	65 c7 00 00 00 00 00 	movl   $0x0,%gs:(%rax)
       217:	e8 00 00 00 00       	callq  21c <__percpu_counter_add+0x49>
       21c:	eb 04                	jmp    222 <__percpu_counter_add+0x4f>
       21e:	65 44 89 20          	mov    %r12d,%gs:(%rax)
       222:	5b                   	pop    %rbx
       223:	41 5c                	pop    %r12
       225:	c9                   	leaveq
       226:	c3                   	retq
      Reviewed-by: default avatarPekka Enberg <penberg@kernel.org>
      Reviewed-by: default avatarTejun Heo <tj@kernel.org>
      Reviewed-by: default avatarMathieu Desnoyers <mathieu.desnoyers@efficios.com>
      Acked-by: default avatarH. Peter Anvin <hpa@zytor.com>
      Signed-off-by: default avatarChristoph Lameter <cl@linux.com>
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      819a72af
  10. 26 Oct, 2010 3 commits
    • Christoph Lameter's avatar
      percpu_counter: use this_cpu_ptr() instead of per_cpu_ptr() · ea00c30b
      Christoph Lameter authored
      
      
      this_cpu_ptr() avoids an array lookup and can use the percpu offset of the
      local cpu directly.
      Signed-off-by: default avatarChristoph Lameter <cl@linux.com>
      Cc: Eric Dumazet <eric.dumazet@gmail.com>
      Cc: Tejun Heo <tj@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      ea00c30b
    • Tejun Heo's avatar
      percpu_counter: add debugobj support · e2852ae8
      Tejun Heo authored
      All percpu counters are linked to a global list on initialization and
      removed from it on destruction.  The list is walked during CPU up/down.
      If a percpu counter is freed without being properly destroyed, the system
      will oops only on the next CPU up/down making it pretty nasty to track
      down.  This patch adds debugobj support for percpu counters so that such
      problems can be found easily.
      
      As percpu counters don't make sense on stack and can't be statically
      initialized, debugobj support is pretty simple.  It's initialized and
      activated on counter initialization, and deactivatd and destroyed on
      counter destruction.  With this patch applied, the bug fixed by commit
      602586a8
      
       (shmem: put_super must
      percpu_counter_destroy) triggers the following warning on tmpfs unmount
      and the system won't oops on the next cpu up/down operation.
      
       ------------[ cut here ]------------
       WARNING: at lib/debugobjects.c:259 debug_print_object+0x5c/0x70()
       Hardware name: Bochs
       ODEBUG: free active (active state 0) object type: percpu_counter
       Modules linked in:
       Pid: 3999, comm: umount Not tainted 2.6.36-rc2-work+ #5
       Call Trace:
        [<ffffffff81083f7f>] warn_slowpath_common+0x7f/0xc0
        [<ffffffff81084076>] warn_slowpath_fmt+0x46/0x50
        [<ffffffff813b45cc>] debug_print_object+0x5c/0x70
        [<ffffffff813b50e5>] debug_check_no_obj_freed+0x125/0x210
        [<ffffffff811577d3>] kfree+0xb3/0x2f0
        [<ffffffff81132edd>] shmem_put_super+0x1d/0x30
        [<ffffffff81162e96>] generic_shutdown_super+0x56/0xe0
        [<ffffffff81162f86>] kill_anon_super+0x16/0x60
        [<ffffffff81162ff7>] kill_litter_super+0x27/0x30
        [<ffffffff81163295>] deactivate_locked_super+0x45/0x60
        [<ffffffff81163cfa>] deactivate_super+0x4a/0x70
        [<ffffffff8117d446>] mntput_no_expire+0x86/0xe0
        [<ffffffff8117df7f>] sys_umount+0x6f/0x360
        [<ffffffff8103f01b>] system_call_fastpath+0x16/0x1b
       ---[ end trace cce2a341ba3611a7 ]---
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Acked-by: Thomas Gleixner <tglxlinutronix.de>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e2852ae8
    • Masanori ITOH's avatar
      percpu: fix list_head init bug in __percpu_counter_init() · 8474b591
      Masanori ITOH authored
      
      
      WARNING: at lib/list_debug.c:26 __list_add+0x3f/0x81()
      Hardware name: Express5800/B120a [N8400-085]
      list_add corruption. next->prev should be prev (ffffffff81a7ea00), but was dead000000200200. (next=ffff88080b872d58).
      Modules linked in: aoe ipt_MASQUERADE iptable_nat nf_nat autofs4 sunrpc bridge 8021q garp stp llc ipv6 cpufreq_ondemand acpi_cpufreq freq_table dm_round_robin dm_multipath kvm_intel kvm uinput lpfc scsi_transport_fc igb ioatdma scsi_tgt i2c_i801 i2c_core dca iTCO_wdt iTCO_vendor_support pcspkr shpchp megaraid_sas [last unloaded: aoe]
      Pid: 54, comm: events/3 Tainted: G        W  2.6.34-vanilla1 #1
      Call Trace:
      [<ffffffff8104bd77>] warn_slowpath_common+0x7c/0x94
      [<ffffffff8104bde6>] warn_slowpath_fmt+0x41/0x43
      [<ffffffff8120fd2e>] __list_add+0x3f/0x81
      [<ffffffff81212a12>] __percpu_counter_init+0x59/0x6b
      [<ffffffff810d8499>] bdi_init+0x118/0x17e
      [<ffffffff811f2c50>] blk_alloc_queue_node+0x79/0x143
      [<ffffffff811f2d2b>] blk_alloc_queue+0x11/0x13
      [<ffffffffa02a931d>] aoeblk_gdalloc+0x8e/0x1c9 [aoe]
      [<ffffffffa02aa655>] aoecmd_sleepwork+0x25/0xa8 [aoe]
      [<ffffffff8106186c>] worker_thread+0x1a9/0x237
      [<ffffffffa02aa630>] ? aoecmd_sleepwork+0x0/0xa8 [aoe]
      [<ffffffff81065827>] ? autoremove_wake_function+0x0/0x39
      [<ffffffff810616c3>] ? worker_thread+0x0/0x237
      [<ffffffff810653ad>] kthread+0x7f/0x87
      [<ffffffff8100aa24>] kernel_thread_helper+0x4/0x10
      [<ffffffff8106532e>] ? kthread+0x0/0x87
      [<ffffffff8100aa20>] ? kernel_thread_helper+0x0/0x10
      
      It's because there is no initialization code for a list_head contained in
      the struct backing_dev_info under CONFIG_HOTPLUG_CPU, and the bug comes up
      when block device drivers calling blk_alloc_queue() are used.  In case of
      me, I got them by using aoe.
      Signed-off-by: default avatarMasanori Itoh <itoumsn@nttdata.co.jp>
      Cc: Tejun Heo <tj@kernel.org>
      Cc: <stable@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      8474b591
  11. 10 Aug, 2010 1 commit
    • Tim Chen's avatar
      tmpfs: add accurate compare function to percpu_counter library · 27f5e0f6
      Tim Chen authored
      
      
      Add percpu_counter_compare that allows for a quick but accurate comparison
      of percpu_counter with a given value.
      
      A rough count is provided by the count field in percpu_counter structure,
      without accounting for the other values stored in individual cpu counters.
      
      The actual count is a sum of count and the cpu counters.  However, count
      field is never different from the actual value by a factor of
      batch*num_online_cpu.  We do not need to get actual count for comparison
      if count is different from the given value by this factor and allows for
      quick comparison without summing up all the per cpu counters.
      Signed-off-by: default avatarTim Chen <tim.c.chen@linux.intel.com>
      Cc: Hugh Dickins <hughd@google.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      27f5e0f6
  12. 06 Jan, 2009 1 commit
  13. 29 Dec, 2008 1 commit
  14. 10 Dec, 2008 3 commits
    • Andrew Morton's avatar
      revert "percpu_counter: new function percpu_counter_sum_and_set" · 02d21168
      Andrew Morton authored
      Revert
      
          commit e8ced39d
      
      
          Author: Mingming Cao <cmm@us.ibm.com>
          Date:   Fri Jul 11 19:27:31 2008 -0400
      
              percpu_counter: new function percpu_counter_sum_and_set
      
      As described in
      
      	revert "percpu counter: clean up percpu_counter_sum_and_set()"
      
      the new percpu_counter_sum_and_set() is racy against updates to the
      cpu-local accumulators on other CPUs.  Revert that change.
      
      This means that ext4 will be slow again.  But correct.
      Reported-by: default avatarEric Dumazet <dada1@cosmosbay.com>
      Cc: "David S. Miller" <davem@davemloft.net>
      Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Mingming Cao <cmm@us.ibm.com>
      Cc: <linux-ext4@vger.kernel.org>
      Cc: <stable@kernel.org>		[2.6.27.x]
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      02d21168
    • Andrew Morton's avatar
      revert "percpu counter: clean up percpu_counter_sum_and_set()" · 71c5576f
      Andrew Morton authored
      Revert
      
          commit 1f7c14c6
          Author: Mingming Cao <cmm@us.ibm.com>
          Date:   Thu Oct 9 12:50:59 2008 -0400
      
              percpu counter: clean up percpu_counter_sum_and_set()
      
      Before this patch we had the following:
      
      percpu_counter_sum(): return the percpu_counter's value
      
      percpu_counter_sum_and_set(): return the percpu_counter's value, copying
      that value into the central value and zeroing the per-cpu counters before
      returning.
      
      After this patch, percpu_counter_sum_and_set() has gone, and
      percpu_counter_sum() gets the old percpu_counter_sum_and_set()
      functionality.
      
      Problem is, as Eric points out, the old percpu_counter_sum_and_set()
      functionality was racy and wrong.  It zeroes out counters on "other" cpus,
      without holding any locks which will prevent races agaist updates from
      those other CPUS.
      
      This patch reverts 1f7c14c6.  This means
      that percpu_counter_sum_and_set() still has the race, but
      percpu_counter_sum() does not.
      
      Note that this is not a simple revert - ext4 has since started using
      percpu_counter_sum() for its dirty_blocks counter as well.
      
      Note that this revert patch changes percpu_counter_sum() semantics.
      
      Before the patch, a call to percpu_counter_sum() will bring the counter's
      central counter mostly up-to-date, so a following percpu_counter_read()
      will return a close value.
      
      After this patch, a call to percpu_counter_sum() will leave the counter's
      central accumulator unaltered, so a subsequent call to
      percpu_counter_read() can now return a significantly inaccurate result.
      
      If there is any code in the tree which was introduced after
      e8ced39d
      
       was merged, and which depends
      upon the new percpu_counter_sum() semantics, that code will break.
      Reported-by: default avatarEric Dumazet <dada1@cosmosbay.com>
      Cc: "David S. Miller" <davem@davemloft.net>
      Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Mingming Cao <cmm@us.ibm.com>
      Cc: <linux-ext4@vger.kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      71c5576f
    • Eric Dumazet's avatar
      percpu_counter: fix CPU unplug race in percpu_counter_destroy() · fd3d664f
      Eric Dumazet authored
      
      
      We should first delete the counter from percpu_counters list
      before freeing memory, or a percpu_counter_hotcpu_callback()
      could dereference a NULL pointer.
      Signed-off-by: default avatarEric Dumazet <dada1@cosmosbay.com>
      Acked-by: default avatarDavid S. Miller <davem@davemloft.net>
      Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Mingming Cao <cmm@us.ibm.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      fd3d664f
  15. 09 Oct, 2008 1 commit
  16. 11 Jul, 2008 1 commit
    • Mingming Cao's avatar
      percpu_counter: new function percpu_counter_sum_and_set · e8ced39d
      Mingming Cao authored
      
      
      Delayed allocation need to check free blocks at every write time.
      percpu_counter_read_positive() is not quit accurate. delayed
      allocation need a more accurate accounting, but using
      percpu_counter_sum_positive() is frequently is quite expensive.
      
      This patch added a new function to update center counter when sum
      per-cpu counter, to increase the accurate rate for next
      percpu_counter_read() and require less calling expensive
      percpu_counter_sum().
      Signed-off-by: default avatarMingming Cao <cmm@us.ibm.com>
      Signed-off-by: default avatar"Theodore Ts'o" <tytso@mit.edu>
      e8ced39d
  17. 30 Apr, 2008 1 commit
  18. 19 Oct, 2007 1 commit
  19. 17 Oct, 2007 8 commits
  20. 16 Jul, 2007 2 commits
  21. 23 Jun, 2006 2 commits