1. 22 Sep, 2009 15 commits
  2. 11 Sep, 2009 1 commit
    • Jens Axboe's avatar
      writeback: switch to per-bdi threads for flushing data · 03ba3782
      Jens Axboe authored
      
      
      This gets rid of pdflush for bdi writeout and kupdated style cleaning.
      pdflush writeout suffers from lack of locality and also requires more
      threads to handle the same workload, since it has to work in a
      non-blocking fashion against each queue. This also introduces lumpy
      behaviour and potential request starvation, since pdflush can be starved
      for queue access if others are accessing it. A sample ffsb workload that
      does random writes to files is about 8% faster here on a simple SATA drive
      during the benchmark phase. File layout also seems a LOT more smooth in
      vmstat:
      
       r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
       0  1      0 608848   2652 375372    0    0     0 71024  604    24  1 10 48 42
       0  1      0 549644   2712 433736    0    0     0 60692  505    27  1  8 48 44
       1  0      0 476928   2784 505192    0    0     4 29540  553    24  0  9 53 37
       0  1      0 457972   2808 524008    0    0     0 54876  331    16  0  4 38 58
       0  1      0 366128   2928 614284    0    0     4 92168  710    58  0 13 53 34
       0  1      0 295092   3000 684140    0    0     0 62924  572    23  0  9 53 37
       0  1      0 236592   3064 741704    0    0     4 58256  523    17  0  8 48 44
       0  1      0 165608   3132 811464    0    0     0 57460  560    21  0  8 54 38
       0  1      0 102952   3200 873164    0    0     4 74748  540    29  1 10 48 41
       0  1      0  48604   3252 926472    0    0     0 53248  469    29  0  7 47 45
      
      where vanilla tends to fluctuate a lot in the creation phase:
      
       r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
       1  1      0 678716   5792 303380    0    0     0 74064  565    50  1 11 52 36
       1  0      0 662488   5864 319396    0    0     4   352  302   329  0  2 47 51
       0  1      0 599312   5924 381468    0    0     0 78164  516    55  0  9 51 40
       0  1      0 519952   6008 459516    0    0     4 78156  622    56  1 11 52 37
       1  1      0 436640   6092 541632    0    0     0 82244  622    54  0 11 48 41
       0  1      0 436640   6092 541660    0    0     0     8  152    39  0  0 51 49
       0  1      0 332224   6200 644252    0    0     4 102800  728    46  1 13 49 36
       1  0      0 274492   6260 701056    0    0     4 12328  459    49  0  7 50 43
       0  1      0 211220   6324 763356    0    0     0 106940  515    37  1 10 51 39
       1  0      0 160412   6376 813468    0    0     0  8224  415    43  0  6 49 45
       1  1      0  85980   6452 886556    0    0     4 113516  575    39  1 11 54 34
       0  2      0  85968   6452 886620    0    0     0  1640  158   211  0  0 46 54
      
      A 10 disk test with btrfs performs 26% faster with per-bdi flushing. A
      SSD based writeback test on XFS performs over 20% better as well, with
      the throughput being very stable around 1GB/sec, where pdflush only
      manages 750MB/sec and fluctuates wildly while doing so. Random buffered
      writes to many files behave a lot better as well, as does random mmap'ed
      writes.
      
      A separate thread is added to sync the super blocks. In the long term,
      adding sync_supers_bdi() functionality could get rid of this thread again.
      Signed-off-by: default avatarJens Axboe <jens.axboe@oracle.com>
      03ba3782
  3. 27 Aug, 2009 1 commit
  4. 10 Jul, 2009 1 commit
  5. 23 Jun, 2009 1 commit
  6. 18 Jun, 2009 1 commit
  7. 17 Jun, 2009 16 commits
    • KAMEZAWA Hiroyuki's avatar
      mm: fix lumpy reclaim lru handling at isolate_lru_pages · ee993b13
      KAMEZAWA Hiroyuki authored
      
      
      At lumpy reclaim, a page failed to be taken by __isolate_lru_page() can be
      pushed back to "src" list by list_move().  But the page may not be from
      "src" list.  This pushes the page back to wrong LRU.  And list_move()
      itself is unnecessary because the page is not on top of LRU.  Then, leave
      it as it is if __isolate_lru_page() fails.
      Reviewed-by: default avatarMinchan Kim <minchan.kim@gmail.com>
      Reviewed-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Acked-by: default avatarMel Gorman <mel@csn.ul.ie>
      Signed-off-by: default avatarKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      ee993b13
    • Mel Gorman's avatar
      vmscan: count the number of times zone_reclaim() scans and fails · 24cf7251
      Mel Gorman authored
      
      
      On NUMA machines, the administrator can configure zone_reclaim_mode that
      is a more targetted form of direct reclaim.  On machines with large NUMA
      distances for example, a zone_reclaim_mode defaults to 1 meaning that
      clean unmapped pages will be reclaimed if the zone watermarks are not
      being met.
      
      There is a heuristic that determines if the scan is worthwhile but it is
      possible that the heuristic will fail and the CPU gets tied up scanning
      uselessly.  Detecting the situation requires some guesswork and
      experimentation so this patch adds a counter "zreclaim_failed" to
      /proc/vmstat.  If during high CPU utilisation this counter is increasing
      rapidly, then the resolution to the problem may be to set
      /proc/sys/vm/zone_reclaim_mode to 0.
      
      [akpm@linux-foundation.org: name things consistently]
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Reviewed-by: default avatarRik van Riel <riel@redhat.com>
      Cc: Christoph Lameter <cl@linux-foundation.org>
      Reviewed-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: <stable@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      24cf7251
    • Mel Gorman's avatar
      vmscan: do not unconditionally treat zones that fail zone_reclaim() as full · fa5e084e
      Mel Gorman authored
      On NUMA machines, the administrator can configure zone_reclaim_mode that
      is a more targetted form of direct reclaim.  On machines with large NUMA
      distances for example, a zone_reclaim_mode defaults to 1 meaning that
      clean unmapped pages will be reclaimed if the zone watermarks are not
      being met.  The problem is that zone_reclaim() failing at all means the
      zone gets marked full.
      
      This can cause situations where a zone is usable, but is being skipped
      because it has been considered full.  Take a situation where a large tmpfs
      mount is occuping a large percentage of memory overall.  The pages do not
      get cleaned or reclaimed by zone_reclaim(), but the zone gets marked full
      and the zonelist cache considers them not worth trying in the future.
      
      This patch makes zone_reclaim() return more fine-grained information about
      what occured when zone_reclaim() failued.  The zone only gets marked full
      if it really is unreclaimable.  If it's a case that the scan did not occur
      or if enough pages were not reclaimed with the limited reclaim_mode, then
      the zone is simply skipped.
      
      There is a side-effect to this patch.  Currently, if zone_reclaim()
      successfully reclaimed SWAP_CLUSTER_MAX, an allocation attempt would go
      ahead.  With this patch applied, zone watermarks are rechecked after
      zone_reclaim() does some work.
      
      This bug was introduced by commit 9276b1bc
      
      
      ("memory page_alloc zonelist caching speedup") way back in 2.6.19 when the
      zonelist_cache was introduced.  It was not intended that zone_reclaim()
      aggressively consider the zone to be full when it failed as full direct
      reclaim can still be an option.  Due to the age of the bug, it should be
      considered a -stable candidate.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Reviewed-by: default avatarWu Fengguang <fengguang.wu@intel.com>
      Reviewed-by: default avatarRik van Riel <riel@redhat.com>
      Reviewed-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Christoph Lameter <cl@linux-foundation.org>
      Cc: <stable@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      fa5e084e
    • Mel Gorman's avatar
      vmscan: properly account for the number of page cache pages zone_reclaim() can reclaim · 90afa5de
      Mel Gorman authored
      
      
      A bug was brought to my attention against a distro kernel but it affects
      mainline and I believe problems like this have been reported in various
      guises on the mailing lists although I don't have specific examples at the
      moment.
      
      The reported problem was that malloc() stalled for a long time (minutes in
      some cases) if a large tmpfs mount was occupying a large percentage of
      memory overall.  The pages did not get cleaned or reclaimed by
      zone_reclaim() because the zone_reclaim_mode was unsuitable, but the lists
      are uselessly scanned frequencly making the CPU spin at near 100%.
      
      This patchset intends to address that bug and bring the behaviour of
      zone_reclaim() more in line with expectations which were noticed during
      investigation.  It is based on top of mmotm and takes advantage of
      Kosaki's work with respect to zone_reclaim().
      
      Patch 1 fixes the heuristics that zone_reclaim() uses to determine if the
      	scan should go ahead. The broken heuristic is what was causing the
      	malloc() stall as it uselessly scanned the LRU constantly. Currently,
      	zone_reclaim is assuming zone_reclaim_mode is 1 and historically it
      	could not deal with tmpfs pages at all. This fixes up the heuristic so
      	that an unnecessary scan is more likely to be correctly avoided.
      
      Patch 2 notes that zone_reclaim() returning a failure automatically means
      	the zone is marked full. This is not always true. It could have
      	failed because the GFP mask or zone_reclaim_mode were unsuitable.
      
      Patch 3 introduces a counter zreclaim_failed that will increment each
      	time the zone_reclaim scan-avoidance heuristics fail. If that
      	counter is rapidly increasing, then zone_reclaim_mode should be
      	set to 0 as a temporarily resolution and a bug reported because
      	the scan-avoidance heuristic is still broken.
      
      This patch:
      
      On NUMA machines, the administrator can configure zone_reclaim_mode that
      is a more targetted form of direct reclaim.  On machines with large NUMA
      distances for example, a zone_reclaim_mode defaults to 1 meaning that
      clean unmapped pages will be reclaimed if the zone watermarks are not
      being met.
      
      There is a heuristic that determines if the scan is worthwhile but the
      problem is that the heuristic is not being properly applied and is
      basically assuming zone_reclaim_mode is 1 if it is enabled.  The lack of
      proper detection can manfiest as high CPU usage as the LRU list is scanned
      uselessly.
      
      Historically, once enabled it was depending on NR_FILE_PAGES which may
      include swapcache pages that the reclaim_mode cannot deal with.  Patch
      vmscan-change-the-number-of-the-unmapped-files-in-zone-reclaim.patch by
      Kosaki Motohiro noted that zone_page_state(zone, NR_FILE_PAGES) included
      pages that were not file-backed such as swapcache and made a calculation
      based on the inactive, active and mapped files.  This is far superior when
      zone_reclaim==1 but if RECLAIM_SWAP is set, then NR_FILE_PAGES is a
      reasonable starting figure.
      
      This patch alters how zone_reclaim() works out how many pages it might be
      able to reclaim given the current reclaim_mode.  If RECLAIM_SWAP is set in
      the reclaim_mode it will either consider NR_FILE_PAGES as potential
      candidates or else use NR_{IN}ACTIVE}_PAGES-NR_FILE_MAPPED to discount
      swapcache and other non-file-backed pages.  If RECLAIM_WRITE is not set,
      then NR_FILE_DIRTY number of pages are not candidates.  If RECLAIM_SWAP is
      not set, then NR_FILE_MAPPED are not.
      
      [kosaki.motohiro@jp.fujitsu.com: Estimate unmapped pages minus tmpfs pages]
      [fengguang.wu@intel.com: Fix underflow problem in Kosaki's estimate]
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Reviewed-by: default avatarRik van Riel <riel@redhat.com>
      Acked-by: default avatarChristoph Lameter <cl@linux-foundation.org>
      Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: <stable@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      90afa5de
    • Daisuke Nishimura's avatar
      vmscan: handle may_swap more strictly · 9198e96c
      Daisuke Nishimura authored
      Commit 2e2e4259
      
       ("vmscan,memcg:
      reintroduce sc->may_swap) add may_swap flag and handle it at
      get_scan_ratio().
      
      But the result of get_scan_ratio() is ignored when priority == 0, so anon
      lru is scanned even if may_swap == 0 or nr_swap_pages == 0.  IMHO, this is
      not an expected behavior.
      
      As for memcg especially, because of this behavior many and many pages are
      swapped-out just in vain when oom is invoked by mem+swap limit.
      
      This patch is for handling may_swap flag more strictly.
      Signed-off-by: default avatarDaisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
      Reviewed-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Minchan Kim <minchan.kim@gmail.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Balbir Singh <balbir@linux.vnet.ibm.com>
      Acked-by: default avatarKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      9198e96c
    • Wu Fengguang's avatar
      vmscan: merge duplicate code in shrink_active_list() · 3eb4140f
      Wu Fengguang authored
      
      
      The "move pages to active list" and "move pages to inactive list" code
      blocks are mostly identical and can be served by a function.
      
      Thanks to Andrew Morton for pointing this out.
      
      Note that buffer_heads_over_limit check will also be carried out for
      re-activated pages, which is slightly different from pre-2.6.28 kernels.
      Also, Rik's "vmscan: evict use-once pages first" patch could totally stop
      scans of active file list when memory pressure is low.  So the net effect
      could be, the number of buffer heads is now more likely to grow large.
      
      However that's fine according to Johannes' comments:
      
        I don't think that this could be harmful.  We just preserve the buffer
        mappings of what we consider the working set and with low memory
        pressure, as you say, this set is not big.
      
        As to stripping of reactivated pages: the only pages we re-activate
        for now are those VM_EXEC mapped ones.  Since we don't expect IO from
        or to these pages, removing the buffer mappings in case they grow too
        large should be okay, I guess.
      
      Cc: Pekka Enberg <penberg@cs.helsinki.fi>
      Acked-by: default avatarPeter Zijlstra <peterz@infradead.org>
      Reviewed-by: default avatarRik van Riel <riel@redhat.com>
      Reviewed-by: default avatarMinchan Kim <minchan.kim@gmail.com>
      Reviewed-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Signed-off-by: default avatarWu Fengguang <fengguang.wu@intel.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      3eb4140f
    • Wu Fengguang's avatar
      vmscan: make mapped executable pages the first class citizen · 8cab4754
      Wu Fengguang authored
      
      
      Protect referenced PROT_EXEC mapped pages from being deactivated.
      
      PROT_EXEC(or its internal presentation VM_EXEC) pages normally belong to some
      currently running executables and their linked libraries, they shall really be
      cached aggressively to provide good user experiences.
      
      Thanks to Johannes Weiner for the advice to reuse the VMA walk in
      page_referenced() to get the PROT_EXEC bit.
      
      [more details]
      
      ( The consequences of this patch will have to be discussed together with
        Rik van Riel's recent patch "vmscan: evict use-once pages first". )
      
      ( Some of the good points and insights are taken into this changelog.
        Thanks to all the involved people for the great LKML discussions. )
      
      the problem
      ===========
      
      For a typical desktop, the most precious working set is composed of
      *actively accessed*
      	(1) memory mapped executables
      	(2) and their anonymous pages
      	(3) and other files
      	(4) and the dcache/icache/.. slabs
      while the least important data are
      	(5) infrequently used or use-once files
      
      For a typical desktop, one major problem is busty and large amount of (5)
      use-once files flushing out the working set.
      
      Inside the working set, (4) dcache/icache have already been too sticky ;-)
      So we only have to care (2) anonymous and (1)(3) file pages.
      
      anonymous pages
      ===============
      
      Anonymous pages are effectively immune to the streaming IO attack, because we
      now have separate file/anon LRU lists. When the use-once files crowd into the
      file LRU, the list's "quality" is significantly lowered. Therefore the scan
      balance policy in get_scan_ratio() will choose to scan the (low quality) file
      LRU much more frequently than the anon LRU.
      
      file pages
      ==========
      
      Rik proposed to *not* scan the active file LRU when the inactive list grows
      larger than active list. This guarantees that when there are use-once streaming
      IO, and the working set is not too large(so that active_size < inactive_size),
      the active file LRU will *not* be scanned at all. So the not-too-large working
      set can be well protected.
      
      But there are also situations where the file working set is a bit large so that
      (active_size >= inactive_size), or the streaming IOs are not purely use-once.
      In these cases, the active list will be scanned slowly. Because the current
      shrink_active_list() policy is to deactivate active pages regardless of their
      referenced bits. The deactivated pages become susceptible to the streaming IO
      attack: the inactive list could be scanned fast (500MB / 50MBps = 10s) so that
      the deactivated pages don't have enough time to get re-referenced. Because a
      user tend to switch between windows in intervals from seconds to minutes.
      
      This patch holds mapped executable pages in the active list as long as they
      are referenced during each full scan of the active list.  Because the active
      list is normally scanned much slower, they get longer grace time (eg. 100s)
      for further references, which better matches the pace of user operations.
      
      Therefore this patch greatly prolongs the in-cache time of executable code,
      when there are moderate memory pressures.
      
      	before patch: guaranteed to be cached if reference intervals < I
      	after  patch: guaranteed to be cached if reference intervals < I+A
      		      (except when randomly reclaimed by the lumpy reclaim)
      where
      	A = time to fully scan the   active file LRU
      	I = time to fully scan the inactive file LRU
      
      Note that normally A >> I.
      
      side effects
      ============
      
      This patch is safe in general, it restores the pre-2.6.28 mmap() behavior
      but in a much smaller and well targeted scope.
      
      One may worry about some one to abuse the PROT_EXEC heuristic.  But as
      Andrew Morton stated, there are other tricks to getting that sort of boost.
      
      Another concern is the PROT_EXEC mapped pages growing large in rare cases,
      and therefore hurting reclaim efficiency. But a sane application targeted for
      large audience will never use PROT_EXEC for data mappings. If some home made
      application tries to abuse that bit, it shall be aware of the consequences.
      If it is abused to scale of 2/3 total memory, it gains nothing but overheads.
      
      benchmarks
      ==========
      
      1) memory tight desktop
      
      1.1) brief summary
      
      - clock time and major faults are reduced by 50%;
      - pswpin numbers are reduced to ~1/3.
      
      That means X desktop responsiveness is doubled under high memory/swap pressure.
      
      1.2) test scenario
      
      - nfsroot gnome desktop with 512M physical memory
      - run some programs, and switch between the existing windows
        after starting each new program.
      
      1.3) progress timing (seconds)
      
        before       after    programs
          0.02        0.02    N xeyes
          0.75        0.76    N firefox
          2.02        1.88    N nautilus
          3.36        3.17    N nautilus --browser
          5.26        4.89    N gthumb
          7.12        6.47    N gedit
          9.22        8.16    N xpdf /usr/share/doc/shared-mime-info/shared-mime-info-spec.pdf
         13.58       12.55    N xterm
         15.87       14.57    N mlterm
         18.63       17.06    N gnome-terminal
         21.16       18.90    N urxvt
         26.24       23.48    N gnome-system-monitor
         28.72       26.52    N gnome-help
         32.15       29.65    N gnome-dictionary
         39.66       36.12    N /usr/games/sol
         43.16       39.27    N /usr/games/gnometris
         48.65       42.56    N /usr/games/gnect
         53.31       47.03    N /usr/games/gtali
         58.60       52.05    N /usr/games/iagno
         65.77       55.42    N /usr/games/gnotravex
         70.76       61.47    N /usr/games/mahjongg
         76.15       67.11    N /usr/games/gnome-sudoku
         86.32       75.15    N /usr/games/glines
         92.21       79.70    N /usr/games/glchess
        103.79       88.48    N /usr/games/gnomine
        113.84       96.51    N /usr/games/gnotski
        124.40      102.19    N /usr/games/gnibbles
        137.41      114.93    N /usr/games/gnobots2
        155.53      125.02    N /usr/games/blackjack
        179.85      135.11    N /usr/games/same-gnome
        224.49      154.50    N /usr/bin/gnome-window-properties
        248.44      162.09    N /usr/bin/gnome-default-applications-properties
        282.62      173.29    N /usr/bin/gnome-at-properties
        323.72      188.21    N /usr/bin/gnome-typing-monitor
        363.99      199.93    N /usr/bin/gnome-at-visual
        394.21      206.95    N /usr/bin/gnome-sound-properties
        435.14      224.49    N /usr/bin/gnome-at-mobility
        463.05      234.11    N /usr/bin/gnome-keybinding-properties
        503.75      248.59    N /usr/bin/gnome-about-me
        554.00      276.27    N /usr/bin/gnome-display-properties
        615.48      304.39    N /usr/bin/gnome-network-preferences
        693.03      342.01    N /usr/bin/gnome-mouse-properties
        759.90      388.58    N /usr/bin/gnome-appearance-properties
        937.90      508.47    N /usr/bin/gnome-control-center
       1109.75      587.57    N /usr/bin/gnome-keyboard-properties
       1399.05      758.16    N : oocalc
       1524.64      830.03    N : oodraw
       1684.31      900.03    N : ooimpress
       1874.04      993.91    N : oomath
       2115.12     1081.89    N : ooweb
       2369.02     1161.99    N : oowriter
      
      Note that the last ": oo*" commands are actually commented out.
      
      1.4) vmstat numbers (some relevant ones are marked with *)
      
                                  before    after
       nr_free_pages              1293      3898
       nr_inactive_anon           59956     53460
       nr_active_anon             26815     30026
       nr_inactive_file           2657      3218
       nr_active_file             2019      2806
       nr_unevictable             4         4
       nr_mlock                   4         4
       nr_anon_pages              26706     27859
      *nr_mapped                  3542      4469
       nr_file_pages              72232     67681
       nr_dirty                   1         0
       nr_writeback               123       19
       nr_slab_reclaimable        3375      3534
       nr_slab_unreclaimable      11405     10665
       nr_page_table_pages        8106      7864
       nr_unstable                0         0
       nr_bounce                  0         0
      *nr_vmscan_write            394776    230839
       nr_writeback_temp          0         0
       numa_hit                   6843353   3318676
       numa_miss                  0         0
       numa_foreign               0         0
       numa_interleave            1719      1719
       numa_local                 6843353   3318676
       numa_other                 0         0
      *pgpgin                     5954683   2057175
      *pgpgout                    1578276   922744
      *pswpin                     1486615   512238
      *pswpout                    394568    230685
       pgalloc_dma                277432    56602
       pgalloc_dma32              6769477   3310348
       pgalloc_normal             0         0
       pgalloc_movable            0         0
       pgfree                     7048396   3371118
       pgactivate                 2036343   1471492
       pgdeactivate               2189691   1612829
       pgfault                    3702176   3100702
      *pgmajfault                 452116    201343
       pgrefill_dma               12185     7127
       pgrefill_dma32             334384    653703
       pgrefill_normal            0         0
       pgrefill_movable           0         0
       pgsteal_dma                74214     22179
       pgsteal_dma32              3334164   1638029
       pgsteal_normal             0         0
       pgsteal_movable            0         0
       pgscan_kswapd_dma          1081421   1216199
       pgscan_kswapd_dma32        58979118  46002810
       pgscan_kswapd_normal       0         0
       pgscan_kswapd_movable      0         0
       pgscan_direct_dma          2015438   1086109
       pgscan_direct_dma32        55787823  36101597
       pgscan_direct_normal       0         0
       pgscan_direct_movable      0         0
       pginodesteal               3461      7281
       slabs_scanned              564864    527616
       kswapd_steal               2889797   1448082
       kswapd_inodesteal          14827     14835
       pageoutrun                 43459     21562
       allocstall                 9653      4032
       pgrotated                  384216    228631
      
      1.5) free numbers at the end of the tests
      
      before patch:
                                   total       used       free     shared    buffers     cached
                      Mem:           474        467          7          0          0        236
                      -/+ buffers/cache:        230        243
                      Swap:         1023        418        605
      
      after patch:
                                   total       used       free     shared    buffers     cached
                      Mem:           474        457         16          0          0        236
                      -/+ buffers/cache:        221        253
                      Swap:         1023        404        619
      
      2) memory flushing in a file server
      
      2.1) brief summary
      
      The number of major faults from 50 to 3 during 10% cache hot reads.
      
      That means this patch successfully stops major faults when the active file
      list is slowly scanned when there are partially cache hot streaming IO.
      
      2.2) test scenario
      
      Do 100000 pread(size=110 pages, offset=(i*100) pages), where 10% of the
      pages will be activated:
      
              for i in `seq 0 100 10000000`; do echo $i 110;  done > pattern-hot-10
              iotrace.rb --load pattern-hot-10 --play /b/sparse
      	vmmon  nr_mapped nr_active_file nr_inactive_file   pgmajfault pgdeactivate pgfree
      
      and monitor /proc/vmstat during the time. The test box has 2G memory.
      
      I carried out tests on fresh booted console as well as X desktop, and
      fetched the vmstat numbers on
      
      (1) begin:     shortly after the big read IO starts;
      (2) end:       just before the big read IO stops;
      (3) restore:   the big read IO stops and the zsh working set restored
      (4) restore X: after IO, switch back and forth between the urxvt and firefox
                     windows to restore their working set.
      
      2.3) console mode results
      
              nr_mapped   nr_active_file nr_inactive_file       pgmajfault     pgdeactivate           pgfree
      
      2.6.29 VM_EXEC protection ON:
      begin:       2481             2237             8694              630                0           574299
      end:          275           231976           233914              633           776271         20933042
      restore:      370           232154           234524              691           777183         20958453
      
      2.6.29 VM_EXEC protection ON (second run):
      begin:       2434             2237             8493              629                0           574195
      end:          284           231970           233536              632           771918         20896129
      restore:      399           232218           234789              690           774526         20957909
      
      2.6.30-rc4-mm VM_EXEC protection OFF:
      begin:       2479             2344             9659              210                0           579643
      end:          284           232010           234142              260           772776         20917184
      restore:      379           232159           234371              301           774888         20967849
      
      The above console numbers show that
      
      - The startup pgmajfault of 2.6.30-rc4-mm is merely 1/3 that of 2.6.29.
        I'd attribute that improvement to the mmap readahead improvements :-)
      
      - The pgmajfault increment during the file copy is 633-630=3 vs 260-210=50.
        That's a huge improvement - which means with the VM_EXEC protection logic,
        active mmap pages is pretty safe even under partially cache hot streaming IO.
      
      - when active:inactive file lru size reaches 1:1, their scan rates is 1:20.8
        under 10% cache hot IO. (computed with formula Dpgdeactivate:Dpgfree)
        That roughly means the active mmap pages get 20.8 more chances to get
        re-referenced to stay in memory.
      
      - The absolute nr_mapped drops considerably to 1/9 during the big IO, and the
        dropped pages are mostly inactive ones. The patch has almost no impact in
        this aspect, that means it won't unnecessarily increase memory pressure.
        (In contrast, your 20% mmap protection ratio will keep them all, and
        therefore eliminate the extra 41 major faults to restore working set
        of zsh etc.)
      
      The iotrace.rb read throughput is
      	151.194384MB/s 284.198252s 100001x 450560b --load pattern-hot-10 --play /b/sparse
      which means the inactive list is rotated at the speed of 250MB/s,
      so a full scan of which takes about 3.5 seconds, while a full scan
      of active file list takes about 77 seconds.
      
      2.4) X mode results
      
      We can reach roughly the same conclusions for X desktop:
      
              nr_mapped   nr_active_file nr_inactive_file       pgmajfault     pgdeactivate           pgfree
      
      2.6.30-rc4-mm VM_EXEC protection ON:
      begin:       9740             8920            64075              561                0           678360
      end:          768           218254           220029              565           798953         21057006
      restore:      857           218543           220987              606           799462         21075710
      restore X:   2414           218560           225344              797           799462         21080795
      
      2.6.30-rc4-mm VM_EXEC protection OFF:
      begin:       9368             5035            26389              554                0           633391
      end:          770           218449           221230              661           646472         17832500
      restore:     1113           218466           220978              710           649881         17905235
      restore X:   2687           218650           225484              947           802700         21083584
      
      - the absolute nr_mapped drops considerably (to 1/13 of the original size)
        during the streaming IO.
      - the delta of pgmajfault is 3 vs 107 during IO, or 236 vs 393
        during the whole process.
      
      Cc: Elladan <elladan@eskimo.com>
      Cc: Nick Piggin <npiggin@suse.de>
      Cc: Andi Kleen <andi@firstfloor.org>
      Cc: Christoph Lameter <cl@linux-foundation.org>
      Acked-by: default avatarRik van Riel <riel@redhat.com>
      Acked-by: default avatarPeter Zijlstra <peterz@infradead.org>
      Acked-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Reviewed-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Reviewed-by: default avatarMinchan Kim <minchan.kim@gmail.com>
      Signed-off-by: default avatarWu Fengguang <fengguang.wu@intel.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      8cab4754
    • Wu Fengguang's avatar
      vmscan: report vm_flags in page_referenced() · 6fe6b7e3
      Wu Fengguang authored
      
      
      Collect vma->vm_flags of the VMAs that actually referenced the page.
      
      This is preparing for more informed reclaim heuristics, eg.  to protect
      executable file pages more aggressively.  For now only the VM_EXEC bit
      will be used by the caller.
      
      Thanks to Johannes, Peter and Minchan for all the good tips.
      Acked-by: default avatarPeter Zijlstra <peterz@infradead.org>
      Reviewed-by: default avatarRik van Riel <riel@redhat.com>
      Reviewed-by: default avatarMinchan Kim <minchan.kim@gmail.com>
      Reviewed-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Signed-off-by: default avatarWu Fengguang <fengguang.wu@intel.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      6fe6b7e3
    • KAMEZAWA Hiroyuki's avatar
      mm: add swap cache interface for swap reference · cb4b86ba
      KAMEZAWA Hiroyuki authored
      
      
      In a following patch, the usage of swap cache is recorded into swap_map.
      This patch is for necessary interface changes to do that.
      
      2 interfaces:
      
        - swapcache_prepare()
        - swapcache_free()
      
      are added for allocating/freeing refcnt from swap-cache to existing swap
      entries.  But implementation itself is not changed under this patch.  At
      adding swapcache_free(), memcg's hook code is moved under
      swapcache_free().  This is better than using scattered hooks.
      Signed-off-by: default avatarKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Reviewed-by: default avatarDaisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
      Acked-by: default avatarBalbir Singh <balbir@in.ibm.com>
      Cc: Hugh Dickins <hugh.dickins@tiscali.co.uk>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Li Zefan <lizf@cn.fujitsu.com>
      Cc: Dhaval Giani <dhaval@linux.vnet.ibm.com>
      Cc: YAMAMOTO Takashi <yamamoto@valinux.co.jp>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      cb4b86ba
    • KOSAKI Motohiro's avatar
      mm: remove CONFIG_UNEVICTABLE_LRU config option · 68377659
      KOSAKI Motohiro authored
      
      
      Currently, nobody wants to turn UNEVICTABLE_LRU off.  Thus this
      configurability is unnecessary.
      Signed-off-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Andi Kleen <andi@firstfloor.org>
      Acked-by: default avatarMinchan Kim <minchan.kim@gmail.com>
      Cc: David Woodhouse <dwmw2@infradead.org>
      Cc: Matt Mackall <mpm@selenic.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      68377659
    • MinChan Kim's avatar
      vmscan: prevent shrinking of active anon lru list in case of no swap space V3 · 69c85481
      MinChan Kim authored
      
      
      shrink_zone() can deactivate active anon pages even if we don't have a
      swap device.  Many embedded products don't have a swap device.  So the
      deactivation of anon pages is unnecessary.
      
      This patch prevents unnecessary deactivation of anon lru pages.  But, it
      don't prevent aging of anon pages to swap out.
      Signed-off-by: default avatarMinchan Kim <minchan.kim@gmail.com>
      Acked-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Acked-by: default avatarRik van Riel <riel@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      69c85481
    • Wu Fengguang's avatar
      vmscan: ZVC updates in shrink_active_list() can be done once · af166777
      Wu Fengguang authored
      
      
      This effectively lifts the unit of updates to nr_inactive_* and
      pgdeactivate from PAGEVEC_SIZE=14 to SWAP_CLUSTER_MAX=32, or
      MAX_ORDER_NR_PAGES=1024 for reclaim_zone().
      
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Acked-by: default avatarRik van Riel <riel@redhat.com>
      Reviewed-by: default avatarMinchan Kim <minchan.kim@gmail.com>
      Signed-off-by: default avatarWu Fengguang <fengguang.wu@intel.com>
      Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Christoph Lameter <cl@linux-foundation.org>
      Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      af166777
    • Wu Fengguang's avatar
      vmscan: cleanup the scan batching code · 6e08a369
      Wu Fengguang authored
      
      
      The vmscan batching logic is twisting.  Move it into a standalone function
      nr_scan_try_batch() and document it.  No behavior change.
      Signed-off-by: default avatarWu Fengguang <fengguang.wu@intel.com>
      Acked-by: default avatarRik van Riel <riel@redhat.com>
      Cc: Nick Piggin <npiggin@suse.de>
      Cc: Christoph Lameter <cl@linux-foundation.org>
      Acked-by: default avatarPeter Zijlstra <a.p.zijlstra@chello.nl>
      Acked-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      6e08a369
    • Rik van Riel's avatar
      vmscan: evict use-once pages first · 56e49d21
      Rik van Riel authored
      
      
      When the file LRU lists are dominated by streaming IO pages, evict those
      pages first, before considering evicting other pages.
      
      This should be safe from deadlocks or performance problems
      because only three things can happen to an inactive file page:
      
      1) referenced twice and promoted to the active list
      2) evicted by the pageout code
      3) under IO, after which it will get evicted or promoted
      
      The pages freed in this way can either be reused for streaming IO, or
      allocated for something else.  If the pages are used for streaming IO,
      this pageout pattern continues.  Otherwise, we will fall back to the
      normal pageout pattern.
      Signed-off-by: default avatarRik van Riel <riel@redhat.com>
      Reported-by: default avatarElladan <elladan@eskimo.com>
      Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
      Acked-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      56e49d21
    • Mel Gorman's avatar
      page allocator: use allocation flags as an index to the zone watermark · 41858966
      Mel Gorman authored
      
      
      ALLOC_WMARK_MIN, ALLOC_WMARK_LOW and ALLOC_WMARK_HIGH determin whether
      pages_min, pages_low or pages_high is used as the zone watermark when
      allocating the pages.  Two branches in the allocator hotpath determine
      which watermark to use.
      
      This patch uses the flags as an array index into a watermark array that is
      indexed with WMARK_* defines accessed via helpers.  All call sites that
      use zone->pages_* are updated to use the helpers for accessing the values
      and the array offsets for setting.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Reviewed-by: default avatarChristoph Lameter <cl@linux-foundation.org>
      Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Pekka Enberg <penberg@cs.helsinki.fi>
      Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Cc: Dave Hansen <dave@linux.vnet.ibm.com>
      Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      41858966
    • KOSAKI Motohiro's avatar
      vmscan: low order lumpy reclaim also should use PAGEOUT_IO_SYNC · 78dc583d
      KOSAKI Motohiro authored
      Commit 33c120ed ("more aggressively use
      lumpy reclaim") increased how aggressive lumpy reclaim was by isolating
      both active and inactive pages for asynchronous lumpy reclaim on
      costly-high-order pages and for cheap-high-order when memory pressure is
      high.  However, if the system is under heavy pressure and there are dirty
      pages, asynchronous IO may not be sufficient to reclaim a suitable page in
      time.
      
      This patch causes the caller to enter synchronous lumpy reclaim for
      costly-high-order pages and for cheap-high-order pages when under memory
      pressure.
      
      Minchan.kim@gmail.com said:
      
      Andy added synchronous lumpy reclaim with
      c661b078.  At that time, lumpy reclaim is
      not agressive.  His intension is just for high-order users.(above
      PAGE_ALLOC_COSTLY_ORDER).
      
      After some time, Rik added aggressive lumpy reclaim with
      33c120ed
      
      .  His intention was to do lumpy
      reclaim when high-order users and trouble getting a small set of
      contiguous pages.
      
      So we also have to add synchronous pageout for small set of contiguous
      pages.
      
      Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
      Cc: Andy Whitcroft <apw@shadowen.org>
      Acked-by: default avatarPeter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Rik van Riel <riel@redhat.com>
      Signed-off-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Reviewed-by: default avatarMinchan Kim <Minchan.kim@gmail.com>
      Reviewed-by: default avatarMel Gorman <mel@csn.ul.ie>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      78dc583d
  8. 12 Jun, 2009 1 commit
  9. 29 May, 2009 1 commit
  10. 02 May, 2009 1 commit
  11. 21 Apr, 2009 1 commit