1. 24 Apr, 2011 1 commit
    • Linus Torvalds's avatar
      vfs: get rid of insane dentry hashing rules · dea3667b
      Linus Torvalds authored
      The dentry hashing rules have been really quite complicated for a long
      while, in odd ways.  That made functions like __d_drop() very fragile
      and non-obvious.
      
      In particular, whether a dentry was hashed or not was indicated with an
      explicit DCACHE_UNHASHED bit.  That's despite the fact that the hash
      abstraction that the dentries use actually have a 'is this entry hashed
      or not' model (which is a simple test of the 'pprev' pointer).
      
      The reason that was done is because we used the normal 'is this entry
      unhashed' model to mark whether the dentry had _ever_ been hashed in the
      dentry hash tables, and that logic goes back many years (commit
      b3423415
      
      : "dcache: avoid RCU for never-hashed dentries").
      
      That, in turn, meant that __d_drop had totally different unhashing logic
      for the dentry hash table case and for the anonymous dcache case,
      because in order to use the "is this dentry hashed" logic as a flag for
      whether it had ever been on the RCU hash table, we had to unhash such a
      dentry differently so that we'd never think that it wasn't 'unhashed'
      and wouldn't be free'd correctly.
      
      That's just insane.  It made the logic really hard to follow, when there
      were two different kinds of "unhashed" states, and one of them (the one
      that used "list_bl_unhashed()") really had nothing at all to do with
      being unhashed per se, but with a very subtle lifetime rule instead.
      
      So turn all of it around, and make it logical.
      
      Instead of having a DENTRY_UNHASHED bit in d_flags to indicate whether
      the dentry is on the hash chains or not, use the hash chain unhashed
      logic for that.  Suddenly "d_unhashed()" just uses "list_bl_unhashed()",
      and everything makes sense.
      
      And for the lifetime rule, just use an explicit DENTRY_RCUACCEES bit.
      If we ever insert the dentry into the dentry hash table so that it is
      visible to RCU lookup, we mark it DENTRY_RCUACCESS to show that it now
      needs the RCU lifetime rules.  Now suddently that test at dentry free
      time makes sense too.
      
      And because unhashing now is sane and doesn't depend on where the dentry
      got unhashed from (because the dentry hash chain details doesn't have
      some subtle side effects), we can re-unify the __d_drop() logic and use
      common code for the unhashing.
      
      Also fix one more open-coded hash chain bit_spin_lock() that I missed in
      the previous chain locking cleanup commit.
      
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      dea3667b
  2. 21 Mar, 2011 1 commit
  3. 18 Mar, 2011 1 commit
  4. 16 Jan, 2011 3 commits
    • David Howells's avatar
      Allow d_manage() to be used in RCU-walk mode · ab90911f
      David Howells authored
      
      
      Allow d_manage() to be called from pathwalk when it is in RCU-walk mode as well
      as when it is in Ref-walk mode.  This permits __follow_mount_rcu() to call
      d_manage() directly.  d_manage() needs a parameter to indicate that it is in
      RCU-walk mode as it isn't allowed to sleep if in that mode (but should return
      -ECHILD instead).
      
      autofs4_d_manage() can then be set to retain RCU-walk mode if the daemon
      accesses it and otherwise request dropping back to ref-walk mode.
      
      Signed-off-by: default avatarDavid Howells <dhowells@redhat.com>
      Signed-off-by: default avatarAl Viro <viro@zeniv.linux.org.uk>
      ab90911f
    • David Howells's avatar
      Add a dentry op to allow processes to be held during pathwalk transit · cc53ce53
      David Howells authored
      
      
      Add a dentry op (d_manage) to permit a filesystem to hold a process and make it
      sleep when it tries to transit away from one of that filesystem's directories
      during a pathwalk.  The operation is keyed off a new dentry flag
      (DCACHE_MANAGE_TRANSIT).
      
      The filesystem is allowed to be selective about which processes it holds and
      which it permits to continue on or prohibits from transiting from each flagged
      directory.  This will allow autofs to hold up client processes whilst letting
      its userspace daemon through to maintain the directory or the stuff behind it
      or mounted upon it.
      
      The ->d_manage() dentry operation:
      
      	int (*d_manage)(struct path *path, bool mounting_here);
      
      takes a pointer to the directory about to be transited away from and a flag
      indicating whether the transit is undertaken by do_add_mount() or
      do_move_mount() skipping through a pile of filesystems mounted on a mountpoint.
      
      It should return 0 if successful and to let the process continue on its way;
      -EISDIR to prohibit the caller from skipping to overmounted filesystems or
      automounting, and to use this directory; or some other error code to return to
      the user.
      
      ->d_manage() is called with namespace_sem writelocked if mounting_here is true
      and no other locks held, so it may sleep.  However, if mounting_here is true,
      it may not initiate or wait for a mount or unmount upon the parameter
      directory, even if the act is actually performed by userspace.
      
      Within fs/namei.c, follow_managed() is extended to check with d_manage() first
      on each managed directory, before transiting away from it or attempting to
      automount upon it.
      
      follow_down() is renamed follow_down_one() and should only be used where the
      filesystem deliberately intends to avoid management steps (e.g. autofs).
      
      A new follow_down() is added that incorporates the loop done by all other
      callers of follow_down() (do_add/move_mount(), autofs and NFSD; whilst AFS, NFS
      and CIFS do use it, their use is removed by converting them to use
      d_automount()).  The new follow_down() calls d_manage() as appropriate.  It
      also takes an extra parameter to indicate if it is being called from mount code
      (with namespace_sem writelocked) which it passes to d_manage().  follow_down()
      ignores automount points so that it can be used to mount on them.
      
      __follow_mount_rcu() is made to abort rcu-walk mode if it hits a directory with
      DCACHE_MANAGE_TRANSIT set on the basis that we're probably going to have to
      sleep.  It would be possible to enter d_manage() in rcu-walk mode too, and have
      that determine whether to abort or not itself.  That would allow the autofs
      daemon to continue on in rcu-walk mode.
      
      Note that DCACHE_MANAGE_TRANSIT on a directory should be cleared when it isn't
      required as every tranist from that directory will cause d_manage() to be
      invoked.  It can always be set again when necessary.
      
      ==========================
      WHAT THIS MEANS FOR AUTOFS
      ==========================
      
      Autofs currently uses the lookup() inode op and the d_revalidate() dentry op to
      trigger the automounting of indirect mounts, and both of these can be called
      with i_mutex held.
      
      autofs knows that the i_mutex will be held by the caller in lookup(), and so
      can drop it before invoking the daemon - but this isn't so for d_revalidate(),
      since the lock is only held on _some_ of the code paths that call it.  This
      means that autofs can't risk dropping i_mutex from its d_revalidate() function
      before it calls the daemon.
      
      The bug could manifest itself as, for example, a process that's trying to
      validate an automount dentry that gets made to wait because that dentry is
      expired and needs cleaning up:
      
      	mkdir         S ffffffff8014e05a     0 32580  24956
      	Call Trace:
      	 [<ffffffff885371fd>] :autofs4:autofs4_wait+0x674/0x897
      	 [<ffffffff80127f7d>] avc_has_perm+0x46/0x58
      	 [<ffffffff8009fdcf>] autoremove_wake_function+0x0/0x2e
      	 [<ffffffff88537be6>] :autofs4:autofs4_expire_wait+0x41/0x6b
      	 [<ffffffff88535cfc>] :autofs4:autofs4_revalidate+0x91/0x149
      	 [<ffffffff80036d96>] __lookup_hash+0xa0/0x12f
      	 [<ffffffff80057a2f>] lookup_create+0x46/0x80
      	 [<ffffffff800e6e31>] sys_mkdirat+0x56/0xe4
      
      versus the automount daemon which wants to remove that dentry, but can't
      because the normal process is holding the i_mutex lock:
      
      	automount     D ffffffff8014e05a     0 32581      1              32561
      	Call Trace:
      	 [<ffffffff80063c3f>] __mutex_lock_slowpath+0x60/0x9b
      	 [<ffffffff8000ccf1>] do_path_lookup+0x2ca/0x2f1
      	 [<ffffffff80063c89>] .text.lock.mutex+0xf/0x14
      	 [<ffffffff800e6d55>] do_rmdir+0x77/0xde
      	 [<ffffffff8005d229>] tracesys+0x71/0xe0
      	 [<ffffffff8005d28d>] tracesys+0xd5/0xe0
      
      which means that the system is deadlocked.
      
      This patch allows autofs to hold up normal processes whilst the daemon goes
      ahead and does things to the dentry tree behind the automouter point without
      risking a deadlock as almost no locks are held in d_manage() and none in
      d_automount().
      
      Signed-off-by: default avatarDavid Howells <dhowells@redhat.com>
      Was-Acked-by: default avatarIan Kent <raven@themaw.net>
      Signed-off-by: default avatarAl Viro <viro@zeniv.linux.org.uk>
      cc53ce53
    • David Howells's avatar
      Add a dentry op to handle automounting rather than abusing follow_link() · 9875cf80
      David Howells authored
      
      
      Add a dentry op (d_automount) to handle automounting directories rather than
      abusing the follow_link() inode operation.  The operation is keyed off a new
      dentry flag (DCACHE_NEED_AUTOMOUNT).
      
      This also makes it easier to add an AT_ flag to suppress terminal segment
      automount during pathwalk and removes the need for the kludge code in the
      pathwalk algorithm to handle directories with follow_link() semantics.
      
      The ->d_automount() dentry operation:
      
      	struct vfsmount *(*d_automount)(struct path *mountpoint);
      
      takes a pointer to the directory to be mounted upon, which is expected to
      provide sufficient data to determine what should be mounted.  If successful, it
      should return the vfsmount struct it creates (which it should also have added
      to the namespace using do_add_mount() or similar).  If there's a collision with
      another automount attempt, NULL should be returned.  If the directory specified
      by the parameter should be used directly rather than being mounted upon,
      -EISDIR should be returned.  In any other case, an error code should be
      returned.
      
      The ->d_automount() operation is called with no locks held and may sleep.  At
      this point the pathwalk algorithm will be in ref-walk mode.
      
      Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
      added to handle mountpoints.  It will return -EREMOTE if the automount flag was
      set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
      symlinks or mountpoints, -EISDIR if the walk point should be used without
      mounting and 0 if successful.  The path will be updated to point to the mounted
      filesystem if a successful automount took place.
      
      __follow_mount() is replaced by follow_managed() which is more generic
      (especially with the patch that adds ->d_manage()).  This handles transits from
      directories during pathwalk, including automounting and skipping over
      mountpoints (and holding processes with the next patch).
      
      __follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
      automount point with nothing mounted on it.
      
      follow_dotdot*() does not handle automounts as you don't want to trigger them
      whilst following "..".
      
      I've also extracted the mount/don't-mount logic from autofs4 and included it
      here.  It makes the mount go ahead anyway if someone calls open() or creat(),
      tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
      or sticks a '/' on the end of the pathname.  If they do a stat(), however,
      they'll only trigger the automount if they didn't also say O_NOFOLLOW.
      
      I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
      inodes as automount points.  This flag is automatically propagated to the
      dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate().  This saves NFS and could
      save AFS a private flag bit apiece, but is not strictly necessary.  It would be
      preferable to do the propagation in d_set_d_op(), but that doesn't normally
      have access to the inode.
      
      [AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
      succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
      that, rather than just returning with ungrabbed *path]
      
      Signed-off-by: default avatarDavid Howells <dhowells@redhat.com>
      Was-Acked-by: default avatarIan Kent <raven@themaw.net>
      Signed-off-by: default avatarAl Viro <viro@zeniv.linux.org.uk>
      9875cf80
  5. 10 Jan, 2011 1 commit
  6. 07 Jan, 2011 22 commits
    • Nick Piggin's avatar
      fs: implement faster dentry memcmp · 9d55c369
      Nick Piggin authored
      
      
      The standard memcmp function on a Westmere system shows up hot in
      profiles in the `git diff` workload (both parallel and single threaded),
      and it is likely due to the costs associated with trapping into
      microcode, and little opportunity to improve memory access (dentry
      name is not likely to take up more than a cacheline).
      
      So replace it with an open-coded byte comparison. This increases code
      size by 8 bytes in the critical __d_lookup_rcu function, but the
      speedup is huge, averaging 10 runs of each:
      
      git diff st   user   sys   elapsed  CPU
      before        1.15   2.57  3.82      97.1
      after         1.14   2.35  3.61      96.8
      
      git diff mt   user   sys   elapsed  CPU
      before        1.27   3.85  1.46     349
      after         1.26   3.54  1.43     333
      
      Elapsed time for single threaded git diff at 95.0% confidence:
              -0.21  +/- 0.01
              -5.45% +/- 0.24%
      
      It's -0.66% +/- 0.06% elapsed time on my Opteron, so rep cmp costs on the
      fam10h seem to be relatively smaller, but there is still a win.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      9d55c369
    • Nick Piggin's avatar
      fs: improve scalability of pseudo filesystems · 4b936885
      Nick Piggin authored
      
      
      Regardless of how much we possibly try to scale dcache, there is likely
      always going to be some fundamental contention when adding or removing children
      under the same parent. Pseudo filesystems do not seem need to have connected
      dentries because by definition they are disconnected.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      4b936885
    • Nick Piggin's avatar
      fs: dcache per-inode inode alias locking · 873feea0
      Nick Piggin authored
      
      
      dcache_inode_lock can be replaced with per-inode locking. Use existing
      inode->i_lock for this. This is slightly non-trivial because we sometimes
      need to find the inode from the dentry, which requires d_inode to be
      stabilised (either with refcount or d_lock).
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      873feea0
    • Nick Piggin's avatar
      fs: dcache per-bucket dcache hash locking · ceb5bdc2
      Nick Piggin authored
      
      
      We can turn the dcache hash locking from a global dcache_hash_lock into
      per-bucket locking.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      ceb5bdc2
    • Nick Piggin's avatar
      fs: rcu-walk aware d_revalidate method · 34286d66
      Nick Piggin authored
      
      
      Require filesystems be aware of .d_revalidate being called in rcu-walk
      mode (nd->flags & LOOKUP_RCU). For now do a simple push down, returning
      -ECHILD from all implementations.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      34286d66
    • Nick Piggin's avatar
      fs: cache optimise dentry and inode for rcu-walk · 44a7d7a8
      Nick Piggin authored
      
      
      Put dentry and inode fields into top of data structure.  This allows RCU path
      traversal to perform an RCU dentry lookup in a path walk by touching only the
      first 56 bytes of the dentry.
      
      We also fit in 8 bytes of inline name in the first 64 bytes, so for short
      names, only 64 bytes needs to be touched to perform the lookup. We should
      get rid of the hash->prev pointer from the first 64 bytes, and fit 16 bytes
      of name in there, which will take care of 81% rather than 32% of the kernel
      tree.
      
      inode is also rearranged so that RCU lookup will only touch a single cacheline
      in the inode, plus one in the i_ops structure.
      
      This is important for directory component lookups in RCU path walking. In the
      kernel source, directory names average is around 6 chars, so this works.
      
      When we reach the last element of the lookup, we need to lock it and take its
      refcount which requires another cacheline access.
      
      Align dentry and inode operations structs, so members will be at predictable
      offsets and we can group common operations into head of structure.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      44a7d7a8
    • Nick Piggin's avatar
      fs: dcache reduce branches in lookup path · fb045adb
      Nick Piggin authored
      
      
      Reduce some branches and memory accesses in dcache lookup by adding dentry
      flags to indicate common d_ops are set, rather than having to check them.
      This saves a pointer memory access (dentry->d_op) in common path lookup
      situations, and saves another pointer load and branch in cases where we
      have d_op but not the particular operation.
      
      Patched with:
      
      git grep -E '[.>]([[:space:]])*d_op([[:space:]])*=' | xargs sed -e 's/\([^\t ]*\)->d_op = \(.*\);/d_set_d_op(\1, \2);/' -e 's/\([^\t ]*\)\.d_op = \(.*\);/d_set_d_op(\&\1, \2);/' -i
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      fb045adb
    • Nick Piggin's avatar
      fs: dcache remove d_mounted · 5f57cbcc
      Nick Piggin authored
      
      
      Rather than keep a d_mounted count in the dentry, set a dentry flag instead.
      The flag can be cleared by checking the hash table to see if there are any
      mounts left, which is not time critical because it is performed at detach time.
      
      The mounted state of a dentry is only used to speculatively take a look in the
      mount hash table if it is set -- before following the mount, vfsmount lock is
      taken and mount re-checked without races.
      
      This saves 4 bytes on 32-bit, nothing on 64-bit but it does provide a hole I
      might use later (and some configs have larger than 32-bit spinlocks which might
      make use of the hole).
      
      Autofs4 conversion and changelog by Ian Kent <raven@themaw.net>:
      In autofs4, when expring direct (or offset) mounts we need to ensure that we
      block user path walks into the autofs mount, which is covered by another mount.
      To do this we clear the mounted status so that follows stop before walking into
      the mount and are essentially blocked until the expire is completed. The
      automount daemon still finds the correct dentry for the umount due to the
      follow mount logic in fs/autofs4/root.c:autofs4_follow_link(), which is set as
      an inode operation for direct and offset mounts only and is called following
      the lookup that stopped at the covered mount.
      
      At the end of the expire the covering mount probably has gone away so the
      mounted status need not be restored. But we need to check this and only restore
      the mounted status if the expire failed.
      
      XXX: autofs may not work right if we have other mounts go over the top of it?
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      5f57cbcc
    • Nick Piggin's avatar
      fs: rcu-walk for path lookup · 31e6b01f
      Nick Piggin authored
      
      
      Perform common cases of path lookups without any stores or locking in the
      ancestor dentry elements. This is called rcu-walk, as opposed to the current
      algorithm which is a refcount based walk, or ref-walk.
      
      This results in far fewer atomic operations on every path element,
      significantly improving path lookup performance. It also avoids cacheline
      bouncing on common dentries, significantly improving scalability.
      
      The overall design is like this:
      * LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
      * Take the RCU lock for the entire path walk, starting with the acquiring
        of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
        not required for dentry persistence.
      * synchronize_rcu is called when unregistering a filesystem, so we can
        access d_ops and i_ops during rcu-walk.
      * Similarly take the vfsmount lock for the entire path walk. So now mnt
        refcounts are not required for persistence. Also we are free to perform mount
        lookups, and to assume dentry mount points and mount roots are stable up and
        down the path.
      * Have a per-dentry seqlock to protect the dentry name, parent, and inode,
        so we can load this tuple atomically, and also check whether any of its
        members have changed.
      * Dentry lookups (based on parent, candidate string tuple) recheck the parent
        sequence after the child is found in case anything changed in the parent
        during the path walk.
      * inode is also RCU protected so we can load d_inode and use the inode for
        limited things.
      * i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
      * i_op can be loaded.
      
      When we reach the destination dentry, we lock it, recheck lookup sequence,
      and increment its refcount and mountpoint refcount. RCU and vfsmount locks
      are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
      not match, we can not drop rcu-walk gracefully at the current point in the
      lokup, so instead return -ECHILD (for want of a better errno). This signals the
      path walking code to re-do the entire lookup with a ref-walk.
      
      Aside from the final dentry, there are other situations that may be encounted
      where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
      a reference on the last good dentry) and continue with a ref-walk. Again, if
      we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
      using ref-walk. But it is very important that we can continue with ref-walk
      for most cases, particularly to avoid the overhead of double lookups, and to
      gain the scalability advantages on common path elements (like cwd and root).
      
      The cases where rcu-walk cannot continue are:
      * NULL dentry (ie. any uncached path element)
      * parent with d_inode->i_op->permission or ACLs
      * dentries with d_revalidate
      * Following links
      
      In future patches, permission checks and d_revalidate become rcu-walk aware. It
      may be possible eventually to make following links rcu-walk aware.
      
      Uncached path elements will always require dropping to ref-walk mode, at the
      very least because i_mutex needs to be grabbed, and objects allocated.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      31e6b01f
    • Nick Piggin's avatar
      fs: dcache rationalise dget variants · dc0474be
      Nick Piggin authored
      
      
      dget_locked was a shortcut to avoid the lazy lru manipulation when we already
      held dcache_lock (lru manipulation was relatively cheap at that point).
      However, how that the lru lock is an innermost one, we never hold it at any
      caller, so the lock cost can now be avoided. We already have well working lazy
      dcache LRU, so it should be fine to defer LRU manipulations to scan time.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      dc0474be
    • Nick Piggin's avatar
      fs: dcache remove dcache_lock · b5c84bf6
      Nick Piggin authored
      
      
      dcache_lock no longer protects anything. remove it.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      b5c84bf6
    • Nick Piggin's avatar
      fs: Use rename lock and RCU for multi-step operations · 949854d0
      Nick Piggin authored
      
      
      The remaining usages for dcache_lock is to allow atomic, multi-step read-side
      operations over the directory tree by excluding modifications to the tree.
      Also, to walk in the leaf->root direction in the tree where we don't have
      a natural d_lock ordering.
      
      This could be accomplished by taking every d_lock, but this would mean a
      huge number of locks and actually gets very tricky.
      
      Solve this instead by using the rename seqlock for multi-step read-side
      operations, retry in case of a rename so we don't walk up the wrong parent.
      Concurrent dentry insertions are not serialised against.  Concurrent deletes
      are tricky when walking up the directory: our parent might have been deleted
      when dropping locks so also need to check and retry for that.
      
      We can also use the rename lock in cases where livelock is a worry (and it
      is introduced in subsequent patch).
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      949854d0
    • Nick Piggin's avatar
      fs: scale inode alias list · b23fb0a6
      Nick Piggin authored
      
      
      Add a new lock, dcache_inode_lock, to protect the inode's i_dentry list
      from concurrent modification. d_alias is also protected by d_lock.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      b23fb0a6
    • Nick Piggin's avatar
      fs: dcache scale subdirs · 2fd6b7f5
      Nick Piggin authored
      
      
      Protect d_subdirs and d_child with d_lock, except in filesystems that aren't
      using dcache_lock for these anyway (eg. using i_mutex).
      
      Note: if we change the locking rule in future so that ->d_child protection is
      provided only with ->d_parent->d_lock, it may allow us to reduce some locking.
      But it would be an exception to an otherwise regular locking scheme, so we'd
      have to see some good results. Probably not worthwhile.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      2fd6b7f5
    • Nick Piggin's avatar
      fs: dcache scale dentry refcount · b7ab39f6
      Nick Piggin authored
      
      
      Make d_count non-atomic and protect it with d_lock. This allows us to ensure a
      0 refcount dentry remains 0 without dcache_lock. It is also fairly natural when
      we start protecting many other dentry members with d_lock.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      b7ab39f6
    • Nick Piggin's avatar
      fs: dcache scale hash · 789680d1
      Nick Piggin authored
      
      
      Add a new lock, dcache_hash_lock, to protect the dcache hash table from
      concurrent modification. d_hash is also protected by d_lock.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      789680d1
    • Nick Piggin's avatar
      hostfs: simplify locking · ec2447c2
      Nick Piggin authored
      
      
      Remove dcache_lock locking from hostfs filesystem, and move it into dcache
      helpers. All that is required is a coherent path name. Protection from
      concurrent modification of the namespace after path name generation is not
      provided in current code, because dcache_lock is dropped before the path is
      used.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      ec2447c2
    • Nick Piggin's avatar
      fs: change d_hash for rcu-walk · b1e6a015
      Nick Piggin authored
      
      
      Change d_hash so it may be called from lock-free RCU lookups. See similar
      patch for d_compare for details.
      
      For in-tree filesystems, this is just a mechanical change.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      b1e6a015
    • Nick Piggin's avatar
      fs: change d_compare for rcu-walk · 621e155a
      Nick Piggin authored
      
      
      Change d_compare so it may be called from lock-free RCU lookups. This
      does put significant restrictions on what may be done from the callback,
      however there don't seem to have been any problems with in-tree fses.
      If some strange use case pops up that _really_ cannot cope with the
      rcu-walk rules, we can just add new rcu-unaware callbacks, which would
      cause name lookup to drop out of rcu-walk mode.
      
      For in-tree filesystems, this is just a mechanical change.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      621e155a
    • Nick Piggin's avatar
      fs: name case update method · fb2d5b86
      Nick Piggin authored
      
      
      smpfs and ncpfs want to update a live dentry name in-place. Rather than
      have them open code the locking, provide a documented dcache API.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      fb2d5b86
    • Nick Piggin's avatar
      fs: change d_delete semantics · fe15ce44
      Nick Piggin authored
      
      
      Change d_delete from a dentry deletion notification to a dentry caching
      advise, more like ->drop_inode. Require it to be constant and idempotent,
      and not take d_lock. This is how all existing filesystems use the callback
      anyway.
      
      This makes fine grained dentry locking of dput and dentry lru scanning
      much simpler.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      fe15ce44
    • Nick Piggin's avatar
      fs: dcache documentation cleanup · 5eef7fa9
      Nick Piggin authored
      
      
      Remove redundant (and incorrect, since dcache RCU lookup) dentry locking
      documentation and point to the canonical document.
      
      Signed-off-by: default avatarNick Piggin <npiggin@kernel.dk>
      5eef7fa9
  7. 11 Aug, 2010 1 commit
  8. 09 Aug, 2010 1 commit
  9. 15 May, 2010 1 commit
    • Al Viro's avatar
      Fix the regression created by "set S_DEAD on unlink()..." commit · d83c49f3
      Al Viro authored
      
      
      1) i_flags simply doesn't work for mount/unlink race prevention;
      we may have many links to file and rm on one of those obviously
      shouldn't prevent bind on top of another later on.  To fix it
      right way we need to mark _dentry_ as unsuitable for mounting
      upon; new flag (DCACHE_CANT_MOUNT) is protected by d_flags and
      i_mutex on the inode in question.  Set it (with dont_mount(dentry))
      in unlink/rmdir/etc., check (with cant_mount(dentry)) in places
      in namespace.c that used to check for S_DEAD.  Setting S_DEAD
      is still needed in places where we used to set it (for directories
      getting killed), since we rely on it for readdir/rmdir race
      prevention.
      
      2) rename()/mount() protection has another bogosity - we unhash
      the target before we'd checked that it's not a mountpoint.  Fixed.
      
      3) ancient bogosity in pivot_root() - we locked i_mutex on the
      right directory, but checked S_DEAD on the different (and wrong)
      one.  Noticed and fixed.
      
      Signed-off-by: default avatarAl Viro <viro@zeniv.linux.org.uk>
      d83c49f3
  10. 12 Jun, 2009 2 commits
  11. 11 Jun, 2009 1 commit
  12. 27 Mar, 2009 1 commit
  13. 31 Dec, 2008 1 commit
    • Nick Piggin's avatar
      shrink struct dentry · c2452f32
      Nick Piggin authored
      
      
      struct dentry is one of the most critical structures in the kernel. So it's
      sad to see it going neglected.
      
      With CONFIG_PROFILING turned on (which is probably the common case at least
      for distros and kernel developers), sizeof(struct dcache) == 208 here
      (64-bit). This gives 19 objects per slab.
      
      I packed d_mounted into a hole, and took another 4 bytes off the inline
      name length to take the padding out from the end of the structure. This
      shinks it to 200 bytes. I could have gone the other way and increased the
      length to 40, but I'm aiming for a magic number, read on...
      
      I then got rid of the d_cookie pointer. This shrinks it to 192 bytes. Rant:
      why was this ever a good idea? The cookie system should increase its hash
      size or use a tree or something if lookups are a problem. Also the "fast
      dcookie lookups" in oprofile should be moved into the dcookie code -- how
      can oprofile possibly care about the dcookie_mutex? It gets dropped after
      get_dcookie() returns so it can't be providing any sort of protection.
      
      At 192 bytes, 21 objects fit into a 4K page, saving about 3MB on my system
      with ~140 000 entries allocated. 192 is also a multiple of 64, so we get
      nice cacheline alignment on 64 and 32 byte line systems -- any given dentry
      will now require 3 cachelines to touch all fields wheras previously it
      would require 4.
      
      I know the inline name size was chosen quite carefully, however with the
      reduction in cacheline footprint, it should actually be just about as fast
      to do a name lookup for a 36 character name as it was before the patch (and
      faster for other sizes). The memory footprint savings for names which are
      <= 32 or > 36 bytes long should more than make up for the memory cost for
      33-36 byte names.
      
      Performance is a feature...
      
      Signed-off-by: default avatarAl Viro <viro@zeniv.linux.org.uk>
      c2452f32
  14. 23 Oct, 2008 3 commits