• Or Kehati's avatar
    IB/addr: Improve address resolution callback scheduling · 346f98b4
    Or Kehati authored
    
    
    Address resolution always does a context switch to a work-queue to
    deliver the address resolution event.  When the IP address is already
    cached in the system ARP table, we're going through the following:
    chain:
    
        rdma_resolve_ip --> addr_resolve (cache hit) -->
    
    which ends up with:
    
        queue_req --> set_timeout (now) --> mod_delayed_work(,, delay=1)
    
    We actually do realize that the timeout should be zero, but the code
    forces it to a minimum of one jiffie.
    
    Using one jiffie as the minimum delay value results in sub-optimal
    scheduling of executing this work item by the workqueue, which on the
    below testbed costs about 3-4ms out of 12ms total time.
    
    To fix that, we let the minimum delay to be zero.  Note that the
    connect step times change too, as there are address resolution calls
    from that flow.
    
    The results were taken from running both client and server on the
    same node, over mlx4 RoCE port.
    
    before -->
    step              total ms     max ms     min us  us / conn
    create id    :        0.01       0.01       6.00       6.00
    resolve addr :        4.02       4.01    4013.00    4016.00
    resolve route:        0.18       0.18     182.00     183.00
    create qp    :        1.15       1.15    1150.00    1150.00
    connect      :        6.73       6.73    6730.00    6731.00
    disconnect   :        0.55       0.55     549.00     550.00
    destroy      :        0.01       0.01       9.00       9.00
    
    after -->
    step              total ms     max ms     min us  us / conn
    create id    :        0.01       0.01       6.00       6.00
    resolve addr :        0.05       0.05      49.00      52.00
    resolve route:        0.21       0.21     207.00     208.00
    create qp    :        1.10       1.10    1104.00    1104.00
    connect      :        1.22       1.22    1220.00    1221.00
    disconnect   :        0.71       0.71     713.00     713.00
    destroy      :        0.01       0.01       9.00       9.00
    Signed-off-by: default avatarOr Kehati <ork@mellanox.com>
    Signed-off-by: default avatarOr Gerlitz <ogerlitz@mellanox.com>
    Acked-by: default avatarSean Hefty <sean.hefty@intel.com>
    Signed-off-by: default avatarRoland Dreier <roland@purestorage.com>
    346f98b4
addr.c 13.5 KB