Commit 346f98b4 authored by Or Kehati's avatar Or Kehati Committed by Roland Dreier
Browse files

IB/addr: Improve address resolution callback scheduling



Address resolution always does a context switch to a work-queue to
deliver the address resolution event.  When the IP address is already
cached in the system ARP table, we're going through the following:
chain:

    rdma_resolve_ip --> addr_resolve (cache hit) -->

which ends up with:

    queue_req --> set_timeout (now) --> mod_delayed_work(,, delay=1)

We actually do realize that the timeout should be zero, but the code
forces it to a minimum of one jiffie.

Using one jiffie as the minimum delay value results in sub-optimal
scheduling of executing this work item by the workqueue, which on the
below testbed costs about 3-4ms out of 12ms total time.

To fix that, we let the minimum delay to be zero.  Note that the
connect step times change too, as there are address resolution calls
from that flow.

The results were taken from running both client and server on the
same node, over mlx4 RoCE port.

before -->
step              total ms     max ms     min us  us / conn
create id    :        0.01       0.01       6.00       6.00
resolve addr :        4.02       4.01    4013.00    4016.00
resolve route:        0.18       0.18     182.00     183.00
create qp    :        1.15       1.15    1150.00    1150.00
connect      :        6.73       6.73    6730.00    6731.00
disconnect   :        0.55       0.55     549.00     550.00
destroy      :        0.01       0.01       9.00       9.00

after -->
step              total ms     max ms     min us  us / conn
create id    :        0.01       0.01       6.00       6.00
resolve addr :        0.05       0.05      49.00      52.00
resolve route:        0.21       0.21     207.00     208.00
create qp    :        1.10       1.10    1104.00    1104.00
connect      :        1.22       1.22    1220.00    1221.00
disconnect   :        0.71       0.71     713.00     713.00
destroy      :        0.01       0.01       9.00       9.00
Signed-off-by: default avatarOr Kehati <ork@mellanox.com>
Signed-off-by: default avatarOr Gerlitz <ogerlitz@mellanox.com>
Acked-by: default avatarSean Hefty <sean.hefty@intel.com>
Signed-off-by: default avatarRoland Dreier <roland@purestorage.com>
parent 514f3ddf
......@@ -176,8 +176,8 @@ static void set_timeout(unsigned long time)
unsigned long delay;
delay = time - jiffies;
if ((long)delay <= 0)
delay = 1;
if ((long)delay < 0)
delay = 0;
mod_delayed_work(addr_wq, &work, delay);
}
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment