Address resolution always does a context switch to a work-queue to
deliver the address resolution event.  When the IP address is already
cached in the system ARP table, we're going through the following:
chain:

    rdma_resolve_ip --> addr_resolve (cache hit) -->

which ends up with:

    queue_req --> set_timeout (now) --> mod_delayed_work(,, delay=1)

We actually do realize that the timeout should be zero, but the code
forces it to a minimum of one jiffie.

Using one jiffie as the minimum delay value results in sub-optimal
scheduling of executing this work item by the workqueue, which on the
below testbed costs about 3-4ms out of 12ms total time.

To fix that, we let the minimum delay to be zero.  Note that the
connect step times change too, as there are address resolution calls
from that flow.

The results were taken from running both client and server on the
same node, over mlx4 RoCE port.

before -->
step              total ms     max ms     min us  us / conn
create id    :        0.01       0.01       6.00       6.00
resolve addr :        4.02       4.01    4013.00    4016.00
resolve route:        0.18       0.18     182.00     183.00
create qp    :        1.15       1.15    1150.00    1150.00
connect      :        6.73       6.73    6730.00    6731.00
disconnect   :        0.55       0.55     549.00     550.00
destroy      :        0.01       0.01       9.00       9.00

after -->
step              total ms     max ms     min us  us / conn
create id    :        0.01       0.01       6.00       6.00
resolve addr :        0.05       0.05      49.00      52.00
resolve route:        0.21       0.21     207.00     208.00
create qp    :        1.10       1.10    1104.00    1104.00
connect      :        1.22       1.22    1220.00    1221.00
disconnect   :        0.71       0.71     713.00     713.00
destroy      :        0.01       0.01       9.00       9.00

Signed-off-by: Or Kehati <ork@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Acked-by: Sean Hefty <sean.hefty@intel.com>
Signed-off-by: Roland Dreier <roland@purestorage.com>