We don't really need to recalculate the effective rate of a clock when a
per-user clock is removed, if the constraints of the later aren't
limiting the requested rate.

This was causing problems with clocks that never had a rate set before,
as rate_req would be zero. Though this could be considered a bug in the
implementation of those clocks, this should be checked somewhere else.

Fixes: 1c8e6004

 ("clk: Add rate constraints to clocks")
Cc: Thierry Reding <thierry.reding@gmail.com>
Cc: Peter De Schrijver <pdeschrijver@nvidia.com>
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Signed-off-by: Michael Turquette <mturquette@linaro.org>