It speeds up loading kernel from SD or USB a lot.

Signed-off-by: Vasily Khoruzhick <anarsoul@gmail.com>
Acked-by: Marek Vasut <marex@denx.de>