This reverts commit bc8bbb77.

This commit breaks U-Boot booting on SiFive Unleashed board, as
there is no such CSR on U54 core.

Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Leo Yu-Chi Liang <ycliang@andestech.com>