clk: at91: avoid sleeping early

commit 658fd65c upstream.

It is not allowed to sleep to early in the boot process and this may lead
to kernel issues if the bootloader didn't prepare the slow clock and main
clock.

This results in the following error and dump stack on the AriettaG25:
   bad: scheduling from the idle thread!

Ensure it is possible to sleep, else simply have a delay.

Reported-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Link: https://lkml.kernel.org/r/20190920153906.20887-1-alexandre.belloni@bootlin.com


Fixes: 80eded6c ("clk: at91: add slow clks driver")
Tested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
[bwh: Backported to 3.16:
 - Drop changes in clk_sama5d4_slow_osc_prepare()
 - Adjust filename, context]
Signed-off-by: Ben Hutchings <ben@decadent.org.uk>