mfd: slimslave: Audio recording doesn't work sometimes

Slimbus ports takes some more time for the port disconnect
to happen. After slimbus port close is called, the time taken
for the actual hardware ports to get closed is more than
expected.

Change-Id: I5b0b2bc7056e80950b76eae9053016cf563ec1d2
CRs-fixed: 364943
Signed-off-by: Swaminathan Sathappan <Swami@codeaurora.org>