Revert "mfd: slimslave: Audio recording doesn't work sometimes"

This reverts commit 62e08ad7.

The patch has a side-effect and causes the audio loss during
device switch

Signed-off-by: Jay Wang <jaywang@codeaurora.org>