Revert "ASoC: soc-pcm: Move debugfs removal out of spinlock"

This reverts commit 44b685c4 which is
commit 9f620684 upstream.

It is part of a series of ASoC patches that break the Android abi.  If
they are needed in the future they can be brought back in an ABI-safe
way.

Bug: 161946584
Change-Id: I33f0193b06e850bfb8f5d5ef30541f86f38e3161
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>