Revert "USB: core: only clean up what we allocated"

commit cf4df407 upstream.

This reverts commit 32fd87b3.

Alan wrote a better fix for this...

Cc: Andrey Konovalov <andreyknvl@google.com>
Cc: stable <stable@vger.kernel.org>
Cc: Nathan Chancellor <natechancellor@gmail.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Lee Jones <joneslee@google.com>
Change-Id: I2d00046a35a07d472fbd2762512150b18ca5635a