Revert "video/aperture: Only remove sysfb on the default vga pci device"

This reverts commit 485ec8f8 which is
commit 5ae3716c upstream.

It breaks the Android ABI and can be brought back later in an abi-safe
way if needed.

Bug: 161946584
Change-Id: Iec66232fff97b07fd0c57975e8b1785710cfb395
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>