perf: Do not enable by default during compile testing

Enabling the compile test should not cause automatic enabling of all
drivers, but only allow to choose to compile them.

Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
Link: https://lore.kernel.org/r/20250417074650.81561-1-krzysztof.kozlowski@linaro.org


Signed-off-by: Will Deacon <will@kernel.org>