RDMA/device: Return error earlier if port in not valid

There is no need to allocate port data if port provided is not valid.

Fixes: c2261dd7 ("RDMA/device: Add ib_device_set_netdev() as an alternative to get_netdev")
Link: https://lore.kernel.org/r/022047a8b16988fc88d4426da50bf60a4833311b.1719235449.git.leon@kernel.org


Signed-off-by: Leon Romanovsky <leonro@nvidia.com>