This reverts commit 40fe264f. "net/packet: rx_owner_map depends on pg_vec" is not needed since the bug it fixes isn't present.
