net/mlx5e: TX, Utilize WQ fragments edge for multi-packet WQEs

For simplicity reasons, the driver avoids crossing work queue fragment
boundaries within the same TX WQE (Work-Queue Element). Until today, as
the number of packets in a TX MPWQE (Multi-Packet WQE) descriptor is not
known in advance, the driver pre-prepared contiguous memory for the
largest possible WQE. For this, when getting too close to the fragment
edge, having no room for the largest WQE possible, the driver was
filling the fragment remainder with NOP descriptors, aligning the next
descriptor to the beginning of the next fragment.

Generating and handling these NOPs wastes resources, like: CPU cycles,
work-queue entries fetched to the device, and PCI bandwidth.

In this patch, we replace this NOPs filling mechanism in the TX MPWQE
flow. Instead, we utilize the remaining entries of the fragment with a
TX MPWQE. If this room turns out to be too small, we simply open an
additional descriptor starting at the beginning of the next fragment.

Performance benchmark:
uperf test, single server against 3 clients.
TCP multi-stream, bidir, traffic profile "2x350B read, 1400B write".
Bottleneck is in inbound PCI bandwidth (device POV).

+---------------+------------+------------+--------+
|               | Before     | After      |        |
+---------------+------------+------------+--------+
| BW            | 117.4 Gbps | 121.1 Gbps | +3.1%  |
+---------------+------------+------------+--------+
| tx_packets    | 15 M/sec   | 15.5 M/sec | +3.3%  |
+---------------+------------+------------+--------+
| tx_nops       | 3  M/sec   | 0          | -100%  |
+---------------+------------+------------+--------+

Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
Reviewed-by: Dragos Tatulea <dtatulea@nvidia.com>
Reviewed-by: Simon Horman <horms@kernel.org>
Link: https://patch.msgid.link/1742391746-118647-1-git-send-email-tariqt@nvidia.com


Signed-off-by: Jakub Kicinski <kuba@kernel.org>