Add RDMA verbs configuration (#13564)

* RDMA device and port configuration added

* RDMA pkey, q_depth, timeout, retry_cnt and sl configuration added

* RDMA mtu configuration added + chage tab to 2 spaces

* Add RDMA_GID_INDEX to configure verbs

Signed-off-by: dariavel <daria@mellanox.com>

* No need in device, context is enough

Signed-off-by: dariavel <daria@mellanox.com>

* Add PORT check during DEVICE setting + some minor refactoring

Signed-off-by: dariavel <daria@mellanox.com>

* RDMA mtu configuration change->check value before channel state

* Typo fix

Signed-off-by: dariavel <daria@mellanox.com>

* Allow GID index without RoCE v2 to run but with a warning

Signed-off-by: dariavel <daria@mellanox.com>

* Add RDMA configuration description

* Post code review fixes and function headers

Signed-off-by: dariavel <daria@mellanox.com>

* create 1 set_param function instead of many functions

Signed-off-by: Noa Ezra <noae@mellanox.com>

* add error if sl value is larger than 7

Signed-off-by: Noa Ezra <noae@mellanox.com>

* Update DEARME. Fix spacing and remove useless query

Signed-off-by: dariavel <daria@mellanox.com>

* Fix spacing, indentation and port_num bug

Signed-off-by: Noa Ezra <noae@mellanox.com>

* Improve GID index checks

Signed-off-by: dariavel <daria@mellanox.com>

* fix bug in set_device

Signed-off-by: Noa Ezra <noae@mellanox.com>

* INFO instead of error in case of Unknown port link layer

Signed-off-by: Noa Ezra <noae@mellanox.com>

* Add traffic class configuration

Signed-off-by: dariavel <daria@mellanox.com>

* Clang code formating

Signed-off-by: dariavel <daria@mellanox.com>

* Cut long line to 80 caracters

Signed-off-by: dariavel <daria@mellanox.com>

* Code styling

Signed-off-by: dariavel <daria@mellanox.com>