Reuse GPUs for multiple ranks in NcclManager test.
Since NCCL 2.1.2, NCCL uses cooperative group launch by default which results in deadlocks in the NcclManager test if number of ranks > number of GPUs. Setting NCCL_LAUNCH_MODE to PARALLEL, which was the default before NCCL 2.1.2, fixes this. PiperOrigin-RevId: 224531851
Loading
Please sign in to comment