Commit d75edc93 authored by Ayush Dubey's avatar Ayush Dubey Committed by Gunhan Gulsoy
Browse files

Fix synchronization across callbacks in collective params initialization.

During initialization of local collective params, we may issue RPCs to other
workers in order to obtain device localities.  Currently, we hold a mutex
across these RPCs, but we do not ensure that the thread that unlocks the mutex
is the same as the one that locked it.

This change releases the mutex (InstanceRec::out_mu) before calling
GetDeviceLocalitiesAsync.  Before releasing out_mu, it marks the mutex
unavailable.  Any thread that wishes to acquire out_mu must wait on a condition
variable if the mutex is unavailable.  The callback for
GetDeviceLocalitiesAsync marks the mutex as available again and notifies the
condition variable.

PiperOrigin-RevId: 202346357
parent 6d5668ab
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment