Fix synchronization across callbacks in collective params initialization.
During initialization of local collective params, we may issue RPCs to other workers in order to obtain device localities. Currently, we hold a mutex across these RPCs, but we do not ensure that the thread that unlocks the mutex is the same as the one that locked it. This change releases the mutex (InstanceRec::out_mu) before calling GetDeviceLocalitiesAsync. Before releasing out_mu, it marks the mutex unavailable. Any thread that wishes to acquire out_mu must wait on a condition variable if the mutex is unavailable. The callback for GetDeviceLocalitiesAsync marks the mutex as available again and notifies the condition variable. PiperOrigin-RevId: 202346357
Loading
Please sign in to comment