Explicitly place local_step variable on worker's local device (was defaulting to PS).
The local step is used by the ConditionalAccumulators to check gradients for staleness and optionally discard them. This fixes the issue if that the local step is increased by one worker and a backup worker pushes a stale gradient to a ConditionalAccumulator, then the gradient will be accepted (and hence, training won't be synchronous anymore.) PiperOrigin-RevId: 156417310
Loading
Please sign in to comment