Override GPU order at each worker via ConfigProto for CollectiveOps.
While collectives includes logic for automatically deducing a good ring order among devices on each worker, after this change the user can override the ring order via the session configuration. If specified, this defines the order of GPU devices along which data is exchanged at each worker. This assumes that the GPU topology is identical at each worker, and hence the same ring order is good for all workers. The ring order in ConfigProto is used for all collective ops in the session. Multiple session.run calls will reuse the same ring order. PiperOrigin-RevId: 219510944
Loading
Please sign in to comment