Implementing ghost batch norm as defined in https://arxiv.org/pdf/1705.08741.
Reuses most of tf.layers.batch_normalization's existing functionality by using some reshaping and transposing tricks. Toggled via additional optional parameter `num_virtual_batches`. Ghost batch norm is essential for large batch training where the true batch size is different than the batch norm batch size. PiperOrigin-RevId: 170368495
Loading
Please sign in to comment