Fix polynomial decay with cycle for global step=0
For polynomial decay with cycle=True the learning rate at step 0 becomes NaN, because in the process of calculating it we devide by 0. This change should fix it, by setting the multiplier for the decay steps to one for global_step=0.
Loading
Please sign in to comment