Do not generate NaN when backpropagating through operators with singularities...
Do not generate NaN when backpropagating through operators with singularities at the origin if the input from the layer above is zero. Example: y = sqrt(x=0) = 0, dy/dy = -1/sqrt(x) * dy = -inf * 0 = NaN if dy=0. To address the problem, we replace division (or multiplication) with an op that always returns 0 if dy is zero. This change is the first of several, and addresses the issue in reciprocal, sqrt, and rsqrt. PiperOrigin-RevId: 238463599
Loading
Please sign in to comment