Do not generate NaN when backpropagating through operators with singularities...
Do not generate NaN when backpropagating through operators with singularities at the origin if the input from the layer above is zero. Example: y = sqrt(x=0) = 0, dx/dy = -1/sqrt(x) * dy = -inf * 0 = NaN if dy=0. To address the problem, we replace division (or multiplication) with an op that always returns 0 if dy is zero. This change is one of several, and addresses the issue in log, log1p, sqrt_grad, tan, acosh, asin, acos, atan2, div, pow. PiperOrigin-RevId: 238664017
Loading
Please sign in to comment