Add IndRNN, IndyGRU and IndyLSTM cells.
These are similar to regular RNN, GRU and LSTM nodes, except that, following the ideas in https://arxiv.org/abs/1803.04831, each node only sees its own state, and not all the states in the same layer. The number of weights for a layer of IndRNN, IndyGRU, and IndyLSTM with input width "m" and output width "n" is n*(m+2), 3*n*(m+2), and 4*n*(m+2) as opposed to n*(m+n+1), 3*n*(m+n+1), and 4*n*(m+n+1) for regular RNN, GRU, and LSTM layers respectively. The computational costs are similarly reduced by replacing an O(n^2) matrix-vector multiplication by an O(n) element-wise vector multiplication. PiperOrigin-RevId: 203932335
Loading
Please sign in to comment