Support the new extended rnn ops of CUDNN, which can accept sequence_lengths...

Support the new extended rnn ops of CUDNN, which can accept sequence_lengths as a parameter. (Still contain many debugging codes; to be removed after correctness check)

Use async memcpy when copying sequence lengths data from GPU to CPU (which is required when creating cudnnRNNDataDescriptor_t

Remove the debugging prints

Reorder the parameters for better back compatibility

Use sequence_lengths as the new parameter

New unittest for the extend cudnn rnn ops added

Unittest for the extended cuDNN-RNN (supporting variable sequences) ops added; remove some debugging statements in its grad ops

Add user-inputs checks (int array required for sequence_lengths); Changed new ops name to CudnnRNNVarSeqLen

Add pbtxt files for VarSeqLen RNN APIs

Rename functions/classes/variables to replace Ex/ex to VarSeqLen/var_seq_len

Fix formatting

Enable the bazel test for the new var_len_seq ops

Minor changes from self.built to self._built with a initializer

Google Feedback: remove the manual cudaMemcpy() in the descriptor class

Google Feedback: remove the new rnn descriptor class and augment the existing one

Google Feedback: renaming the VarSeqLen to V3

Google Feedback: removing unnecessary spaces

Google feedback: redesign the cudnn_rnn_ops_test.py to include the v3 ops

Remove previous cudnn_rnn_ops_var_seq_len_test.py

Test Macro branches (unfinished)

Change the version number back to 7.2.1 for new cudnn rnn ops

add a dedicated ExtractForwardInput to process sequence lengths; seq_length -> max_seq_length; raise error if cudnn_version < 7.2.1 and is_var_seq_lengths()==true

add 'host_reserved' to v3; change v3 to subclass CudnnRNNForwardOp/CudnnRNNBackwardOp

Used 'absl::Span' instead of passing an int pointer

Changed 'seq_length' to 'max_seq_length' and 'seq_lens' to 'seq_lengths'

Changed to use only one omnipotent constructor to create 'CudnnRnnSequenceTensorDescriptor'

Separated 'CudnnSupport::createRnnSequenceTensorDescriptor()' for the extra 'seq_lengths' param. Other corresponding wrappers are changed as well.

Removed branches to check the sequence_lengths param; Added some comments on autotuning for V3

Changed the order of sequence_lengths for _cudnn_rnn()

Made the 'ExtractForwardInput()' share codes with the original function

Changed the order of 'sequence_lengths' for all classes/ops; Changed '_var_seq_len' to 'v3'

Updated test cases of new cudnn RNN API using 'ExpandNamedTestCases'

Changed to use unified 'CreateForwardAndBackwardIODescriptors' and 'DoForward' and 'DoBackward'

Changed the order of 'sequence_lengths' in '_forward()'; Changed all 'equal to' to 'equal'

Made Span argument to const reference; Removed seq_lengths_ field in 'CudnnRnnSequenceTensorDescriptor'

Changed handle() to data_handle()

Some sentences changed and V3 shape test added

Changed orders of 'sequence_lengths'; Changed the way to get pointer from Tensor

Changed the ops register

Changed to use seq_length for v1 and v2

Fixed some formating issues of python scripts and syntax issue in cudnn_rnn_ops_test.cc

Added macro check over the RNNDataDescriptorDeleter

Changed the init order to be in the order of declaration

Turn RNNDataDescriptor to be void pointer when cuDNN version < 7.2.1

Avoid the RNNDataDescriptor when cudnn < 7.2.1

Delete the unreached return

Another place needs to avoid the RNNDataDescriptor

Avoid data_handle() function when cudnn < 7.2.1