Avoid reading the input file twice for InitializableLookupTable in combination with HashTable.
Before this cl, TextFileLineIterator::total_size() was called for HashTable::DoPrepare, even though HashTable::DoPrepare ignores the size parameter. In order to have a result ready for TextFileLineIterator::total_size(), Init() called GetNumLinesInTextFile(), which read the whole file. Just to throw away the result :-/ This cl: - adds a DoLazyPrepare, that gets a functor to get the size, only if needed. - add HashTable::DoLazyPrepare which does not call this functor. - modify TextFileLineIterator::Init() to not call GetNumLinesInTextFile() anymore, when vocab_size was given as -1. - modify TextFileLineIterator::total_size() to call GetNumLinesInTextFile() lazily on the first call, if vocab_size_ was passed as -1. PiperOrigin-RevId: 190593744
Loading
Please sign in to comment