tfdbg: improve file paths & causality check under suboptimal timestamp resolution
Older versions of Windows (e.g., Windows 7) have lower temporal resolution in timestamps than other platforms. On such Windows machines, multiple dumps of the same or different tensors may end up with the same timestamp, causing 1) failures in the causality check in the constructor of DebugDumpDir, 2) occasional overwriting of previous tensor dumps and core metadata files, 3) loss of ordering information for debug watches of a tensor that is dumped multiple times during a Session::Run() call, e.g., due to While loops. This CL addresses those two issues: 1) DebugDumpDir._validate_dump_with_graphs now examines tensor dumps of the same timestamp to take into account the possibility of timestamp collision during dependency validation. 2) During the dumping of tensors, file path collisions are checked and if they occur, an index suffix (e.g., "-1", "-2") is appended to the file path. 3) The core metadata Events files now include session_run_count in their names, to avoid collision due to same timestamps. Change: 150670298
Loading
Please sign in to comment