Make TensorBoard histogram processing 3x faster
We discovered that on one of our datasets, the TensorBoard backend was spending 80% of its time processing histograms. Particularly the histogram compression routine, which did a lot of math in pure Python on raw histograms which are potentially quite large considering the C++ Histogram class uses a default bucket count of 774. These routines have now been optimized so that the _ProcessHistogram method of EventAccumulator and its subroutines take 39% of the time they did previously. This has been accomplished by doing the following: - Use numpy fast vector math to compute cumulative sum - Use numpy to binary search compression points in cumulative sum - Only convert histo.bucket to a list once rather than twice Change: 132285757
Loading
Please sign in to comment