Commit b4ae5ba3 authored by Rachel Lim's avatar Rachel Lim Committed by TensorFlower Gardener
Browse files

[tf.data] Add a ChooseFastestDataset, which will allow our optimizations to...

[tf.data] Add a ChooseFastestDataset, which will allow our optimizations to dynamically pick between two equivalent input datasets

- For the first N iterations, run two threads simultaneously that each pulls from one of the inputs. The threads also measure time taken. Wait for both threads to complete before returning.
- After N iterations, decide which dataset is faster and just use that.

There are more sophisticated implementations that could be better, e.g. instead of waiting for both threads to complete, return when the faster of them is finished. However, the code here is significantly simpler (I also implemented the other one elsewhere), at the cost of slower iterations for the first N iterations -- the iteration time for the first N iters is max(input_a, input_b). Open to ideas.

PiperOrigin-RevId: 229586112
parent b59aa4b8
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment