Commit 82c1af4a authored by A. Unique TensorFlower's avatar A. Unique TensorFlower Committed by TensorFlower Gardener
Browse files

Simplify and improve AdaptiveSharedBatchScheduler implementation. The new...

Simplify and improve AdaptiveSharedBatchScheduler implementation. The new implementation will exist alongside the old one (selectable through the scheduler options) until its superiority is confirmed, at which point the old rate-based implementation will be removed.

The new implementation requires fewer options and no user feedback to achieve a low latency batching. Instead of processing batches at an adjustable rate, we limit the number of batches which can be concurrently processed. Below the limit, batches are immediately processed upon creation. At the limit, the oldest batch is processed once an in-processing batch finishes.  The scheduler continuously adjusts the limit in order to maintain the smallest overall latency.

PiperOrigin-RevId: 178960621
parent 38b20f83
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment