Support evaluation in `_TrainingExecutor.run_master()`.
This CL aims to address the following TODO:
# TODO(b/66720832): Once listener API is added into Estimator.train, the
# eval and export process should be wrapped as a listener and passed to
# _start_distributed_training. The expected behavior should be
# 1. The export is invoked after each intermediate evaluation.
# 2. The evaluation and export should be invoked correctly at the end of
# training. This should be fine if the listener works as intended (it will
# send the `after_save` signal for the final ckpt saving).
1. is achieved as follows:
a. saving_evaluators are added to the CheckpointSaverHook's listeners inside the Estimator.
b. MonitoredSession calls after_run() of CheckpointSaverHook, which in turn calls after_save on the listeners.
2. is achieved in a similar way, but when MonitoredSession calls .end() on CheckpointSaverHook.
PiperOrigin-RevId: 170945961
Loading
Please sign in to comment