Commit 4ddf1bbd authored by Xusong Wang's avatar Xusong Wang Committed by Przemyslaw Szczepaniak
Browse files

Fix logics for floating-point comparision.

Set the acceptable error range based on both absolute tolerance and
relative tolerance.

Currently, absolute tolerance is set to 1e-5 for FP32 and 5 epsilon
(~5e-3) for FP16 relaxed computation. The relative tolerance is set to
5ULP of the corresponding precision. Add a TODO mark for potential
future adjustment on error limit based on testing.

Bug: 111768023

Test: NeuralNetworksTest_static
Change-Id: I8052d979bc45a83dba4c69ef20b8a740ae640953
Merged-In: I8052d979bc45a83dba4c69ef20b8a740ae640953
(cherry picked from commit 3afdbda56805b6ca1d8ceec66f861c57db616778)
parent 0a27020c
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment