Fix logics for floating-point comparision.
Set the acceptable error range based on both absolute tolerance and relative tolerance. Currently, absolute tolerance is set to 1e-5 for FP32 and 5 epsilon (~5e-3) for FP16 relaxed computation. The relative tolerance is set to 5ULP of the corresponding precision. Add a TODO mark for potential future adjustment on error limit based on testing. Bug: 111768023 Test: NeuralNetworksTest_static Change-Id: I8052d979bc45a83dba4c69ef20b8a740ae640953 Merged-In: I8052d979bc45a83dba4c69ef20b8a740ae640953 (cherry picked from commit 3afdbda56805b6ca1d8ceec66f861c57db616778)
Loading
Please sign in to comment