Commit 8e535474 authored by Przemyslaw Szczepaniak's avatar Przemyslaw Szczepaniak
Browse files

Calculate quantized convolution multiplier using double type.

NNAPI has been using the float type for quantized convolution multiplier
calculation, which was causing an occasional off-by-one differences for
convolution operation results between TFLite/CPU and NNAPI/CPU implementations.

Test: MLTS com.android.nn.benchmark.app.NNTest#testNNAPI[mobilenet_*_quant] error rates
Test: NeuralNetworksTest_static
Bug: 115377572
Change-Id: If0bb9ff7229f5b3864a69c66566a4e4008f94c37
parent 33c1e053
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment