Makes inference efficient.

4 times faster than the current code, with more speed to come with
models that allow better feature caching.

Exported by export_to_aosp.sh from Google3 reviewed code.

Test: Built and tested on device. Google3 unit and regression tests
pass.

Bug: 36885469

Change-Id: I75348a0c9da917dc3b2979b63135411d4267bb95