[XLA] Split CPU and GPU fastmath into separate flags.
The motivation for this change is to allow us to change the default for CPU and GPU independently. In particular, we want to disable fastmath on both platforms, but I think it will be much easier on GPU, and I don't want that blocked on getting things perfect on CPU. The primary motivation for turning off fastmath is that on GPU with f16, model authors may use automatic loss scaling, which looks for NaNs in the loss and tweaks the loss scale factor. If fastmath is enabled, the compiler is free to produce any fp value instead of NaN, so this is unsound. The reason I expect this will be easier to do without regression on GPU is that on CPU some of the important vectorization we do relies on fastmath being enabled, but we have nothing of the sort on GPU. No functional change, since this patch leaves the defaults as they currently are (fastmath on for both platforms). PiperOrigin-RevId: 208502191
Loading
Please sign in to comment