f32 have an 8-bit exponent. However, while
f32 has a 23-bit
bf16 has only a 7-bit one, keeping only the most significant bits.
As a result, while these data types support a very close numerical range of
bf16 has a significantly reduced precision. Therefore,
a spot between
f32 and the IEEE 16-bit half-precision floating point data
f16. Compared directly to
f16, which has a 5-bit exponent and a
bf16 trades increased range for reduced precision.
More details of the bfloat16 data type can be found here.
The main difference between implementing training with the
f32 data type and
bf16 data type is the way the weights updates are treated. With the
f32 data type, the weights gradients have the same data type as the weights
themselves. This is not necessarily the case with the
bf16 data type as oneDNN
allows some flexibility here. For example, one could maintain a master copy of
all the weights, computing weights gradients in
f32 and converting the result
Most of the primitives can support the
bf16 data type for source and weights
tensors. Destination tensors can be specified to have either the
data type. The latter is intended for cases in which the output is to be fed to
operations that do not support bfloat16 or require higher precision.