Talk:Bfloat16 floating-point format

Error in bit layout illustration

The file Bfloat16_format.svg shows the wrong bit indices (first field is 0-7 when it should be 0-6). I tried uploading a newer version with that fixed. It shows up. But the SVG itself is the old version.

Maybe it's because I used the exact same filename. I still believe this behavior is incorrect. Tried changing the filename and reuploading. That version is not even showing up. Now I can't even upload newer versions. Maybe this was flagged for moderation.

I don't know. I give up. Uploading experience is really bad. --nachokb (talk) 09:21, 27 September 2018 (UTC)

It seems to be correct, now. I suppose that your issue was due to caching. Vincent Lefèvre (talk) 09:49, 27 September 2018 (UTC)

History

It would be good to have a section explaining where this format was introduced and a history of its adoption. — Steven G. Johnson (talk) 20:04, 3 June 2021 (UTC)

Add additional information about rounding

• Specific text to be added or removed

1. Use Shortened instead of Truncated

To avoid confusion with the rounding mechanism, the first change is to replace “truncated” with “shortened” when it comes to describing bfloat16. “The bfloat16 format, being a shortened IEEE 754 single-precision 32-bit float, …”. In such a way, the text won’t be misunderstood with the truncation rounding in conversion.

2. Add a Format Conversion Section Add a format conversion section to detail the rounding mechanism. The proposed text is as follows:

Rounding and Conversion

The most common use case is the conversion between binary32 and bfloat16. The following section describes the conversion process and its rounding scheme in the conversion. Note that there are other possible scenarios of format conversions to or from bfloat16. For example, int16 and bfloat16.

a. From IEEE754 binary32 (32-bit floating point) to bfloat16

When bfloat16 was first introduced as a storage format[1], the conversion from IEEE754 binary32 (32-bit floating point) to bfloat16 is truncation (round-to-zero). Later on, when it becomes the input of matrix multiplication units, the conversion can have various rounding mechanisms. Unfortunately, the rounding mechanisms supported by the hardware are platform-dependent, due to the lack of an industry standard. For example, Google TPU uses round-to-nearest-even[2]; ARM uses round-to-nearest-odd; NVIDIA supports four rounding schemes outlined in IEEE754, including round-to-nearest-even.

b. From bfloat16 to IEEE754 binary32 (32-bit floating point)

Since IEEE binary32 can represent all exact values in bfloat16, the conversion simply pads 16 zeros in the mantissa bits.

[1] Joshua V. Dillon, Ian Langmore, Dustin Tran, Eugene Brevdo, Srinivas Vasudevan, Dave Moore, Brian Patton, Alex Alemi, Matt Hoffman, Rif A. Saurous (2017-11-28). TensorFlow Distributions (Report). arXiv:1711.10604. Bibcode:2017arXiv171110604D. Accessed 2018-05-23. All operations in TensorFlow Distributions are numerically stable across half, single, and double floating-point precisions (as TensorFlow dtypes: tf.bfloat16 (truncated floating point), tf.float16, tf.float32, tf.float64). Class constructors have a validate_args flag for numerical asserts

[2] Google The bfloat16 numerical format https://cloud.google.com/tpu/docs/bfloat16

[3] ARM bfloat16-processing-for-neural-networks-on-armv8_2d00_a https://community.arm.com/arm-community-blogs/b/ai-and-ml-blog/posts/bfloat16-processing-for-neural-networks-on-armv8_2d00_a

[4] NVIDIA CUDA devtools: 1.3.5. Bfloat16 Precision Conversion and Data Movementhttps://docs.nvidia.com/cuda/cuda-math-api/group__CUDA__MATH____BFLOAT16__MISC.html#group__CUDA__MATH____BFLOAT16__MISC_1gf532ce241c3f0b983136d3b130ce0cf3

• Reason for the change: To reflect the recent status that the industry vendors have supported various rounding mechanisms other than truncation.
• References supporting change: NVIDIA, ARM

2A00:79E1:ABC:12B:BDC7:5327:147A:E174 (talk) 22:09, 18 July 2023 (UTC)

Disclosure: the requestor works at Google, and some content of the edit request is related to Google's product. 2A00:79E1:ABC:12B:BDC7:5327:147A:E174 (talk) 22:09, 18 July 2023 (UTC)

Reply 18-JUL-2019

  Unable to review  
Your edit request could not be reviewed because the request is not formatted correctly.

1. The citation style predominantly used by the bfloat16 floating-point format article appears to be Citation Style 1. The citation style used in the edit request consists of bare URL's. Any requested edit of yours which may be implemented will need to resemble the current style already in use in the article – in this case, CS1. (See WP:CITEVAR.)
2. Citation ref tags have not been placed within the requested text indicating which portions of the text the source is referencing. (See WP:INTEGRITY.)
3. The ref note for the Dillon source is only used once, and it is placed in a part of the sentence which confirms no information apart from the fact that bloat16 was once a storage format: "When bloat16 was first introduced as a storage format, ....." The reference entry for Dillon is also missing the |page= parameter.
4. Two of the references provided, one from Google and another from docs.nvidia.com, do not confirm the information provided.
5. Significant portions of text have no ref tags.

In the collapsed section below titled Request edit examples, I have illustrated two: The first shows how the edit request was submitted; the second shows how requests should be submitted in the future.

Request edit examples

INCORRECT The Sun's diameter is 864,337.3 miles, while the Moon's diameter is 2,159 miles. The Sun's temperature is 5,778 Kelvin. https://www.booksource.com https://www.journalsource.com https://www.websource.com In the example above there are three URL's provided with the claim statements, but these URL's have not been placed using Citation Style 1, which is the style predominantly used by the bfloat16 floating-point format article. Additionally, ref tags have not been placed within the text at the exact positions where the information they reference resides. Using the correct style and the correct positioning of the ref tags, the WikiFormatted text should resemble the following: CORRECT The Sun's diameter is 864,337.3 miles,<ref>{{cite book|last1=Sjöblad|first1=Tristan|title=The Sun|url=http://www.booksource.com|publisher=Academic Press|date=2020|page=1}}</ref> while the Moon's diameter is 2,159 miles.<ref>{{cite journal|last1=Harinath|first1=Prisha|title=Size of the Moon|journal=Science|issue=78|volume=51|url=http://www.journalsource.com|date=2020|page=46}}</ref> The Sun's temperature is 5,778 Kelvin.<ref>{{cite book|last1=Uemura|first1=Shu|title=The Sun's Heat|url=http://www.websource.com|publisher=Academic Press|date=2020|page=2}}</ref> Which displays as: The Sun's diameter is 864,337.3 miles, while the Moon's diameter is 2,159 miles. The Sun's temperature is 5,778 Kelvin. References In the example above the references have been formatted according to Citation Style 1, which shows the author, the source's name, date, etc. Also, the ref tags are placed in the exact location where the text which they reference resides. As Wikipedia is a volunteer project, edit requests such yours are generally expected to have this formatting done before the request is submitted for review.

Kindly rewrite your edit request so that it aligns more with the second example shown in the collapsed section above, and feel free to re-submit that edit request at your earliest convenience. If you have any questions about this formatting please don't hesitate to ask myself or another editor. Regards,  Spintendo  22:53, 18 July 2023 (UTC)

Notes