Jump to content

Unicode in Microsoft Windows

From Wikipedia, the free encyclopedia
This is an old revision of this page, as edited by Spitzak (talk | contribs) at 12:53, 25 February 2023. The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.

Microsoft was one of the first companies to implement Unicode in their products. Windows NT was the first operating system that used "wide characters" in system calls. Using the (now obsolete) UCS-2 encoding scheme at first, it was upgraded to the variable-width encoding UTF-16 starting with Windows 2000, allowing a representation of additional planes with surrogate pairs. However Microsoft did not support UTF-8 in its API until May 2019, though it now appears to be encouraging its use.[1]

A large amount of Microsoft documentation uses the word "Unicode" to refer explicitly to the UTF-16 encoding. Anything else, including UTF-8, is not "Unicode".

In various Windows families

Windows NT based systems

Current Windows versions and all back to Windows XP and prior Windows NT (3.x, 4.0) are shipped with system libraries that support string encoding of two types: 16-bit "Unicode" (UTF-16 since Windows 2000) and a (sometimes multibyte) encoding called the "code page" (or incorrectly referred to as ANSI code page). 16-bit functions have names suffixed with 'W' (from "wide") such as SetWindowTextW. Code page oriented functions use the suffix 'A' for "ANSI" such as SetWindowTextA (some other conventions were used for APIs that were copied from other systems, such as _wfopen/fopen or wcslen/strlen). This split was necessary because many languages, including C, did not provide a clean way to pass both 8-bit and 16-bit strings to the same function.

Most 'A' functions are implemented as wrappers that translate the text using the current code page to UTF-16 and then call the corresponding 'W' functions.[citation needed] 'A' functions that return strings do the opposite conversion, turning characters that don't exist in the current locale into '?'.

Microsoft attempted to support Unicode "portably" by providing a "UNICODE" switch to the compiler, that switches unsuffixed "generic" calls from the 'A' to the 'W' interface and converts all string constants to "wide" UTF-16 versions.[2][3] This does not actually work because it does not translate UTF-8 outside of string constants, resulting in code that attempts to open files just not compiling.[citation needed]

Earlier, and independent of the "UNICODE" switch, Windows also provided the Multibyte Character Sets (MBCS) API switch.[4] This changes some functions that don't work in MBCS such as strrev to an MBCS-aware one such as _mbsrev.[5][6]

Windows CE

In (the now discontinued) Windows CE, UTF-16 was used almost exclusively, with the 'A' API mostly missing.[7] A limited set of ANSI API is available in Windows CE 5.0, for use on a reduced set of locales that may be selectively built onto the runtime image.[8]

Windows 9x

Main article: Microsoft Layer for Unicode

In 2001, Microsoft released a special supplement to Microsoft's old Windows 9x systems. It includes a dynamic link library, 'unicows.dll', (only 240 KB) containing the 16-bit flavor (the ones with the letter W on the end) of all the basic functions of Windows API. It is merely a translation layer: SetWindowTextW will simply convert its input using the current codepage and call SetWindowTextA.

UTF-8

Microsoft Windows (Windows XP and later) has a code page designated for UTF-8, code page 65001[9] or CP_UTF8. For a long time, it was impossible to set the locale code page to 65001, leaving this code page only available for (a) explicit conversion functions such as MultiByteToWideChar and/or (b) the Win32 console command chcp 65001 to translate stdin/out between UTF-8 and UTF-16. This meant that "narrow" functions, in particular fopen (which opens files), couldn't be called with UTF-8 strings, and in fact there was no way to open all possible files using fopen no matter what the locale was set to and/or what bytes were put in the string, as none of the available locales could produce all possible UTF-16 characters. This problem also applied to all other APIs that take or return 8-bit strings, including Windows ones such as SetWindowText.

On all modern non-Windows platforms, the file-name string passed to fopen is effectively UTF-8. This produced an incompatibility between other platforms and Windows. The usual work-around was to add code to convert UTF-8 to UTF-16 using MultiByteToWideChar and call the "wide" function instead of fopen.[10] Another popular work-around was to convert the name to the 8.3 filename equivalent, this is necessary if the fopen is inside a library function that takes a string filename and thus calling another function is not possible. There were also proposals to add new APIs to portable libraries such as Boost to do the necessary conversion, by adding new functions for opening and renaming files. These functions would pass filenames through unchanged on Unix, but translate them to UTF-16 on Windows. Such a library, Boost.Nowide,[11] was accepted into Boost[12] and will be part of the 1.73 release.[needs update] This would allow code to be "portable", but required just as many code changes as calling the wide functions.

In April 2018 (or possibly November 2017[13]), with insider build 17035 (nominal build 17134) for Windows 10, a "Beta: Use Unicode UTF-8 for worldwide language support" checkbox appeared for setting the locale code page to UTF-8.[a] This allows for calling "narrow" functions, including fopen and SetWindowTextA, with UTF-8 strings. However this is a system-wide setting and a program cannot assume it is set.

In May 2019, Microsoft added the ability for a program to set the code page to UTF-8 itself,[14] allowing programs written to use UTF-8 to be run by non-expert users.

In Windows 11 some system files are required to use UTF-8 and do not require a Byte Order Mark.[15] Notepad can now recognize UTF-8 without the Byte Order Mark, and can be told to write UTF-8 without a Byte Order Mark.[citation needed] Some other Microsoft products are using UTF-8 internally, including Visual Studio[citation needed] and their SQL Server 2019, with Microsoft claiming 35% speed increase from use of UTF-8, and "nearly 50% reduction in storage requirements."[16]

Programming platforms

Microsoft's compilers often fail at producing UTF-8 string constants from UTF-8 source files. The most reliable method is to turn off UNICODE, not mark the input file as being UTF-8 (i.e. do not use a BOM), and arrange the string constants to have the UTF-8 bytes. If a BOM was added, a Microsoft compiler will interpret the strings as UTF-8, convert them to UTF-16, then convert them back into the current locale, thus destroying the UTF-8.[17] Without a BOM and using a single-byte locale, Microsoft compilers will leave the bytes in a quoted string unchanged. On modern systems setting the code page to UTF-8 helps considerably, but invalid byte sequences are still not preserved (using \x can work around this).

See also

Notes

  1. ^ Found under control panel, "Region" entry, "Administrative" tab, "Change system locale" button.

References

  1. ^ "Use UTF-8 code pages in Windows apps".
  2. ^ "Unicode in the Windows API". Retrieved 7 May 2018.
  3. ^ "Conventions for Function Prototypes (Windows)". MSDN. Retrieved 7 May 2018.
  4. ^ "Support for Multibyte Character Sets (MBCSs)". Retrieved 2020-06-15.
  5. ^ "Double-byte Character Sets". MSDN. 2018-05-31. Retrieved 2020-06-15. our applications use DBCS Windows code pages with the "A" versions of Windows functions.
  6. ^ _strrev, _wcsrev, _mbsrev, _mbsrev_l Microsoft Docs
  7. ^ "Differences Between the Windows CE and Windows NT Implementations of TAPI". MSDN. Retrieved 7 May 2018. Windows CE is Unicode-based. You might have to recompile source code that was written for a Windows NT-based application.
  8. ^ "Code Pages (Windows CE 5.0)". Microsoft Docs. Retrieved 7 May 2018.
  9. ^ "Code Page Identifiers (Windows)". msdn.microsoft.com.
  10. ^ "UTF-8 in Windows". Stack Overflow. Retrieved July 1, 2011.
  11. ^ "Boost.Nowide". GitHub.
  12. ^ "Boost mailing list".
  13. ^ "Windows10 Insider Preview Build 17035 Supports UTF-8 as ANSI". Hacker News. Retrieved 7 May 2018.
  14. ^ "Use the Windows UTF-8 code page - UWP applications". docs.microsoft.com. Retrieved 2020-06-06. As of Windows Version 1903 (May 2019 Update), you can use the ActiveCodePage property in the appxmanifest for packaged apps, or the fusion manifest for unpackaged apps, to force a process to use UTF-8 as the process code page. [..] CP_ACP equates to CP_UTF8 only if running on Windows Version 1903 (May 2019 Update) or above and the ActiveCodePage property described above is set to UTF-8. Otherwise, it honors the legacy system code page. We recommend using CP_UTF8 explicitly.
  15. ^ themar-msft. "Customize the Windows 11 Start menu". docs.microsoft.com. Retrieved 2021-06-29. Make sure your LayoutModification.json uses UTF-8 encoding.
  16. ^ "Introducing UTF-8 support for SQL Server". techcommunity.microsoft.com. 2019-07-02. Retrieved 2021-08-24. For example, changing an existing column data type from NCHAR(10) to CHAR(10) using an UTF-8 enabled collation, translates into nearly 50% reduction in storage requirements. [..] In the ASCII range, when doing intensive read/write I/O on UTF-8, we measured an average 35% performance improvement over UTF-16 using clustered tables with a non-clustered index on the string column, and an average 11% performance improvement over UTF-16 using a heap.
  17. ^ UTF-8 Everywhere FAQ: How do I write UTF-8 string literal in my C++ code?
Retrieved from "https://en.wikipedia.org/w/index.php?title=Unicode_in_Microsoft_Windows&oldid=1141515922"