If I enter
print("®".length)
in smjs, it prints 2. If I enter javascript:alert("®".length) in my firefox as well as opera, it prints 1. Rhino prints 1 too.
Is it possible to tell smjs that I want to treat such characters as single character? Os: linux(Ubuntu 9.04), locale: UTF-8.
The JS shell is (maybe unfortunately :-) not locale-aware. It looks like this problem has been fixed by Bug 648102 -- all input lines in the shell are decoded as UTF-8. Yay progress!
$ ./js
js> print("®".length)
1
And I get the same result in Firefox's fancy new JS scratchpad, which confirms your original javascript:-based result!
Related
I am working on a project which accepts user input via the command line. I am using up-to-date Windows 10 and (after much running around in circles...) I am aware that it is notoriously bad when it comes to handling UTF-8 characters. Consequently, I looked to VS Code and the integrated terminal (PowerShell) to perform input into the program. Sadly, the terminal seemed unable to accept accented UTF-8 characters such as "ë". I then did more research and configured the settings.json for VS Code for UTF-8 BOM encoding. Still, the terminal failed to read accented characters. I am certain that my program is not the issue, nor is my font. I have reduced my code to a test algorithm that simply accepts input using readline-sync (which the developers confirm is compatible with UTF-8: https://github.com/anseki/readline-sync/issues/58) and "console.log"s it.
The test case I have been using is "Hëllo". When I input "Hëllo" into the VS Code terminal, my program outputs "H�llo". When I tried converting all of my apps to UTF-8 encoding using the administrative language settings for Windows 10 and subsequently input "Hëllo" via the command terminal, it output "Hllo". I also tried forcing CMD to use Code Page 65001 with chcp 65001 for UTF-8 encoding, but it still produced "Hllo".
Here is the code I used to configure the VS Code PowerShell terminal via settings.json:
{
"[powershell]": {
"files.encoding": "utf8bom",
"files.autoGuessEncoding": true
}
}
And here is the brief code I wrote to test my input/output and whether the "ë" is being read successfully (which it is not):
const rlSync = require('readline-sync');
const name = rlSync.question('Enter Player 1 Username (Case Sensitive): ');
console.log(name);
If y'all see any issues, please let me know!
I am looking for any way to properly configure my CLI to accept accented characters for use in my program. I do not mean to restrict this question to VS Code or Powershell. If there is a way to accomplish this with the basic Windows 10 CMD, I would love that. Thank you for any help y'all can provide! <3
Is there any particular reason you're using VSCode? I think you're looking for the System.Console InputEncoding/OutputEncoding - unfortunately my default encoding just works with "Hëllo" so couldn't accurately test, and I don't know if this works with VSCode.
Try this (one line at a time)
# store current encoding settings
$i = [System.Console]::InputEncoding
$o = [System.Console]::OutputEncoding
# set encoding to UTF8
[System.Console]::InputEncoding = [System.Text.Encoding]::UTF8
[System.Console]::OutputEncoding = [System.Text.Encoding]::UTF8
# test
"Hëllo"
# revert (if you want. if you don't want, I would at least note the default encoding)
[System.Console]::InputEncoding = $i
[System.Console]::OutputEncoding = $o
Python 3.7, on Windows print does not work as expected for ANSI color codes until shell=True once in subprocess.call().
In the below links it appears to imply that the ANSI color codes should work using the "print" command out of the box.
How to print colour/color in python?
Print in terminal with colors using Python?
the second one mentions VT100 emulation... not sure what exactly that means. I am able to write a batch file that outputs the color fine so I would think (naively) that it should work the same way in Python.
However I am not able to use the ANSI color codes as it seems that the ESC character is being "commented out"(?) because for instance when I
print(u"\u001b[31mHelloWorld")
I am not able to see the colored output, as the ESC character seems to be necessary in Windows and prints in the python shell as "[?]" (a box with a question mark)
Is there something I am missing here?
I found myself an answer. As often happens I just did not look far enough.
the Colorama module can be installed with
py -m pip install colorama
and comes with a method definition at the root of the module called init
colorama.init()
This is a cross platform function in that it is only useful on windows (it saves the active Terminal state for reversal and writes the Terminal to preprocess ANSI codes), it does nothing for other operating systems.
I am thinking about implementing an even more lightweight solution using ctypes and setting the Interpret flags on the active terminal myself.
If you are interested in more information, see here:
https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences
Output Sequences
The following terminal sequences are intercepted by the console host when written into the output stream, if the ENABLE_VIRTUAL_TERMINAL_PROCESSING flag is set on the screen buffer handle using the SetConsoleMode flag. Note that the DISABLE_NEWLINE_AUTO_RETURN flag may also be useful in emulating the cursor positioning and scrolling behavior of other terminal emulators in relation to characters written to the final column in any row.
Emphasis mine.
On Windows, there's some basic emoji support in the console, so that I can get a monochrome glyph if I type, e.g. ☕ or 📜. I can output a string from PowerShell or a C# console application or Python and they all show those characters fine enough.
However, from Node.js, I can only get a couple of emoji to display (e.g. ☕), but not other (instead of 📜 I see �). However, if I throw a string with those characters, they display correctly.
console.log(' 📜 ☕ ');
throw ' 📜 ☕ ';
If I run the above script, the output is
� ☕
C:\Code\emojitest\emojitest.js:2
throw ' 📜 ☕ ';
^
📜 ☕
Is there anyway that I can output those emojis correctly without throwing an error? Or is that exception happening outside of what's available to me through the standard Node.js APIs?
What you want may not be possible without a change to libuv. When you (or the
console) write to stdout or stderr on Windows and the stream is a TTY,
libuv does its own conversion from UTF‑8 to UTF‑16. In doing so it explicitly
refuses to output surrogate pairs, emitting instead the replacement character
U+FFFD � for any codepoint beyond the BMP.
Here’s the culprit in uv/src/win/tty.c:
/* We wouldn't mind emitting utf-16 surrogate pairs. Too bad, the */
/* windows console doesn't really support UTF-16, so just emit the */
/* replacement character. */
if (utf8_codepoint > 0xffff) {
utf8_codepoint = UNICODE_REPLACEMENT_CHARACTER;
}
The throw message appears correctly because Node lets Windows do the
conversion from UTF‑8 to UTF‑16 with MultiByteToWideChar() (which does emit
surrogate pairs) before writing the message to the console. (See
PrintErrorString() in src/node.cc.)
Note: A pull request has been submitted to resolve this issue.
(Disclaimer: I don't have a solution, I explored what makes exception handling special with regards to printing emoji, with the tools I have on Windows 10 -- With some luck that might sched some light on the issue, and perhaps someone will recognize something and come up with a solution)
Looks like Node's exception reporting code for Windows calls to a different Windows API, that happens to support Unicode better.
Let's see with Node 7.10 sources:
ReportException
→
AppendExceptionLine
→
PrintErrorString
In PrintErrorString, the Windows-specific section detects output type (tty/console or not):
- For non-tty/console context it will print to stderr (e.g. if you redirect to a file)
- In a cmd console (with no redirection), it will convert text with MultiByteToWideChar() and then pass that to WriteConsoleW().
If I run your program using ConEmu (easier than getting standard cmd to work with unicode & emoji -- yes I got a bit lazy here), I see something similar as what you saw: console.log fails to print emoji, but the emoji in exception message are printed OK (even the scroll glyph).
If I redirect all output to a file (node test.js > out.txt 2>&1, yes that works in Windows cmd too), I get "clean" Unicode in both cases.
So it seems when a program prints to stdout or stderr in a Windows console, the console does some (bad) reencoding work before printing. When the program uses Windows console API directly (doing the conversion itself with MultiByteToWideChar then write to console with WriteConsoleW()), the console shows the glorious unaltered emoji.
When a JS program uses console API to log stuff, maybe Node could try (on Windows) to detect console and do the same thing as it does for reporting exceptions. See #BrianNixon's answer that explains what is actually happening in libuv.
The next "Windows Terminal" (from Kayla Cinnamon) and the Microsoft/Terminal project should be able to display emojis.
This will be available starting June 2019.
Through the use of the Consolas font, partial Unicode support will be provided.
The request is in progress in Microsoft/Terminal issue 387.
And Microsoft/Terminal issue 190 formally demands "Add emoji support to Windows Console".
But there are still issues (March 2019):
I updated my Win10 from 1803 to 1809 several days ago, and now all characters >= U+10000 (UTF-8 with 4 bytes or more) no longer display.
I have also tried the newest insider version(Windows 10 Insider Preview 18358.1 (19h1_release)), unfortunately, this bug still exists.
As you can see, D fails to output german Umlaute. At least on Windows. On Linux or BSD the same program outputs the string as I've saved it.
I already tried wstring or dstring, but the output is the same.
What am I doing wrong?
D will output UTF-8 regardless of the operating system. How the output will be interpreted depends on how it is displayed. In this particular case, it looks like your IDE is interpreting the output as if it was encoded in the Windows-1252 encoding.
For the standard Windows console, you could change the output encoding by calling SetConsoleOutputCP(65001), but note that this may have some undesired side effects (you should restore the codepage before your progam exits, and batch files may not run while the console output codepage is set to 65001).
CyberShadows post guided me to an acceptable answer. :-)
In Eclipse it is possible to change the output-encoding without changing global settings of the OS.
Go to Run --> Run-Configurations...
There select the Common-Tab and change the encoding to UTF-8. Now german Umlaute are displayed correctly. At least in Eclipse. :-)
Another possibility is to use https://babun.github.io/ . It is a Cygwin-based Shell that ouputs UTF-8:
I use cassandra in my project, and it works well, but now I want to use cassandra-cli.bat to initialize some data(include some UTF8 words), when I run cassandra-cli.bat and input some words, the UTF8 words displayed as ????, I don't know why, can you help me?
This sounds like a terminal issue, rather than specifically related to cassandra-cli. If your terminal application supports UTF8, it should just work. I'm not familiar with windows at all, but searching for 'UTF8 Windows terminal' gives a few hints on things you might try.