Vim runs shell command turning Chinese character into sign '?' - vim

Question

I've tried to run set .*encoding (for example, encoding, fileencoding, etc.) into utf-8 but the question does not solved.
The question is: when I run command in normal shell, everything going well:
$ make SQL
+-----+-----------+------------------+-------+
| Sno | Sname | Cname | Grade |
+-----+-----------+------------------+-------+
| 01 | Peter | C Language | 98.1 |
| 01 | Peter | Data Struct | 89.3 |
...
| 08 | 王小明 | NULL | NULL |
| 09 | 王明儿 | NULL | NULL |
| 10 | A明B | NULL | NULL |
| 11 | A明B | NULL | NULL |
...
+-----+-----------+------------------+-------+
It works well, but when I run it under vim with vim-command :r!make SQL, it will print with '?' sign:
+-----+-----------+------------------+-------+
| Sno | Sname | Cname | Grade |
+-----+-----------+------------------+-------+
| 01 | Peter | C Language | 98.1 |
| 01 | Peter | Data Struct | 89.3 |
...
| 08 | ??? | NULL | NULL |
| 09 | ??? | NULL | NULL |
| 10 | A?B | NULL | NULL |
| 11 | A?B | NULL | NULL |
...
+-----+-----------+------------------+-------+
I am not sure what happen, but I have a solution that can work:
Run make SQL > tmp in shell.
Run :r!cat tmp in vim.
I also know that :r!make SQL > tmp && cat tmp still has the '?' problem. I do want to use r!make SQL directly in vim, and that's why I ask this question. Thanks for your help.
The :r!locale's result is:
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en
LANGUAGE=
LC_CTYPE="en"
LC_NUMERIC="en"
LC_TIME="en"
LC_COLLATE="en"
LC_MONETARY="en"
LC_MESSAGES="en"
LC_PAPER="en"
LC_NAME="en"
LC_ADDRESS="en"
LC_TELEPHONE="en"
LC_MEASUREMENT="en"
LC_IDENTIFICATION="en"
LC_ALL=

Answer 1

en is not a valid locale on many systems, and even where accepted, it's not going to have the results you want. Typically a locale is of the format xx_XX.CHARSET, where xx is a language code, XX is a country code, and CHARSET is a character set. For example, you could write en_US.UTF-8 for American English with UTF-8, or fr_CH.UTF-8 for Swiss French with UTF-8.
Because most Unix operating systems precede the existence of UTF-8 (macOS being a possible exception), generally the default locales on a system are not UTF-8, so if you specify a locale that does not declare UTF-8 explicitly, your tools will not use UTF-8.
As a result, you'll need to set your locale appropriately, in the format above, so that your system uses UTF-8 everywhere. That will make programs behave correctly.
Specifically to Vim, if you read data in from a piped command like this, Vim will do character set detection (controlled by fileencodings). If you don't have encoding set and your locale is a Unicode locale, Vim will look for a byte-order mark, then Unicode, and then default to latin1. If it's not a Unicode locale and the value is not set, then Vim will just your current locale, which is probably going to be interpreted as latin1, and you'll be sad. This looks like what's happening here.
You can, of course, adjust encoding, fileencoding, and fileencodings, but the easiest way to get sane behavior in Vim and across all programs is to set your locale to one that uses UTF-8.