I am using following command to search and print non-ascii characters:
grep --color -R -C 2 -P -n "[\x80-\xFF]" .
The output that I get, prints the line which has non-ascii characters in it.
However it does not print the actual unicode character.
Is there a way to print the unicode character?
output
./test.yml-35-
./test.yml-36-- name: Flush Handlers
./test.yml:37: meta: flush_handlers
./test.yml-38-
--
This was answered in Searching for non-ascii characters. The real issue as shown in Filtering invalid utf8 is that the regular expression you are using is for single bytes, while UTF-8 is a multibyte encoding (and the pattern must therefore cover multiple bytes).
The extensive answer by #Peter O in the latter Q/A appears to be the best one, using Perl. grep is the wrong tool.
Related
I would like to see the actual file contents without it being formatted to print. For example, to show:
\n0.032,170\n0.034,290
Instead of:
0.032,170
0.34,290
Is there a command to echo the file's actual data in bash? I've tried using head, cat, more, etc. but all those seem to echo the "print-formatted" text. For example:
$ cat example.csv
0.032,170
0.34,290
How can I print the actual characters within the file?
This reads as if you miss understand what the "actual characters in the file" are. You will not find the characters \ and n in that file. But only a line feed, which is a specific character. So the utilities like cat do actually output exactly the characters in the file.
Putting it the other way around: if you really had those two characters literally in the file, then a utility like cat would actually output them. I just checked that, just to be sure.
You can easily check that yourself if you open the file using a hexeditor. There you will see the character 0A (decimal 10) which is a line feed character. You will not see the pair of the two characters \ and n somewhere in that file.
Many programming languages and also shell environments use escape sequences like \n in string definitions and identify those as control characters which would not be typable otherwise. So maybe that is where your impression comes from that your files should contain those two characters.
To display newlines as \n, you might try:
awk 1 ORS='\\n' input-file
This is not the "actual characters in the file", as \n is merely a conventional method of displaying a newline, but this does seem to be what you want.
This question already has an answer here:
Prevent "echo" from interpreting backslash escapes
(1 answer)
Closed 4 years ago.
I have a bash script that contains lines:
remote_installer_svc_args="$local_cifs_mount/eset-remote-installer.args"
svc_arg_x86="%SYSTEMROOT%\\$(basename $remote_temp_dir)\\$(basename $INSTALLER_BAT)"
svc_arg_x64="%SYSTEMROOT%\\$(basename $remote_temp_dir)\\$(basename $INSTALLER_BAT)"
echo "$svc_arg_x86" > $remote_installer_svc_args
echo "$svc_arg_x64" >> $remote_installer_svc_args
It should produce a file that looks like this (in notepad++ on windows):
instead the file looks like this:
or in vim:
What is wrong with the script? Because when I copy those lines into bash it works, only if I run the script it does produce those strange characters...
You've run into part of the mess of inconsistent behavior that plagues the echo command. Specifically, some versions of echo (in some modes) interpret escape (backslash) sequences in the string they're asked to print. Others don't. When you ask echo to print %SYSTEMROOT%\era_rd_6HbUKJTR\EraAgentInstaller.bat, it might see the \e part and think it's supposed to convert that to the ASCII escape character.
Note that there are two different characters being called "escape" here: The backslash is used by the shell as an escape character, meaning that it and the characters immediately following it have some special meaning. The ASCII escape, on the other hand, is treated as a special character by the terminal (and vim and some other things) in a somewhat similar manner. Since the ASCII escape is a nonprinting character, when notepad++ and vim have to display it, they show some sort of alternate representation ("ESC" or "^]").
Anyway, since echo is inconsistent about its treatment of the backslash character, it's best to avoid it for strings that might contain backslash. Use printf instead (see "Why is printf better than echo?" on unix.se). It's a little more complicated to use, but not too bad. The main things to realize are that the first argument to printf is a "format" string that's used to control how the rest of the arguments are printed, and that unlike echo it doesn't automatically add a newline to the end.
What you want to use is:
printf '%s\n' "$svc_arg_x86" > $remote_installer_svc_args
printf '%s\n' "$svc_arg_x64" >> $remote_installer_svc_args
Or you can simplify it to:
printf '%s\n' "$svc_arg_x86" "$svc_arg_x64" > $remote_installer_svc_args
That first argument, %s\n, says to print a plain string followed by a newline. Backslash escapes in the format string are always interpreted, but strings formatted with the %s format never have escapes interpreted. Note that in the single-command version, the format string gets applied to each of the other two arguments, so each gets a newline at the end, so each winds up on a separate line in the output file.
I am trying to format some text that was converted from UTF-16 to ASCII, the output looks like this:
C^#H^#M^#M^#2^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#
T^#h^#e^#m^#e^# ^#M^#a^#n^#a^#g^#e^#r^# ^#f^#o^#r^# ^#3^#D^#S^#^#^#^#^#^#^#^#^#^#^#^#^#^#
The only text I want out of that is:
CHMM2
Theme Manager for 3DS
So there is a line break "\n" at the end of each line and when I use
tr -cs 'a-zA-Z0-9' 'newtext' infile.txt > outfile.txt
It is stripping the new line as well so all the text ends up in one big string on one line.
Can anyone assist with figuring out how to strip out only the ^#'s and keeping spaces and new lines?
The ^#s are most certainly null characters, \0s, so:
tr -d '\0'
Will get rid of them.
But this is not really the correct solution. You should simply use theiconv command to convert from UTF-16 to UTF-8 (see its man page for more information). That is, of course, what you're really trying to accomplish here, and this will be the correct way to do it.
This is an XY problem. Your problem is not deleting the null characters. Your real problem is how to convert from UTF-16 to either UTF-8, or maybe US-ASCII (and I chose UTF-8, as the conservative answer).
I have read a BASH script, and found the following line:
lines="$lines"$'\n'
After testing, I know the meaning of this line is adding a "\n" after the string "$lines".
But after checking the bash manual, I can't find "$" can be used as a concatenated symbol. Could anyone give explainations on this usage of "$"? Thanks very much in advance!
A slightly closer read of the Bash Manual under Quoting would reveal where this gem is hidden. Specifically:
Words of the form $'string' are treated specially. The word expands to
string, with backslash-escaped characters replaced as specified by the
ANSI C standard.
Used specifically in the context of \n it provdes a new line. You most often see this form of quoting used in regard to the Bash IFS (internal field separator) whose default is space tab newline written:
IFS=$' \t\n'
I have a text file containing Arabic characters and some other characters (punctuation marks, numbers, English characters, ... ).
How can I tell sed to remove all the characters in the file, except Arabic ones? In short I can say that we typically tell sed to remove/replace some specific characters and print others, but now I am looking for a way to tell sed just print my desired characters, and remove all other characters.
With GNU sed, you should be able to specify characters by their hex code. You can use those in a a character class:
sed 's/[\x00-\x7F]//g' # hex notation
sed 's/[\o000-\o177]//g' # octal notation
You should also be able to achieve the same effect with the tr command:
tr -d '[\000-\177]'
Both methods assume UTF8 encoding of your input file. Multi-byte characters have their highest bit set, so you can simply strip everything that's a standard ASCII (7 bits) character.
To keep everything except some well defined characters, use a negative character classe:
sed 's/[^characters you want to keep]//g'
Using a pattern alike to [^…]\+ might improve performance of the regex.