When I use a text editor to see the actual content, I see
baliÄ<8d>ky 0 b a l i ch k i
and when I use cat to see it, I see
baličky 0 b a l i ch k i
How can I make it so, it is actually baličky in the text editor as well?
I've tried numerous commands such as iconv -f UTF-8 -t ISO-8859-15, iconv -f ISO-8859-15 -t UTF-8, recode utf8..l9.
None of them works. It's still baliÄ<8d>ky instead of baličky. This is a Czech word. If I do a simple sed command (/Ä<8d>/č), it works but I have so many other characters like this and manual work is basically really mundane at this point.
Any suggestions?
Related
By most accounts, one ought to be able to change the encoding of a UTF-8
file to a Latin-1 (ISO-8859-1) encoding by a trivial invocation of iconv such as:
iconv -c -f utf-8 -t ISO-8859-1//TRANSLIT
However, this fails to deal with accented characters properly. Consider
for example:
$ echo $LC_ALL
C
$ cat Gonzalez.txt
González, M.
$ file Gonzalez.txt
Gonzalez.txt: UTF-8 Unicode text
$ iconv -c -f utf-8 -t ISO-8859-1//TRANSLIT < Gonzalez.txt > out
$ file out
out: ASCII text
$ cat out
Gonzalez, M.
I've tried various variations of the above, but none properly handles
the accented "a", the point being that Latin-1 does have an accented "a".
Indeed, uconv does handle the situation properly:
$ uconv -x Any-Accents -f utf-8 -t l1 < Gonzalez.txt > out
$ file out
out: ISO-8859 text
Opening the file in emacs or
Sublime shows the accented "a" properly. Same thing using -x nfc.
Unfortunately, my target environment does not permit a solution using "uconv",
so I am looking for a simple solution using either iconv or Python3.
python3 attempts
My attempts using python3 so far have not been successful.
For example, the following:
import sys
import fileinput # allows file to be specified or else reads from STDIN
for line in fileinput.input():
l=line.encode("latin-1","replace")
sys.stdout.buffer.write(l)
produces:
Gonza?lez, M.
(That's a literal "?".)
I've tried various other Python3 possibilities, so far without success.
Please note that I've reviewed numerous SO questions on this topic, but the answers using iconv or Python3 do not handle Gonzalez.txt properly.
There are two ways to encode A WITH ACUTE ACCENT in Unicode.
One is to use a combined character, as illustrated here with Python's built-in ascii function:
>>> ascii('á')
"'\\xe1'"
But you can also use a combining accent following an unaccented letter a:
>>> ascii('á')
"'a\\u0301'"
Depending on the displaying applications, the two variants may look indistinguishable (in my terminal, the latter looks a bit odd with the accent being too large).
Now, Latin-1 has an accented letter a, but no combining accents, so that's why the acute becomes a question mark when encoding with errors="replace".
Fortunately, you can automatically switch between the two variants.
Without going into details (there are many details here), Unicode defined two normalization forms, called composed and decomposed, abbreviated NFC and NFD, respectively.
In Python, you can use the standard-library module unicodedata:
>>> import unicodedata as ud
>>> ascii(ud.normalize('NFD', 'á'))
"'a\\u0301'"
>>> ascii(ud.normalize('NFC', 'á'))
"'\\xe1'"
In your specific case, you can convert the input strings to NFC form, which will increase coverage of Latin-1 characters:
>>> n = 'Gonza\u0301lez, M.'
>>> print(n)
González, M.
>>> n.encode('latin1', errors='replace')
b'Gonza?lez, M.'
>>> ud.normalize('NFC', n).encode('latin1', errors='replace')
b'Gonz\xe1lez, M.'
My file names look the same but they are not.
I copied many_img/ from Debian1 to OS X, then from OS X to Debian2 (for maintenance purpose) with using rsync -a -e ssh on each step to preserve everything.
If i do ls many_img/img1/* i get visually the same output on Debian1 and Debian2 :
prévisionnel.jpg
But somehow, ls many_img/img1/* | od -c gives different results:
On Debian1:
0000000 p r 303 251 v i s i o n n e l . j p
0000020 g \n
On Debian2:
0000000 p r e 314 201 v i s i o n n e l . j
0000020 p g \n
Thus my web app on Debian2 cannot match the picture in the file system with filename in database.
i thought maybe i need to change file encoding, but it looks like it's already utf-8 on every OS:
convmv --notest -f iso-8859-15 -t utf8 many_img/img1/*
Returns:
Skipping, already UTF-8
Is there a command to get back all my 40 thousands file names like on my Debian 1 from my Debian 2 (without transfering all again) ?
I am confused if it is a file name encoding problem or anything else ?
I finaly found command line conversion tools i was looking for (thanks #Mark for setting me on the right track !)
Ok, i didn't know OS X was encoding file names under the hood with a different UTF-8 Normalization.
It appears OS X is using Unicode Normalization Form D (NFD)
while Linux OS are using Unicode Normalization Form C (NFC)
HSF+ file system encode every single file name character in UTF-16.
Unicode characters are Decomposed on OS X versus Precomposed on Linux OS.
é for instance (Latin small letter e with acute accent), is technically a (U+00E9) character on Linux
and is decomposed into a base letter "e" (U+0065) and an acute accent (U+0301) in its decomposed form (NFD) on OS X.
Now about conversion tools:
This command executed from Linux OS will convert file name from NFD
to NFC:
convmv --notest --nfc -f utf8 -t utf8 /path/to/my/file
This command executed from OS X will rsync over ssh with NFD to NDC
on the fly conversion:
rsync -a --iconv=utf-8-mac,utf-8 -e ssh path/to/my/local/directory/* user#destinationip:/remote/path/
I tested the two methods and it works like a charm.
Note:
--iconv option is only available with rsync V3 whereas OS X provides an old 2.6.9 version by default so you'll need to update it first.
Typically to check and upgrade :
rsync --version
brew install rsync
echo 'export PATH=/usr/local/bin:$PATH' >> ~/.profile
The first filename contains the single character é while the second contains a simple e followed by the combining character ́ (COMBINING ACUTE ACCENT). They're both valid Unicode, they're just normalized differently. It appears the OS normalized the filename as it created the file.
I'm generating binary data files that are simply a series of records concatenated together. Each record consists of a (binary) header followed by binary data. Within the binary header is an ascii string 80 characters long. Somewhere along the way, my process of writing the files got a little messed up and I'm trying to debug this problem by inspecting how long each record actually is.
This seems extremely related, but I don't understand perl, so I haven't been able to get the accepted answer there to work. The other answer points to bgrep which I've compiled, but it wants me to feed it a hex string and I'd rather just have a tool where I can give it the ascii string and it will find it in the binary data, print the string and the byte offset where it was found.
In other words, I'm looking for some tool which acts like this:
tool foobar filename
or
tool foobar < filename
and its output is something like this:
foobar:10
foobar:410
foobar:810
foobar:1210
...
e.g. the string which matched and a byte offset in the file where the match started. In this example case, I can infer that each record is 400 bytes long.
Other constraints:
ability to search by regex is cool, but I don't need it for this problem
My binary files are big (3.5Gb), so I'd like to avoid reading the whole file into memory if possible.
grep --byte-offset --only-matching --text foobar filename
The --byte-offset option prints the offset of each matching line.
The --only-matching option makes it print offset for each matching instance instead of each matching line.
The --text option makes grep treat the binary file as a text file.
You can shorten it to:
grep -oba foobar filename
It works in the GNU version of grep, which comes with linux by default. It won't work in BSD grep (which comes with Mac by default).
You could use strings for this:
strings -a -t x filename | grep foobar
Tested with GNU binutils.
For example, where in /bin/ls does --help occur:
strings -a -t x /bin/ls | grep -- --help
Output:
14938 Try `%s --help' for more information.
162f0 --help display this help and exit
I wanted to do the same task. Though strings | grep worked, I found gsar was the very tool I needed.
http://tjaberg.com/
The output looks like:
>gsar.exe -bic -sfoobar filename.bin
filename.bin: 0x34b5: AAA foobar BBB
filename.bin: 0x56a0: foobar DDD
filename.bin: 2 matches found
I have a bunch of Arabic, English, Russian files which are encoded in utf-8. Trying to process these files using a Perl script, I get this error:
Malformed UTF-8 character (fatal)
Manually checking the content of these files, I found some strange characters in them.
Now I'm looking for a way to automatically remove these characters from the files.
Is there anyway to do it?
This command:
iconv -f utf-8 -t utf-8 -c file.txt
will clean up your UTF-8 file, skipping all the invalid characters.
-f is the source format
-t the target format
-c skips any invalid sequence
Your method must read byte by byte and fully understand and appreciate the byte wise construction of characters. The simplest method is to use an editor which will read anything but only output UTF-8 characters. Textpad is one choice.
iconv
can do it
iconv -f cp1252 foo.txt
None of the methods here or on any other similar questions worked for me.
In the end what worked was simply opening the file in Sublime Text 2. Go to File > Reopen with Encoding > UTF-8. Copy the entire content of the file into a new file and save it.
May not be the expected solution but putting this out here in case it helps anyone, since I've been struggling for hours with this.
i work with txt files, and i recently found e.g. these characters in a few of them:
http://pastebin.com/raw.php?i=Bdj6J3f4
what could these characters be? wrong character-encoding? i just want to use normal UTF-8 TXT files, but when i use:
iconv -t UTF-8 input.txt > output.txt
it's still the same.
When i open the files in gedit, copy+paste them in another txt files, then there's no characters like in the ones in pastebin. so gedit can solve this problem, it encodes the TXT files well. but there are too many txt files.
why are there http://pastebin.com/raw.php?i=Bdj6J3f4 -like chars in the text files? can they be converted to "normal chars"? I can't see e.g.: the "Ì" char, when i open the files with vim, only after i "work with them" (e.g.: awk, etc)
It would help if you posted the actual binary content of your file (perhaps by using the output of od -t x1). The pastebin returns this as HTML:
"Ì"
"Ã"
"é"
The first line corresponds to U+00C3 U+0152. THe last line corresponds to U+00C3 U+00A9, which is the string "\ux00e9" in UTF ("\xc3\xa9") with the UTF-8 bytes reinterpreted as Latin-1.
From man iconv:
The iconv program converts text from
one encoding to another encoding. More
precisely, it converts from the
encoding given for the -f option to
the encoding given for the -t option.
Either of these encodings defaults to
the encoding of the current locale
Because you didn't specify the -f option it assumes the file is encoded with your current locale's encoding (probably UTF-8), which apparently is not true. Your text editors (gedit, vim) do some encoding detection - you can check which encoding do they detect (I don't know how - I don't use any of them) and use that as -f iconv option (or save the open file with your desired encoding using one of those text editors).
You can also use some tool for encoding detection like Python chardet module:
$ python -c "import chardet as c; print c.detect(open('file.txt').read(4096))"
{'confidence': 0.7331842298102511, 'encoding': 'ISO-8859-2'}
..solved !
how:
i just right clicked on the folders containing the TXT files, and pasted them to another folder.. :O and presto..theres no more ugly chars..