I have a function analyze_text: string -> unit to analyze a text. As a result, (most of the time,) ./analyze aText launches the function with the argument.
let usage_msg = "./analyze [options] TEXT" in
Arg.parse options analyze_text usage_msg;
However, I realize that when the text contains special characters like ", ' or !, it cannot be well read. Does anyone know if there is a way to well wrap the text and give it to the function?
On the shell there are many shell characters. You can escape the shell characters by enclosing your input in single quotes.
$ echo 'a*$b"$c"!d'
a*$b"$c"!d
If your input itself contains single quote. You'll have to enclose that in the double quotes and concatenate with the rest of substrings of input which are enclosed in single quotes.
e.g. You want to print: He$l!o Wo$r'ld
You can do it like:
$ echo 'He$l!o Wo$r'"'"'ld'
He$l!o Wo$r'ld
In your case, the culprit is not your OCaml code, but the behavior of your shell, e.g., bash. When you enter text in the bash command line prompt many characters have special meaning, e.g., ", ', $, \ and so on. To hush the special meaning of a character in bash you can either escape it with the backslash, e.g., \$, \\, \' or delimit with single quotes (but you will still need to escape single quotes in the single-quotes-delimited text.
The general approach is that when your input is actual text or data, not a sequence of commands and options, you should read the input from a file or from the standard input channel. This also helps, when the size of the input is large, as most of the shells limit (sometimes significantly) the total number of characters that can be passed through the command line. In vanilla OCaml, you can input the whole file into a single string using the following simple code
let read_file filename =
let buf = Buffer.create 4096 in
let chan = open_in filename in
begin
try while true do Buffer.add_channel buf chan 4096 done
with End_of_file -> ()
end;
Buffer.contents buf
Then you don't need to deal with any special characters, as your input will be the file and no shell in between will do any interpretations. You can even analyze binary data with that.
Related
I have a string that is like so:
"string content here
"
because it is too long to fit on the screen in one line
The string is the name of a file i would like to read, but i always get an error message that the file name wasn't found because it includes the new line character in the string when this obviously isn't in the file name. I cannot rename the file and I have tried the strip function to remove it, but this doesn't work. How can I remove the enter character from my string so I can load my file?
You can use the function strip to remove any trailing whitespace from a string.
>> text = "hello" + newline; %Create test string.
>> disp(text)
hello
>> text_stripped = strip(text);
>> disp(text_stripped)
hello
>>
In the above ">>" has been included to better present the removal of the whitespace in the string.
Consider replacing the newline character with nothing using strrep. Link
As an example:
s = sprintf('abc\ndef') % Create a string s with a newline character in the middle
s = strrep(s, newline, '') % Replace newline with nothing
Alternatively, you could use regular expressions if there are several characters causing you issues.
Alternatively, you could use strip if you know the newline always occurs at the beginning or end.
I'm new to perl, and I'm trying to print out the folderName from mork files (from Thunderbird).
From: https://github.com/KevinGoodsell/mork-converter/blob/master/doc/mork-format.txt
The second type of special character sequence is a dollar sign
followed by two hexadecimal digits which give the value of the
replacement byte. This is often used for bytes that are non-printable
as ASCII characters, especially in UTF-16 text. For example, a string
with the Unicode snowman character (U+2603):
☃snowman☃
may be represented as UTF-16 text in an Alias this way:
<(83=$03$26s$00n$00o$00w$00m$00a$00n$00$03$26)>
From all the Thunderbird files I've seen it's actually encoded in UTF-8 (2 to 4 bytes).
The following characters need to be escaped (with \) within the string to be used literally: $, ) and \
Example: aaa\$AA$C3$B1b$E2$98$BA$C3$AD\\x08 should print aaa$AAñb☺í\x08
$C3$B1 is ñ; $E2$98$BA is ☺; $C3$ADis í
I tried using the regex to replaced unescaped $ into \x
my $unescaped = qr/(?<!\\)(?:(\\\\)*)/;
$folder =~ s/$unescaped\$/\\x/g;
$folder =~ s/\\([\\$)])/$1/g; # unescape "\ $ ("
Within perl it just prints the literal string.
My workaround is feeding it into bash's printf and it succeeds... unless there's a literal "\x" in the string
$ folder=$(printf "$(mork.pl 8777646a.msf)")
$ echo "$folder"
aaa$AAñb☺í
Questions i consulted:
Convert UTF-8 character sequence to real UTF-8 bytes
But it seems it interprets every byte by itself, not in groups.
In Perl, how can I convert an array of bytes to a Unicode string?
I don't know how to apply this solution to my use case.
Is there any way to achieve this in perl?
The following substitution seems to work for your input:
s/\\([\$\\])|\$(..)/$2 ? chr hex $2 : $1/ge;
Capture \$ or \\, if matched, replace them with $ or \. Otherwise, capture $.. and convert to the corresponding byte.
If you want to work with the result in Perl, don't forget to decode it from UTF-8.
$chars = decode('UTF-8', $bytes);
I have a large (4 GB) Windows .csv text file (each lines end in "\r\n") in a Linux environment that was supposed to have been a csv delimited file (delimiter = '|', text qualifier = '"') with each field separated by a pipe and enclosed in double quotes. Any narrative text field with embedded double quotes was supposed to have the double quote escaped with a second double quote (ie. " the quick "brown" fox" was supposed to have been represented as "the quick ""brown"" fox"). Unfortunately escaping the embedded double quotes did not occur. Further the text fields may include embedded new lines (i.e. Windows CR (\r\n)) which need to be retained.
Sample lines might look as follows:
"1234567890123456"|"2016-07-30"|"2016-08-01"|"123"|"456"|"789"|"text narrative field starts\r\n
with text lines that may have embedded double quotes "For example"\r\n
and may include measurements such as 1/2" x 2" with \r\n
the text continuing and includes embedded line breaks \r\n
which will finally be terminated with a double quote"\r\n
"9876543210654321"|"2017-01-31"|"2018-08-01"|"123"|"456"|"789"|"text narrative field"\r\n
"2345678901234567"|"...."\r\n
with the objective to have the output appear as follows:
~1234567890123456~|~2016-07-30~|~2016-08-01~|~123~|~456~|~789~|~text narrative field starts\r\n
with text lines that may have embedded double quotes ""For example""\r\n
and may include measurements such as 1/2"" x 2"" with \r\n
the text continuing and includes embedded line breaks \r\n
which will finally be terminated with a double quote~\r\n
~9876543210654321~|~2017-01-31~|~2018-08-01~|~123~|~456~|~789~|~text narrative field~\r\n
~2345678901234567~|~....~\r\n
The solution I was attempting to implement was to:
SUCCESSFUL: change all the "|" sequences to ~|~
SUCCESSFUL: change the double quote (")at the start of the first line and end of the last line to a tilde (~)
change the ending and starting double quotes to tildes for any lines ending in a double quote at the end of the first line and terminated with a CR (\r\n) (eg. ..."\r\n) and the next line begins with a double quote, followed by 16 digit number and a tilde (eg. "1234567890123456~...) (i.e. it is the start of a new record)
convert all remaining double quote characters to two successive double quotes (change " to "")
then reverse the first 3 steps above changing all ~ back to double quotes.
I started by using sed to replace all strings with double quote, followed by a pipe, followed by a double quote (i.e. "|") with a tilde, pipe, tilde (i.e. ~|~). I then manually replaced the first and last doublequote in the file with a tilde.
This is where I ran into issues as I tried to count the number of occurrences where a line ends with a doublequote(") and the start of the next line begins with a doublequote followed by a 16 digit number and a "~" which will tell me the actual number of csv records in the file (minus one) as opposed to the number of lines. I attempted to do this using grep: grep '"\r\n"\d{16}~' | wc -l but that didn't work
I then need to replace those double quotes wherein a double quote ends a record and the succeeding record begins with a double quote followed by a 16 digit number and a "~" leaving everything else intact.
I tried to use sed: sed 's/"\r\n"(\d{16}~)/~\r\n~\1' windows_file.txt but it is not working as hoped.
I would welcome any recommendations as to how to accomplish the above.
The script below does what you expect using awk, except for the very last line in the file since it does not know where that record ends.
It could be fixed counting lines in the file but would be impractical since it's a big file.
Looking at data structure records are separated by "\r\n" and fields by "|" let's use that with awk.
gawk 'BEGIN{
RS="\"\r\n\"" # input record separator RS, 2 double quotes with a DOS line ending in the middle
FS="\"\\|\"" # input field separator FS, 2 double quotes with a pipe in the middle
ORS="~\r\n~" # your record separator
OFS="~|~" # your field separator
} {
$1=$1 # trick awk into believing something has changed
if (NR == 1){ # first record, replace first character
print "~" substr($0,2)
}else{
print $0
}
} ' test.txt
Result (assuming lines end with \r\n):
~1234567890123456~|~2016-07-30~|~2016-08-01~|~123~|~456~|~789~|~text narrative field starts
with text lines that may have embedded double quotes "For example"
and may include measurements such as 1/2" x 2" with
the text continuing and includes embedded line breaks
which will finally be terminated with a double quote~
~9876543210654321~|~2017-01-31~|~2018-08-01~|~123~|~456~|~789~|~text narrative field~
~10654321~|~2018-09-31~|~2018-08-01~|~123~|~456~|~789~|~asdasdasdasdad asasda"
~
~
PS: will break if a field contains a line that starts with " and the preceding line within the same ends with "\r\n since the pattern will match the proposed RS.
"10654321"|"2018-09-31"|"2018-08-01"|"123"|"456"|"789"|"asdasdasdasdad asasda"\r\n
"some more"\r\n
"22222"|".... (another record)
Say I have the following string:
"abcdefghijklmnopqrstuvwxyz"
And I think its too long for one line in my YAML file, is there some way to split that over several lines?
>-
abcdefghi
jklmnopqr
stuvwxyz
Would result in "abcdefghi jklmnopqr stuvwxyz" which is close, but it shouldn't have any spaces.
Use double-quotes, and escape the newline:
"abcdefghi\
jklmnopqr\
stuvwxyz"
There are some subtleties that Jesse's answer will miss.
YAML (like many programming languages) treats single and double quotes differently. Consider this document:
regexp: "\d{4}"
This will fail to parse with an error such as:
found unknown escape character while parsing a quoted scalar at line 1 column 9
Compare that to:
regexp: '\d{4}'
Which will parse correctly. In order to use backslash character inside double-quoted strings you would need to escape them, as in:
regexp: "\\d{4}"
I'd also like to highlight Steve's comment about single-quoted strings. Consider this document:
s1: "this\
is\
a\
test"
s2: 'this\
is\
a\
test'
When parsed, you will find that it is equivalent to:
s1: thisisatest
s2: "this\\ is\\ a\\ test"
This is a direct result of the fact that YAML treats single-quoted strings as literals, while double-quoted strings are subject to escape character expansion.
Is there an easy way, using a subroutine maybe, to print a string in Perl without escaping every special character?
This is what I want to do:
print DELIMITER <I don't care what is here> DELIMITER
So obviously it will great if I can put a string as a delimiter instead of special characters.
perldoc perlop, under "Quote and Quote-like Operators", contains everything you need.
While we usually think of quotes as literal values, in Perl they function as operators, providing various kinds of interpolating and pattern matching
capabilities. Perl provides customary quote characters for these behaviors, but also provides a way for you to choose your quote character for any of
them. In the following table, a "{}" represents any pair of delimiters you choose.
Customary Generic Meaning Interpolates
'' q{} Literal no
"" qq{} Literal yes
`` qx{} Command yes*
qw{} Word list no
// m{} Pattern match yes*
qr{} Pattern yes*
s{}{} Substitution yes*
tr{}{} Transliteration no (but see below)
<<EOF here-doc yes*
* unless the delimiter is ''.
$str = q(this is a "string");
print $str;
if you mean quotes and apostrophes with 'special characters'
You can use the __DATA__ directive which will treat all of the following lines as a file that can be accessed from the DATA handle:
while (<DATA>) {
print # or do something else with the lines
}
__DATA__
#!/usr/bin/perl -w
use Some::Module;
....
or you can use a heredoc:
my $string = <<'END'; #single quotes prevent any interpolation
#!/usr/bin/perl -b
use Some::Module;
....
END
The printing is not doing special things to the escapes, double quoted strings are doing it. You may want to try single quoted strings:
print 'this is \n', "\n";
In a single quoted string the only characters that must be escaped are single quotes and a backslash that occurs immediately before the end of the string (i.e. 'foo\\').
It is important to note that interpolation does not work with single quoted strings, so
print 'foo is $foo', "\n";
Will not print the contents of $foo.
You can pretty much use any character you want with q or qq. For example:
#!/usr/bin/perl
use utf8;
use strict; use warnings;
print q∞This is a test∞;
print qq☼\nThis is another test\n☼;
print q»But, what is the point?»;
print qq\nYou are just making life hard on yourself!\n;
print qq¿That last one is tricky\n¿;
You cannot use qq DELIMITER foo DELIMITER. However, you could use heredocs for a similar effect:
print <<DELIMITER
...
DELIMETER
;
or
print <<'DELIMETER'
...
DELIMETER
;
but your source code would be really ugly.
If you want to print a string literally and you have Perl 5.10 or later then
say 'This is a string with "quotes"' ;
will print the string with a newline.. The importaning thing is to use single quotes ' ' rather than double ones " "