I am using the switch statement to match some options which may or may not have VALUES associated with them, then extracting the values (which I imagine could be nothing or just a bunch of empty strings).
In Tcl are these equivalent? I.e. will trailing white space be fed as an option or parsed out? Should I "string trim" the value or is this unnecessary?
**
By default, whitespace is not trimmed from the start or end of any strings. After all, it can sometimes be significant. If you do want to strip the leading and trailing whitespace, use:
set str [string trim $str]
You can just strip the leading space or trailing space by using string trimleft and string trimright respectively.
In Tcl is whitespace equal to Empty
string?
Whitespace is not equal to an empty string. Using TCL 8.5.2, the expression expr {"" eq " "} evaluates to 0.
will trailing white space be fed as an
option or parsed out?
If you are accepting the strings as-is and performing no processing on them, then you will get any whitespace that is present.
Should I "string trim" the value or is
this unnecessary?
The answer to this depends on your application. If the whitespace is not significant, trim it away using the strim trim command (as Donal Fellows mentioned). Doing so will likely simplify your logic. You can also use regsub -all {\s+} $input_string { } to collapse all repeated whitespace characters inside of a string.
Related
I have a string looks like this:
(The whole code block is a string, aka, this string contains quotation marks.)
"he\"llo", "world\n", "fro,m"
[update] Aka, the "actual" string is this:
"\"he\\\"llo\", \"world\\n\", \"fro,m\""
I want to get an array of strings like this:
[ "\"he\\\"llo\"", "\"world\\n\"", "\"fro,m\"" ]
[update] Comma inside quotation marks should be remained.
In my opinion, there are several ways to solve this:
build a automata (DFA or NFA) for this syntax
using several status flags like inQuote, handle judging logics with lots of if else
write a complex but clever Regular Expression for this
Are there any general solutions to this problem? Or how should I actually do using those thinkings above?
P.S. It couldn't be better if some syntax errors like "unclosed quotation mark" can be found.
You need to first define your grammar. This is a simple grammar for your case:
document = *WS [string *WS *(',' *WS string *WS)]
string = %x22 *char %x22
char = %x20-21 / %x23-5B / escape / %x5D-10FFFF
escape = %x5C (%x5C / %x22 / 't' / 'n' / 'r')
WS = %x9 / %x20
You can read it as:
A document may begin/end with a white space, then may have one or more strings separated by commas. Before and after each comma there may be some white space.
A string is made of characters and begins and ends with double quotes Unicode/ASCII hex code 22.
Each character (char), may be: 1) any non control Unicode character before the double quotes i.e. hex 20 (space) or hex 21 (exclamation mark); 2) any character after the double quotes and before the escape slash \ (hex 5C); 3) an escape character sequence; 4) any other Unicode character after the slash (hex 5C).
The escape sequence (rule escape) begins with the escape slash \ and is followed by another slash, or the characters t for tab, n for line feed and r for carriage return. You may add and other escapable characters if you want, as for a C++ string syntax you can see here: https://en.cppreference.com/w/cpp/language/escape .
A white space (WS) is a tab or space, you may add and %xA and %xD for line feed and carriage return respectively.
By the use of this grammar you will get this tree for your input:
The screenshort is from the Tunnel Grammar Studio online laboratory that can run ABNF grammars (as the one above), and I work on.
After you have the grammar, you may use tools to generate a parser, or you can write one yourself. If you want to do it by hand (preferable for so small and simple grammar), you may have one function per each grammar rule that reads one character and checks is it the expected one. If your input finishes when you are parsing the string rule, then you have an input with a started but not finished string.
Your actual string syntax tree will look like that:
I've been learning ksh for quite some time but I still cannot understand %(pattern-pair(s)) in the manual. Anyone can give a simple meaningful example?
A pattern of the form %(pattern-pair(s)) is a sub-pattern that can be
used to match nested character expressions. Each pattern-pair is a two
character sequence which cannot contain & or |. The first pattern-pair
specifies the starting and ending characters for the match. Each sub-
sequent pattern-pair represents the beginning and ending characters of
a nested group that will be skipped over when counting starting and
ending character matches. The behavior is unspecified when the first
character of a pattern-pair is alpha-numeric except for the following:
D Causes the ending character to terminate the search for
this pattern without finding a match.
E Causes the ending character to be interpreted as an
escape character.
L Causes the ending character to be interpreted as a quote
character causing all characters to be ignored when look-
ing for a match.
Q Causes the ending character to be interpreted as a quote
character causing all characters other than any escape
character to be ignored when looking for a match.
I suppose it's useful for JSON parsing:
json='{"foo":{"bar":"baz"} }'
#remove all quoted values nested in punctuation except the first
jsonroot=${json%%[[:punct:]][[:punct:]]%(\"\")*}
#remove initial curly
jsonroot=${jsonroot#?}
So I have this string:
{"scores":{"1":["John",60],"2":["Jude",60],"3":["Max",60],"4":["Kyle",60],"5":["Smith",60],"6":["Mark",50],"7":["Luke",40],"8":["Anne",30],"9":["Bruce",20],"10":["kazuo",10]}}
There are a number of integers there that have quotes around them, and I want to get rid of them. How do I do that? I already tried out:
print(string.gsub(string, '/"(\d)"/', "%1"));
but it does not work. :(
Lua does not have regular expressions like Perl, instead, it does have patterns. These are similar with a few differences.
There is no need for delimiting slashes / /, and the escaping character is % but not \. Otherwise, your trial is essentially correct:
print(string.gsub(str, '"(%d+)"', "%1"))
Where str is the variable containing the input string. Also note that string.gsub returns 2 values, which are both printed, the second result being the number of substitutions. Use an extra pair of parentheses to keep only the first result.
You can simplify a little the notation using the colon : operator :
print((str:gsub('"(%d+)"', "%1")))
While running an R-plugin in SPSS, I receive a Windows path string as input e.g.
'C:\Users\mhermans\somefile.csv'
I would like to use that path in subsequent R code, but then the slashes need to be replaced with forward slashes, otherwise R interprets it as escapes (eg. "\U used without hex digits" errors).
I have however not been able to find a function that can replace the backslashes with foward slashes or double escape them. All those functions assume those characters are escaped.
So, is there something along the lines of:
>gsub('\\', '/', 'C:\Users\mhermans')
C:/Users/mhermans
You can try to use the 'allowEscapes' argument in scan()
X=scan(what="character",allowEscapes=F)
C:\Users\mhermans\somefile.csv
print(X)
[1] "C:\\Users\\mhermans\\somefile.csv"
As of version 4.0, introduced in April 2020, R provides a syntax for specifying raw strings. The string in the example can be written as:
path <- r"(C:\Users\mhermans\somefile.csv)"
From ?Quotes:
Raw character constants are also available using a syntax similar to the one used in C++: r"(...)" with ... any character sequence, except that it must not contain the closing sequence )". The delimiter pairs [] and {} can also be used, and R can be used in place of r. For additional flexibility, a number of dashes can be placed between the opening quote and the opening delimiter, as long as the same number of dashes appear between the closing delimiter and the closing quote.
First you need to get it assigned to a name:
pathname <- 'C:\\Users\\mhermans\\somefile.csv'
Notice that in order to get it into a name vector you needed to double them all, which gives a hint about how you could use regex. Actually, if you read it in from a text file, then R will do all the doubling for you. Mind you it not really doubling the backslashes. It is being stored as a single backslash, but it's being displayed like that and needs to be input like that from the console. Otherwise the R interpreter tries (and often fails) to turn it into a special character. And to compound the problem, regex uses the backslash as an escape as well. So to detect an escape with grep or sub or gsub you need to quadruple the backslashes
gsub("\\\\", "/", pathname)
# [1] "C:/Users/mhermans/somefile.csv"
You needed to doubly "double" the backslashes. The first of each couple of \'s is to signal to the grep machine that what next comes is a literal.
Consider:
nchar("\\A")
# returns `[1] 2`
If file E:\Data\junk.txt contains the following text (without quotes): C:\Users\mhermans\somefile.csv
You may get a warning with the following statement, but it will work:
texinp <- readLines("E:\\Data\\junk.txt")
If file E:\Data\junk.txt contains the following text (with quotes): "C:\Users\mhermans\somefile.csv"
The above readlines statement might also give you a warning, but will now contain:
"\"C:\Users\mhermans\somefile.csv\""
So, to get what you want, make sure there aren't quotes in the incoming file, and use:
texinp <- suppressWarnings(readLines("E:\\Data\\junk.txt"))
i need to create a string to store couples of key/value data, for example:
key1::value1||key2::value2||key3::value3
in deserializing it, i may encounter an error if the key or the value happen to contain || or ::
What are common techniques to deal with such situation? thanks
A common way to deal with this is called an escape character or qualifier. Consider this Comma-Separated line:
Name,City,State
John Doe, Jr.,Anytown,CA
Because the name field contains a comma, it of course gets split improperly and so on.
If you enclose each data value by qualifiers, the parser knows when to ignore the delimiter, as in this example:
Name,City,State
"John Doe, Jr.",Anytown,CA
Qualifiers can be optional, used only on data fields that need it. Many implementations will use qualifiers on every field, needed or not.
You may want to implement something similar for your data encoding.
Escape || when serializing, and unescape it when deserializing. A common C-like way to escape is to prepend \. For example:
{ "a:b:c": "foo||bar", "asdf": "\\|||x||||:" }
serialize => "a\:b\:c:foo\|\|bar||asdf:\\\\\|\|\|x\|\|\|\|\:"
Note that \ needs to be escaped (and double escaped due to being placed in a C-style string).
If we assume that you have total control over the input string, then the common way of dealing with this problem is to use an escape character.
Typically, the backslash-\ character is used as an escape to say that "the next character is a special character", so in this case it should not be used as a delimiter. So the parser would see || and :: as delimiters, but would see \|\| as two pipe characters || in either the key or the value.
The next problem is that we have overloaded the backslash. The problem is then, "how do I represent a backslash". This is sovled by saying that the backslash is also escaped, so to represent a \, you would have to say \\. So the parser would see \\ as \.
Note that if you use escape characters, you can use a single character for the delimiters, which might make things simpler.
Alternatively, you may have to restict the input and say that || and :: are just baned and fail/remove when the string is encoded.
A simple solution is to escape a separator (with a backslash, for instance) any time it occurs in data:
Name,City,State
John Doe\, Jr.,Anytown,CA
Of course, the separator will need to be escaped when it occurs in data as well; in this case, a backslash would become \\.
You can use non-ascii character as separator (e.g. vertical tab :-) ).
You can escape separator character in your data during serialization. For example: if you use one character as separator (key1:value1|key2:value2|...) and your data is:
this:is:key1 this|is|data1
this:is:key2 this|is|data2
you double every colon and pipe character in you data when you serialize it. So you will get:
this::is::key1:this||is||data1|this::is::key2:this||is||data2|...
During deserialization whenever you come across two colon or two pipe characters you know that this is not your separator but part of your data and that you have to change it to one character. On the other hand, every single colon or pipe character is you separator.
Use a prefix (say "a") for your special characters (say "b") present in the key and values to store them. This is called escaping.
Then decode the key and values by simply replacing any "ab" sequence with "b". Bear in mind that the prefix is also a special character. An example:
Prefix: \
Special characters: :, |, \
Encoded:
title:Slashdot\: News for Nerds. Stuff that Matters.|shortTitle:\\.
Decoded:
title=Slashdot: News for Nerds. Stuff that Matters.
shortTitle=\.
The common technique is escaping reserved characters, for example:
In urls you escape some characters
using %HEX representation:
http://example.com?aa=a%20b
In programming languages you escape
some characters with a slash prefix:
"\"hello\""