Insert quoted and unquoted parts of string in table - string

Question

I've been working on this part of a saycommand system which is supposed to separate parts of a string and put them in a table which is sent to a function, which is queried at the beginning of the string. This would look like, for example, !save 1 or !teleport 0 1, or !tell 5 "a private message".
I would like this string to turn into a table:
[[1 2 word 2 9 'more words' 1 "and more" "1 2 34"]]
(Every non-quoted part of the string gets its own key, and the quoted parts get grouped into a key)
1 = 1
2 = 2
3 = word
4 = 2
5 = 9
6 = more words
7 = 1
8 = and more
9 = 1 2 34
I've tried doing this with Lua pattern, but I'm stuck trying to find out how to capture both quoted and unquoted pieces of the string. I've tried a lot of things, but nothing helped.
My current pattern attempts look like this:
a, d = '1 2 word 2 9 "more words" 1 "and more" "1 2 34"" ', {}
-- previous attempts
--[[
This one captures quotes
a:gsub('(["\'])(.-)%1', function(a, b) table.insert(d, b) end)
This one captures some values and butchered quotes,
which might have to do with spaces in the string
a:gsub('(["%s])(.-)%1', function(a, b) table.insert(d, b) end)
This one captures every value, but doesn't take care of quotes
a:gsub('(%w+)', function(a) table.insert(d, a) end)
This one tries making %s inside of quotes into underscores to
ignore them there, but it doesn't work
a = a:gsub('([%w"\']+)', '%1_')
a:gsub('(["\'_])(.-)%1', function(a, b) table.insert(d, b) end)
a:gsub('([%w_]+)', function(a) table.insert(d, a) end)
This one was a wild attempt at cracking it, but no success
a:gsub('["\']([^"\']-)["\'%s]', function(a) table.insert(d, a) end)
--]]
-- This one adds spaces, which would later be trimmed off, to test
-- whether it helped with the butchered strings, but it doesn't
a = a:gsub('(%w)(%s)(%w)', '%1%2%2%3')
a:gsub('(["\'%s])(.-)%1', function(a, b) table.insert(d, b) end)
for k, v in pairs(d) do
print(k..' = '..v)
end
This would not be needed for simple commands, but a more complex one like !tell 1 2 3 4 5 "a private message sent to five people" does need it, first to check if it's sent to multiple people and next to find out what the message is.
Further down the line I want to add commands like !give 1 2 3 "component:material_iron:weapontype" "food:calories", which is supposed to add two items to three different people, would benefit greatly from such a system.
If this is impossible in Lua pattern, I'll try doing it with for loops and such, but I really feel like I'm missing something obvious. Am I overthinking this?

Answer 1

You cannot process quoted strings with Lua patterns. You need to parse the string explicitly, as in the code below.
function split(s)
local t={}
local n=0
local b,e=0,0
while true do
b,e=s:find("%s*",e+1)
b=e+1
if b>#s then break end
n=n+1
if s:sub(b,b)=="'" then
b,e=s:find(".-'",b+1)
t[n]=s:sub(b,e-1)
elseif s:sub(b,b)=='"' then
b,e=s:find('.-"',b+1)
t[n]=s:sub(b,e-1)
else
b,e=s:find("%S+",b)
t[n]=s:sub(b,e)
end
end
return t
end
s=[[1 2 word 2 9 'more words' 1 "and more" "1 2 34"]]
print(s)
t=split(s)
for k,v in ipairs(t) do
print(k,v)
end

Answer 2

Lua string patterns and regex for that matter generally aren't well suited when you need to do parsing that requires varying nesting levels or token count balancing like parenthesis ( ). But there is another tool available to Lua that's powerful enough to deal with that requirement: LPeg.
The LPeg syntax is a bit archaic and takes some getting use to so I'll use the lpeg re module instead to make it easier to digest. Keep in mind that anything you can do in one form of the syntax you can also express in the other form as well.
I'll start by defining the grammar for parsing your format description:
local re = require 're'
local cmdgrammar =
[[
saycmd <- '!' cmd extra
cmd <- %a%w+
extra <- (singlequote / doublequote / unquote / .)*
unquote <- %w+
singlequote <- "'" (unquote / %s)* "'"
doublequote <- '"' (unquote / %s)* '"'
]]
Next, compile the grammar and use it to match some of your test examples:
local cmd_parser = re.compile(cmdgrammar)
local saytest =
{
[[!save 1 2 word 2 9 'more words' 1 "and more" "1 2 34"]],
[[!tell 5 "a private message"]],
[[!teleport 0 1]],
[[!say 'another private message' 42 "foo bar" baz]],
}
There are currently no captures in the grammar so re.match returns the last character position in the string it was able to match up to + 1. That means a successful parse will return the full character count of the string + 1 and therefore is a valid instance of your grammar.
for _, test in ipairs(saytest) do
assert(cmd_parser:match(test) == #test + 1)
end
Now comes the interesting part. Once you have the grammar working as desired you can now add captures that automatically extracts the results you want into a lua table with relatively little effort. Here's the final grammar spec + table captures:
local cmdgrammar =
[[
saycmd <- '!' {| {:cmd: cmd :} {:extra: extra :} |}
cmd <- %a%w+
extra <- {| (singlequote / doublequote / { unquote } / .)* |}
unquote <- %w+
singlequote <- "'" { (unquote / %s)* } "'"
doublequote <- '"' { (unquote / %s)* } '"'
]]
Running the tests again and dumping the re.match results:
for i, test in ipairs(saytest) do
print(i .. ':')
dump(cmd_parser:match(test))
end
You should get output similar to:
lua say.lua
1:
{
extra = {
"1",
"2",
"word",
"2",
"9",
"more words",
"1",
"and more",
"1 2 34"
},
cmd = "save"
}
2:
{
extra = {
"5",
"a private message"
},
cmd = "tell"
}
3:
{
extra = {
"0",
"1"
},
cmd = "teleport"
}
4:
{
extra = {
"another private message",
"42",
"foo bar",
"baz"
},
cmd = "say"
}