Newlines not quoted properly in ls -Q - linux

Question

Using ls -Q with --quoting-style=shell, newlines in file names (yes, I know...) are turned into ?. Is this a bug? Is there a way how to get the file names in a format that's 100% compatible with a shell (sh or bash if possible)?
Example (bash):
$ touch a$'\n'b
$ for s in literal shell shell-always c c-maybe escape locale clocale ; do
ls -Q a?b --quoting-style=$s
done
a?b
'a?b'
'a?b'
"a\nb"
"a\nb"
a\nb
‘a\nb’
‘a\nb’

Answer 1

coreutils 8.25 has the new 'shell-escape' quoting style, and in fact enables it by default to allow the output from ls to be always usable, and to be safe to copy and paste back to other commands.

Answer 2

Maybe not quite what you are looking for, but the "escape" style seems to work well with the upcoming ${...#E} parameter expansion in bash 4.4.
$ touch $'a\nb' $'c\nd'
$ ls -Q --quoting-style=escape ??? | while IFS= read -r fname; do echo =="${fname#E}==="; done
==a
b==
==c
d==
Here is the relevant part of the man page (link is to the raw source):
${parameter#operator}
Parameter transformation. The expansion is either a transforma-
tion of the value of parameter or information about parameter
itself, depending on the value of operator. Each operator is a
single letter:
Q The expansion is a string that is the value of parameter
quoted in a format that can be reused as input.
E The expansion is a string that is the value of parameter
with backslash escape sequences expanded as with the
$'...' quoting mechansim.
P The expansion is a string that is the result of expanding
the value of parameter as if it were a prompt string (see
PROMPTING below).
A The expansion is a string in the form of an assignment
statement or declare command that, if evaluated, will
recreate parameter with its attributes and value.
a The expansion is a string consisting of flag values rep-
resenting parameter's attributes.
If parameter is # or *, the operation is applied to each posi-
tional parameter in turn, and the expansion is the resultant
list. If parameter is an array variable subscripted with # or
*, the case modification operation is applied to each member of
the array in turn, and the expansion is the resultant list.
The result of the expansion is subject to word splitting and
pathname expansion as described below.

Answer 3

From a bit of experimentation, it looks like --quoting-style=escape is compatible with being wrapped in $'...', with two exceptions:
it escapes spaces by prepending a backslash; but $'...' doesn't discard backslashes before spaces.
it doesn't escape single-quotes.
So you could perhaps write something like this (in Bash):
function ls-quote-shell () {
ls -Q --quoting-style=escape "$#" \
| while IFS= read -r filename ; do
filename="${filename//'\ '/ }" # unescape spaces
filename="${filename//"'"/\'}" # escape single-quotes
printf "$'%s'\n" "$filename"
done
}
To test this, I've created a directory with a bunch of filenames with weird characters; and
eval ls -l $(ls-quote-shell)
worked as intended . . . though I won't make any firm guarantees about it.
Alternatively, here's a version that uses printf to process the escapes followed by printf %q to re-escape in a shell-friendly manner:
function ls-quote-shell () {
ls -Q --quoting-style=escape "$#" \
| while IFS= read -r escaped_filename ; do
escaped_filename="${escaped_filename//'\ '/ }" # unescape spaces
escaped_filename="${escaped_filename//'%'/%%}" # escape percent signs
# note: need to save in variable, rather than using command
# substitution, because command substitution strips trailing newlines:
printf -v filename "$escaped_filename"
printf '%q\n' "$filename"
done
}
but if it turns out that there's some case that the first version doesn't handle correctly, then the second version will most likely have the same issue. (FWIW, eval ls -l $(ls-quote-shell) worked as intended with both versions.)