I am developing on mac and use the following command to confirm file names to a json array:
ls **/*.test.json | jq -R -s -c 'split("\n")[:-1]'
Which gives me the json array:
['folder1/a.test.json', 'folder2/b.test.json', 'c.test.json']
Which is exactly what I want. However, when executing on github action (with linux), the above command produces outcome:
['c.test.json]
And the files within folders are not included.
I confirmed that the folders where checkout successfully because echo $(ls folder1) gives a.test.json.
What is the best way to achieve what I want for the command?
** is not a standard sh feature; your Github action probably requires a POSIX shell script.
To traverse arbitrarily deep directory structure with a shell which doesn't support **, try find:
find . -name '*.test.json' -print |
jq -R -s -c 'split("\n")[:-1]'
If there is only a limited set of directory levels, maybe try
printf '%s\n' */*.test.json *.test.json | jq ...
(Also don't use ls in scripts and perhaps see also useless use of echo.)
Both of these have some gnarly corner cases if you have file names which contain newlines; find could probably be coerced to handle that case, too, but I'll not complicate this answer further; perhaps see https://mywiki.wooledge.org/BashFAQ/020 for a fuller discussion.
If your shell is Bash, ** is available, but typically not enabled out of the box; you enable it with shopt -s globstar ... but perhaps it's better to stick to proper sh in case Github changes the default shell for Actions.
Related
I am trying to get the most recent .CSV or .csv file name among other comma separated value files where the extension name is case insensitive.
I am achieving this with the following command, provided by someone else without any explanation:
ls -t ~(i:*.CSV) | head -1
or
ls -t -- ~(i:*.CSV) | head -1
I have two questions:
What is the use of ~ and -- in this case? Does -- helps here?
How can I get a blank response when there is no .csv or .CSV file in
the folder? At the moment I get:
/bin/ls: cannot access ~(i:*.CSV): No such file or directory
I know I can test the exit code of the last command, but I was wondering maybe there is a --silent option or something.
Many thanks for your time.
PS: I made my research online quite thorough and I was unable to find an answer.
The ~ is just a literal character; the intent would appear to be to match filenames starting with ~ and ending with .csv, with i: being a flag to make the match case-insensitive. However, I don't know of any shell that supports that particular syntax. The closest thing I am aware of would be zsh's globbing flags:
setopt extended_glob # Allow globbing flags
ls ~(#i)*.csv
Here, (#i) indicates that anything after it should be matched without regard to case.
Update: as #baptistemm points out, ~(i:...) is syntax defined by ksh.
The -- is a conventional argument, supported by many commands, to mean that any arguments that follow are not options, but should be treated literally. For example, ls -l would mean ls should use the -l option to modify its output, while ls -- -l means ls should try to list a file named -l.
~(i:*.CSV) is to tell to shell (this is only supported apparently in ksh93) the enclosed text after : must be treated as insensitive, so in this example that could all these possibilites.
*.csv or
*.Csv or
*.cSv or
*.csV or
*.CSv or
*.CSV
Note this could have been written ls -t *.[CcSsVv] in bash.
To silent errors I suggest you to look for in this site for "standard error /dev/null" that will help.
I tried running commands like what you have in both bash and zsh and neither worked, so I can't help you out with that, but if you want to discard the error, you can add 2>/dev/null to the end of the ls command, so your command would look like the following:
ls -t ~(i:*.CSV) 2>/dev/null | head -1
This will redirect anything written to STDERR to /dev/null (i.e. throw it out), which, in your case, would be /bin/ls: cannot access ~(i:*.CSV): No such file or directory.
In bash (or zsh), is there a way to 'look back' on previous output?
Say I have the following:
$ find . -depth 1 -name "d*"
dir1
dir2
dir3
$ cd [3rd line]
Can I pluck 'dir3' from the previous output without repeating the previous command and using a pipe? Just sort of "grab it" from onscreen?
The shell doesn't see that output at all. When findruns, it uses the file handle that it inherits from the shell to write its output, but that isn't visible to the shell (though even if it were, the shell isn't caching it).
However, given that you are using zsh, the first question you should ask involving an external command is, "Do I even need the external command?" As is often the case, you don't here. Instead of using find, just use print and a glob to get the same listing:
% print -l d*
then use the same glob again with a qualifier to select the 3rd result.
% cd d*([3])
(This assumes that the expansion of d* won't change between running the print command and running the cd command.)
Or, you can cache the result yourself:
% choices=( d* )
% print -l $choices
dir1
dir2
dir3
% cd $choices[3]
The array-based solution could work in bash as well, though with slightly different syntax (and assuming your find command is simple enough to be replaced with one of bash's far less powerful globs.)
Finally, there is the select command:
select d in d*; do
cd "$d"; break;
done
which will also work as-is in bash, again assuming a valid glob.
A cursory search reveals neither bash nor zsh provide this functionality. Happy to be corrected if there's a way to do this. :)
ls /tmp
How can I run the same command but using reverse polish notation?
Is there a mode that would allow me to do this or something similar to that?
I could use xargs but that's a lot more typing:
echo /tmp | xargs ls
This would be ideal:
/tmp ls
or
/tmp | ls
Bash (I assume you are using it) is a shell for unixoid systems.
As far as I know, bash doesn't provide such a mode. You could use a different shell that provides this feature. Searching in the web, this was my first result: https://github.com/iconmaster5326/RPOS, but maybe it is far from stable ;)
Alternatively, you can make a command that reverses it's argument list and execute it.
The usage would be like this:
reversex /tmp ls
reversex A.txt B.txt cp
Here is an example of such a command:
#!/bin/bash
for i in "$#"
do
CMDLINE="$i $CMDLINE"
done
$CMDLINE
If you name it /usr/local/bin/reversex and make it executable, you should be able to use simple reverse commands with the prefix reversex. I can not give a warranty that it works. Note that the arguments are parsed twice and have to be escaped twice, too.
Using ubuntu 10.10 I have the following that I run on the command-line:
result="$(ls -d !(*.*))"
chmod +x $result
This gets a list of files that have no extensions and makes them executable.
But when I move it to a script file (shell) it does not work. From what I have read around the forum this is something to do with command substitution being run in a different a subshell.
But I could not find a solution yet that works in my scrpt :(
So how do you get the result of a command and store it in a variable within a script?
(Since #user000001 does not seem to write their comment into an answer, I'll do the toiling of writing the answer. So credit should got to them, though.)
The feature you are using is the extglob (extended globbing) feature of the bash. This is per default enabled for interactive shells, and per default disabled for non-interactive shells (i. e. shell scripts). To enable it, use the command shopt -s extglob.
Note that this command only has effect for lines below it:
shopt -s extglob
ls -d !(*.*)
It does not effect parsing of the same line:
shopt -s extglob; ls -d !(*.*) # won't work!!
In general I want to warn about using such special features of the bash. It makes the code rather unportable. I'd propose to use POSIX features and tools instead which enable porting the code to another platform rather easily, and they also represent a certain subset of possibilities more developers understand without having to consult the documentation first.
What you want to achieve could also be done using find. This also has the advantage of being unproblematic in combination with strange file names (e. g. containing spaces, quotes, etc.):
find . -maxdepth 1 -type f -name '*.*' -o -exec chmod +x "{}" \;
Is it possible to get, using Bash, a list of commands starting with a certain string?
I would like to get what is printed hitting <tab> twice after typing the start of the command and, for example, store it inside a variable.
You should be able to use the compgen command, like so:
compgen -A builtin [YOUR STRING HERE]
For example, "compgen -A builtin l" returns
let
local
logout
You can use other keywords in place of "builtin" to get other types of completion. Builtin gives you shell builtin commands. "File" gives you local filenames, etc.
Here's a list of actions (from the BASH man page for complete which uses compgen):
alias Alias names. May also be specified as -a.
arrayvar Array variable names.
binding Readline key binding names.
builtin Names of shell builtin commands. May also be specified as -b.
command Command names. May also be specified as -c.
directory Directory names. May also be specified as -d.
disabled Names of disabled shell builtins.
enabled Names of enabled shell builtins.
export Names of exported shell variables. May also be specified as -e.
file File names. May also be specified as -f.
function Names of shell functions.
group Group names. May also be specified as -g.
helptopic Help topics as accepted by the help builtin.
hostname Hostnames, as taken from the file specified by the HOSTFILE shell
variable.
job Job names, if job control is active. May also be specified as
-j.
keyword Shell reserved words. May also be specified as -k.
running Names of running jobs, if job control is active.
service Service names. May also be specified as -s.
setopt Valid arguments for the -o option to the set builtin.
shopt Shell option names as accepted by the shopt builtin.
signal Signal names.
stopped Names of stopped jobs, if job control is active.
user User names. May also be specified as -u.
variable Names of all shell variables. May also be specified as -v.
A fun way to do this is to hit M-* (Meta is usually left Alt).
As an example, type this:
$ lo
Then hit M-*:
$ loadkeys loadunimap local locale localedef locale-gen locate
lockfile-create lockfile-remove lockfile-touch logd logger login
logname logout logprof logrotate logsave look lorder losetup
You can read more about this in man 3 readline; it's a feature of the readline library.
If you want exactly how bash would complete
COMPLETIONS=$(compgen -c "$WORD")
compgen completes using the same rules bash uses when tabbing.
JacobM's answer is great. For doing it manually, i would use something like this:
echo $PATH | tr : '\n' |
while read p; do
for i in $p/mod*; do
[[ -x "$i" && -f "$i" ]] && echo $i
done
done
The test before the output makes sure only executable, regular files are shown. The above shows all commands starting with mod.
Interesting, I didn't know about compgen. Here a script I've used to do it, which doesn't check for non-executables:
#!/bin/bash
echo $PATH | tr ':' '\0' | xargs -0 ls | grep "$#" | sort
Save that script somewhere in your $PATH (I named it findcmd), chmod u+w it, and then use it just like grep, passing your favorite options and pattern:
findcmd ^foo # finds all commands beginning with foo
findcmd -i -E 'ba+r' # finds all commands matching the pattern 'ba+r', case insensitively
Just for fun, another manual variant:
find -L $(echo $PATH | tr ":" " ") -name 'pattern' -type f -perm -001 -print
where pattern specifies the file name pattern you want to use. This will miss commands that are not globally executable, but which you have permission for.
[tested on Mac OS X]
Use the -or and -and flags to build a more comprehensive version of this command:
find -L $(echo $PATH | tr ":" " ") -name 'pattern' -type f
\( \
-perm -001 -or \
\( -perm -100 -and -user $(whoami)\) \
\) -print
will pick up files you have permission for by virtue of owning them. I don't see a general way to get all those you can execute by virtue of group affiliation without a lot more coding.
Iterate over the $PATH variable and do ls beginningofword* for each directory in the path?
To get it exactly equivalent, you would need to filter out only executable files and sort by name (should be pretty easy with ls flags and the sort command).
What is listed when you hit are the binary files in your PATH that start with that string. So, if your PATH variable contains:
PATH=/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/lib/java/bin:/usr/lib/java/jre/bin:/usr/lib/qt/bin:/usr/share/texmf/bin:.
Bash will look in each of those directories to show you the suggestions once you hit . Thus, to get the list of commands starting with "ls" into a variable you could do:
MYVAR=$(ls /usr/local/bin/ls* /usr/bin/ls* /bin/ls*)
Naturally you could add all the other directories I haven't.