I'm trying to debug a shell script that calls an awk file. Which is a nightmare, cause I've never used either before, nor am I very fluent with linux, but anyway
A dev made an awk file and is trying to run it in a shell script.
To try and run it from a separate location, without needing to specify the exact location, they put the awk script in a folder that's in the PATH variable. So that awk file should be available everywhere, right?
When you run it like this...
awk -f script.awk arg1
...can awk find that script? It spits out an error, when the shell script tries to run that awk command:
awk: fatal: can't open source file `script.awk' for reading (No such file or directory)
As you know, awk can't find the script itself.
If the script is marked as executable, and if you have a which command, then you should be able to do:
awk -f `which script.awk` arg1
Alternatively, and probably better, make the script into an executable:
#!/usr/bin/awk -f
BEGIN { x = 23 }
{ x += 2 }
END { print x }
The shebang needs the '-f' option to work on MacOS X 10.7.1 where I tested it:
$ script.awk script.awk
31
$
That gives you a self-contained single-file solution.
No, that's not going to work. awk needs the path to the script to run, it won't use the PATH variable to find it. PATH is only used to find executables.
Related
I have a complex command I am passing via ssh to a remote server. I am trying to unzip a file and then change its naming structure and extension in a second ssh command. The command I have is:
ssh root#server1 "gzip -d /tmp/file.out-20171119.gz; echo file* | awk -F'[.-]' '{print $1$3".log"}'"
Obviously the " around the .log portion of the print statement are failing me. The idea is that I would strip the .out portion from the filename and end up with file20171119.log as an ending result. I am just a bit confused on the syntax or on how to escape that properly so bash interprets the .log appropriately.
The easiest way to deal with this problem is to avoid it. Don't bother trying to escape your script to go on a command line: Pass it on stdin instead.
ssh root#server1 bash -s <<'EOF'
gzip -d /tmp/file.out-20171119.gz
# note that (particularly w/o a cd /tmp) this doesn't do anything at all related to the
# line above; thus, probably buggy as given in the original question.
echo file* | awk -F'[.-]' '{print $1$3".log"}'
EOF
A quoted heredoc -- one with <<'EOF' or <<\EOF instead of <<EOF -- is passed literally, without any shell expansions; thus, $1 or $3 will not be replaced by the calling shell as they would with an unquoted heredoc.
If you don't want to go the avoidance route, you can have the shell do the quoting for you itself. For example:
external_function() {
gzip -d /tmp/file.out-20171119.gz
echo file* | awk -F'[.-]' '{print $1$3".log"}'
}
ssh root#server1 "$(declare -f external_function); external_function"
declare -f prints a definition of a function. Putting that function literally into your SSH command ensures that it's run remotely.
You need to escape the " to prevent them from closing your quoted string early, and you need to escape the $ in the awk script to prevent local parameter expansion.
ssh root#server1 "gzip -d /tmp/file.out-20171119.gz; echo file* | awk -F'[.-]' '{print \$1\$3\".log\"}'"
The most probable reason (as you don't show the contents of the root home directory in the server) is that you are uncompressing the file in the /tmp directory, but feeding to awk filenames that should exist in the root home directory.
" allows escaping sequences with \. so the correct way to do is
ssh root#server1 "gzip -d /tmp/file.out-20171119.gz; echo file* | awk -F'[.-]' '{print \$1\$3\".log\"}'"
(like you wrote in your question) this means the following command is executed with a shell in the server machine.
gzip -d /tmp/file.out-20171119.gz; echo file* | awk - F'[.-]' '{print $1$3".log"}'
You are executing two commands, the first to gunzip /tmp/file.out-2017119.gz (beware, as it will be gunzipped in /tmp). And the second can be the source for the problem. It is echoing all the files in the local directory (this is, the root user home directory, probably /root in the server) that begin with file in the name (probably none), and feeding that to the next awk command.
As a general rule.... test your command locally, and when it works locally, just escape all special characters that will go unescaped, after being parsed by the first shell.
another way to solve the problem is to use gzip(1) as a filter... so you can decide the name of the output file
ssh root#server1 "gzip -d </tmp/file.out-20171119.gz >file20171119.log"
this way you save an awk(1) execution just to format the output file. Or if you have the date from an environment variable.
DATE=`date +%Y%m%d`
ssh root#server1 "gzip -d </tmp/file.out-${DATE}.gz >file${DATE}.log"
Finally, let me give some advice: Don't use /tmp to uncompress files. /tmp is used by several distributions as a high speed temporary dir. It is normally ram based, too quick, but limited space, so uncompressing a log file there can fill up the memory of the kernel used for the ram based filesystem, which is not a good idea. Also, a log file normally expands a lot and /tmp is a local system general directory, where other users can store files named file<something> and you can clash with those files (in case you do searches with wildcard patterns, like you do in your command) Also, it is common once you know the name of the file to assign it to environment variables and use those variables, so case you need to change the format of the filename, you do it in only one place.
I know this seems fairly trivial. But I have no idea where I am going wrong. I have a shell script where I download a package based on the input argument and then extract the package name. This is how I do it:
wget $1
echo $1 | awk -F/ '{print $NF}'
I run it like this bash scrip.sh http://apache.claz.org/phoenix/apache-phoenix-4.10.0-HBase-1.2/bin/apache-phoenix-4.10.0-HBase-1.2-bin.tar.gz
I download the package and then the second line splits the input variable along the / delimiter I get apache-phoenix-4.10.0-HBase-1.2-bin.tar.gz. Now I want to assign the result of the second line to a variable. I change my script to a dir=$($1 | awk -F/ '{print $NF}') and add an echo $dir to the script to see the result. However I keep running into this error : line 2: http://apache.claz.org/phoenix/apache-phoenix-4.10.0-HBase-1.2/bin/apache-phoenix-4.10.0-HBase-1.2-bin.tar.gz: No such file or directory
I tried wrapping the command into `` but the problem persists. I am not cd-ing into any directory so I have no idea why this error keeps showing up.
I am trying to create a BASH script that will run a command for me. This is an example of one of the commands:
function systemStart {
./ORBMarkerDetection $1 $2 $3 | xargs -n3 java -jar ../../system/layers/out/artifacts/layers_jar/layers.jar
}
But when this is ran I am getting the error (referring to the above line):
./runActivities.sh: line 7: xargs: command not found
I am able to run this command in the terminal with success so I am not sure why this will not run within a BASH script?
I am calling the function like so:
systemStart $PATH/1.1/cupCupboard.png $PATH/1.1/kitchenDoor.png $PATH/1.1/tap.png
You are apparently using the variable name PATH for your own purposes, but you can't do that -- PATH is a reserved variable, and changing it will cause the shell to not find commands (not just xargs but basically any command).
In general, you should avoid using uppercase variable names; then you can be sure yours will never conflict with a built-in shell variable.
(You may need to put the fully-qualified path in your script?)
The command which can tell you the fully-qualified path for things.
robert#debian:~$ which xargs
/usr/bin/xargs
locate can also tell you the location of files
Lastly, then a brute-force full filesystem search using find:
robert#debian:~$ find / -name "xargs" 2> /dev/null
/usr/bin/xargs
I've a bash script that need to use some variables included in a separate file.
Normally, for including the entire file I would use source otherfile.sh in the main script.
In this case I need to use just a portion of this file. I can't use (include) the rest of the variables included in the rest of the file.
To filter the content of the config file (let's say just as example from the tag "# start" to "# end") I use awk, but I can't redirect the output to the soruce command.
Below my code:
awk ' /'"# start"'/ {flag=1;next} /'"# end"'/{flag=0} flag { print }' config.sh
Do you know any way to redirect the output of this command into source? Is there any other way to include the output in my bash script at run-time?
By the way, I wouldn't like to create temporaty files (it seems me too dirty...)..and I've tried something found on this site, but for some reasons it doesn't work. Below there's the solution I've found.
awk ' /'"# start"'/ {flag=1;next} /'"# end"'/{flag=0} flag { print }' config.sh | source /dev/stdin
Thank you,
Luca
source can read from a process substitution in bash:
source <( awk ' ... ' config.sh )
sed allows a simpler way to get the subset of lines:
sed -n '/#start/,/#end/p' config.sh
UPDATE: It appears that this may only work in bash 4 or later.
A correct way of doing it provided by a friend today. Here's the code:
source /dev/stdin <<EOF
$(awk ' /'"# 10.216.33.133 - start"'/ {flag=1;next} /'"# 10.216.33.133 - end"'/{flag=0} flag { print }' testbed.sh)
EOF
Working perfectly...thanks Andrea! :) (and of course everyone tried to answer)
I tried but I failed, I have file like:
06faefb38081b44e35b4ee846bfb84b61694a5c4.zip D:/code/3635/
0a386c77a3ae35033006efec07bfc37f5236212c.zip D:/code/3622/
0b425b29c3e51f29b9af03db46df0aa8f726705b.zip D:/code/3624/
0ecb477c82ec540c8eb0642da144a400bbf3f7c0.zip D:/code/3624/
...
My goal is to move file in first column to location in second column. I tried with while+awk but this did not worked. I would appreciate any help!
awk '{ system("mv "$1" "$2) }' filename
With awk, you can use the system function to build a move command and excute it. Obviously ensure that you are running the command in the directory with the files.
Let's assume, you file has name "data.txt", thus your code might look like this:
while read line; do mv $line; done < data.txt
What you need is just add a mv (space) to the beginning of each line. So you have 100+ ways to do that. If you love awk:
awk '$1="mv "$1' file
will create the mv command, to execute them, you can:
awk '$1="mv "$1' file |sh
I prefer this than the system() function call.