I would like to extract a string as below:
setenv VARIABLE /opt/Application/File/Version22
I want to display only
File/Version22
on screen. It means /opt/Application/ is always the prefix. I want to display the rest of the string. How do I do it in csh?
Regards
Now that you've updated the question (thank you for that), it's clear that you always want to remove the /opt/Application prefix.
The most straightforward way to do that, which will work in any shell, is:
echo $VARIABLE | sed 's|^/opt/Application/||'
(It's usual to use / as a delimiter for replacements like this, but you can use any punctuation character; I'm using | to avoid conflicting with the / characters in the pattern.)
A more tcsh-specific, and possibly more efficient, way to do it is:
echo $VARIABLE:s|/opt/Application/||
It's likely to be more efficient because it's done within the shell and avoids invoking the external sed command. On the other hand, the overhead of executing sed is unlikely to be significant.
Note carefully that the :s syntax is not supported in the original csh. It is supported in tcsh and in some newer versions of csh. If you want to do this portably, just use sed.
C-Shell has the built-in string modifiers that can do this with very little code:
echo $VARIABLE:h:t/$VARIABLE:t
string modifiers:
:h = remove the last directory (aka head)
:t = remove every directory except the last one (aka tail)
I assume that you want to get the 2 last names in a filename:
setenv VARIABLE /opt/Application/File/Version22
set lastfile=`basename $VARIABLE`
set prevpath=`dirname $VARIABLE`
set previousfile=`basename $prevpath`
set file=$previousfile/$lastfile
echo $file
Related
I want to replace A with T and T with A
sed -e 's/T/A/g;s/A/T/g
as an example above line changes A:T to T:T
I am hoping to get T:A.
How do I do this?
If you want to change single characters, it is simply:
sed 'y/TA/AT/'
If you want to change longer (non-overlapping) strings, you need a temporary value that you know is never used. Conveniently, newline can never appear. So:
sed '
s/T/\n/g
s/A/T/g
s/\n/A/g
'
I'm not a SED expert - so not sure if that can be done as a single command. Just wondering if you've thought about doing that swap like you would in a programming language that would need a temporary variable to do the switch?
Maybe like change the A to a value you know you don't have in the string like Y for example. Then change the T to A and then change Y to T. Would something like that work?
Edit: I did a quick search just out of curiosity. Found this: https://unix.stackexchange.com/questions/528994/swapping-words-with-sed
In case that helps, but with regex stuff, the result is highly dependent on how structured and unique your inputs are. Not sure how to just swap two arbitrary sub-strings or characters throughout an entire string if there's no particular structure that tells you when you're about to get that sub-string or character like the answer above looking for the parenthesis.
Use this Perl one-liner for case-sensitive replacement:
echo 'TATAtata' | perl -pe 'tr{AT}{TA}'
ATATtata
Or this one-liner for case-insensitive replacement:
echo 'TATAtata' | perl -pe 'tr{ATat}{TAta}'
ATATatat
The Perl one-liner uses these command line flags:
-e : Tells Perl to look for code in-line, instead of in a file.
-p : Loop over the input one line at a time, assigning it to $_ by default. Add print $_ after each loop iteration.
SEE ALSO:
perldoc perlrun: how to execute the Perl interpreter: command line switches
perldoc tr
perldoc tr/SEARCHLIST/REPLACEMENTLIST/cdsr
I have a file1.txt with the below contents:
time="2016-04-25T17:43:11Z" level=info msg="SHA1 Fingerprint=9F:AD:D4:FD:22:24:20:A2:1E:0C:7F:D0:19:C5:80:42:66:56:AC:6F"
I want the file to look as below:
9F:AD:D4:FD:22:24:20:A2:1E:0C:7F:D0:19:C5:80:42:66:56:AC:6F
Actually, I need to pass the command as a string. That is why the bash command needs to be encapsulated in a string with double quotes. However, when i include
" grep -Po '(?<=Fingerprint=)[^"]*' "
I don't get the desired output. It seems that i need to escape the double quotes correctly.
To answer the literal question, you can use a backslash to escape literal double quotes in your command. However, for the reasons given in BashFAQ #50, this is exceedingly poor practice:
# Avoid this absent a very good reason
grep_cmd_str="grep -Po '(?<=Fingerprint=)[^\"]*'"
eval "$grep_cmd_str" <file1.txt # eval is necessary, with all the issues that implies
Better practice when you need to store a simple command (with no redirections or other shell constructs)[1] in a variable is to use an array[2], not a scalar variable, to hold its arguments:
# Use this principally if you need to dynamically build up an argument list
grep_args=( grep -Po '(?<=Fingerprint=)[^"]*' )
"${grep_args[#]}" <file1.txt
If you don't have any constraints that require you to use either of the above, consider a function (which does allow redirections and shell constructs so long as they're hardcoded):
# Use this whenever possible, in preference to the above
grep_fp() { grep -Po '(?<=Fingerprint=)[^"]*' "$#"; }
grep_fp <file1.txt
[1] - Not evaluating shell constructs, in this context, is a security feature: it protects you against malicious filenames or similar content in values which have been substituted into the value you're using as a command.
[2] - Note that arrays are not available in POSIX sh, which your question is also tagged for. That said, similar functionality is available by overriding "$#" (presumably within a limited scope, such as a function, such that its original value is preserved).
my goal is to sed the 100th line and convert it to a string, then separate the data of the sentence to word
#!/bin/bash
fid=log.txt;
sentence=`expr sed -n '100p' ${fid}`;
for word in $sentence
do
echo $word
done
but apparently this has failed.
expr: syntax error
would somebody please let me know what have I done wrong? previously for number it worked.
The expr does not seem to serve a useful purpose here, and if it did, a sed command would certainly not be a valid or useful thing to pass to it, under most circumstances. You should probably just take it out.
However, the following loop is also problematic. Unquoted variables in shell script are very frequently an error. In this case, you can't quote the thing you pass to the for loop (that would cause the loop to only run once, with the loop variable set to the quoted string) but you also cannot prevent the shell from performing wildcard expansion on the unquoted string. So if the string happened to contain *, the shell will expand that to a list of files in the current directory, for example.
Fortunately, this can all be done in an only slightly more complicated sed script.
sed '100!d;s/[ \t]\+/\n/g;q' "$fid"
That is, if the line number is not 100, delete this line and start over with the next line. Otherwise, we are at line 100; replace runs of whitespace with newlines, (print) and quit.
(The backslash escape codes \t and \n are not universally portable; and \+ for repetition is also an optional extension. I believe there are also sed variants which dislike semicolon as a command separator. Consult your sed manual page, experiment, and if everything else fails, maybe switch to Awk or Perl. Just in case, here is a version which works even on Mac OSX:
sed '100!d
s/[ ][ ]*/\
/g;q' log.txt
The stuff inside the square brackets are a space and a literal tab; in Bash, with default keybindings, type ctrl-V, tab to produce a literal tab.)
Incidentally, this also gets rid of the variable capture antipattern. There are good reasons to capture output to a variable, but if it can be avoided, you often end up with a simpler, more robust and efficient, as well as more idiomatic and elegant script. (I see no reason to put the log file name in a variable, either, in this isolated case; but in a larger script, it might make sense.)
I don't think you need expr command in this case.
expr is used to do calculations. Something like:
expr 1 + 1
Just this one is fine:
sentence=`sed -n '100p' ${fid}`;
#!/bin/bash
fid=log.txt;
sentence=$(sed -n '100p' ${fid});
for word in $sentence
do
echo $word
done
put a dollar sign and parenthesis solve the problem
I'm having a heck of a time removing characters in Bash. I have a string that's formatted like temp=53.0'C. I want to remove everything thats not 53.0.
I'm normally a Python programmer, and the way I'd do this in Python would be to split the string into an array of characters, and remove the unnecessary elements, before putting the array back onto string form.
But I can't figure out how to do that in Bash.
How do I remove the desired characters?
You can use Bash parameter substitution like this:
a="temp=53.0'C"
a=${a/*=/} # Remove everything up to and including = sign
a=${a/\'*/} # Remove single quote and everything after it
echo $a
53.0
Further examples are available here.
You could use sed with a regex which corresponds to the format of the string you want to be returned:
$ var="temp=53.0'C"
$ echo "$var" | sed -r 's/.*=([0-9][0-9]\.[0-9]).*/\1/g'
53.0
What exactly are the "rules" around what your original string looks like, and what the section to output looks like?
Same thing with BASH_REMATCH
> [[ $tmp =~ [0-9]+\.[0-9]+ ]] && echo ${BASH_REMATCH[0]}
53.0
Also agree with Josh but would improve the pattern match to consider the full range of floating point numbers.
.*=[ ]*([0-9]*\.[0-9]+)[cC].*
If you do not understand the pattern above, take the time to find out. Learning pattern matching will be one of the most useful things you ever do.
Test your pattern with something like http://www.freeformatter.com/regex-tester.html and then tailor for the platform you are using (e.g. Unix will probably need the brackets escaped with a backslash)
What idiom should one use in Bash scripts (no Perl, Python, and such please) to build up a command line for another program out of the script's arguments while handling filenames correctly?
By correctly, I mean handling filenames with spaces or odd characters without inadvertently causing the other program to handle them as separate arguments (or, in the case of < or > — which are, after all, valid if unfortunate filename characters if properly escaped — doing something even worse).
Here's a made-up example of what I mean, in a form that doesn't handle filenames correctly: Let's assume this script (foo) builds up a command line for a command (bar, assumed to be in the path) by taking all of foo's input arguments and moving anything that looks like a flag to the front, and then invoking bar:
#!/bin/bash
# This is clearly wrong
FILES=
FLAGS=
for ARG in "$#"; do
echo "foo: Handling $ARG"
if [ x${ARG:0:1} = "x-" ]; then
# Looks like a flag, add it to the flags string
FLAGS="$FLAGS $ARG"
else
# Looks like a file, add it to the files string
FILES="$FILES $ARG"
fi
done
# Call bar with the flags and files (we don't care that they'll
# have an extra space or two)
CMD="bar $FLAGS $FILES"
echo "Issuing: $CMD"
$CMD
(Note that this just an example; there are lots of other times one needs to do this and that to a bunch of args and then pass them onto other programs.)
In a naive scenario with simple filenames, that works great. But if we assume a directory containing the files
one
two
three and a half
four < five
then of course the command foo * fails miserably in its task:
foo: Handling four < five
foo: Handling one
foo: Handling three and a half
foo: Handling two
Issuing: bar four < five one three and a half two
If we actually allow foo to issue that command, well, the results won't be what we're expecting.
Previously I've tried to handle this through the simple expedient of ensuring that there are quotes around each filename, but I've (very) quickly learned that that is not the correct approach. :-)
So what is? Constraints:
I want to keep the idiom as simple as possible (not least so I can remember it).
I'm looking for a general-purpose idiom, hence my making up the bar program and the contrived example above instead of using a real scenario where people might easily (and reasonably) go down the route of trying to use features in the target program.
I want to stick to Bash script, I don't want to call out to Perl, Python, etc.
I'm fine with relying on (other) standard *nix utilities, like xargs, sed, or tr provided we don't get too obtuse (see #1 above). (Apologies to Perl, Python, etc. programmers who think #3 and #4 combine to draw an arbitrary distinction.)
If it matters, the target program might also be a Bash script, or might not. I wouldn't expect it to matter...
I don't just want to handle spaces, I want to handle weird characters correctly as well.
I'm not bothered if it doesn't handle filenames with embedded nul characters (literally character code 0). If someone's managed to create one in their filesystem, I'm not worried about handling it, they've tried really hard to mess things up.
Thanks in advance, folks.
Edit: Ignacio Vazquez-Abrams pointed me to Bash FAQ entry #50, which after some reading and experimentation seems to indicate that one way is to use Bash arrays:
#!/bin/bash
# This appears to work, using Bash arrays
# Start with blank arrays
FILES=()
FLAGS=()
for ARG in "$#"; do
echo "foo: Handling $ARG"
if [ x${ARG:0:1} = "x-" ]; then
# Looks like a flag, add it to the flags array
FLAGS+=("$ARG")
else
# Looks like a file, add it to the files array
FILES+=("$ARG")
fi
done
# Call bar with the flags and files
echo "Issuing (but properly delimited, not exactly as this appears): bar ${FLAGS[#]} ${FILES[#]}"
bar "${FLAGS[#]}" "${FILES[#]}"
Is that correct and reasonable? Or am I relying on something environmental above that will bite me later. It seems to work and it ticks all the other boxes for me (simple, easy to remember, etc.). It does appear to rely on a relatively recent Bash feature (FAQ entry #50 mentions v3.1, but I wasn't sure whether that was arrays in general of some of the syntax they were using with it), but I think it's likely I'll only be dealing with versions that have it.
(If the above is correct and you want to un-delete your answer, Ignacio, I'll accept it provided I haven't accepted any others yet, although I stand by my statement about link-only answers.)
Why do you want to "build up" a command? Add the files and flags to arrays using proper
quoting and issue the command directly using the quoted arrays as arguments.
Selected lines from your script (omitting unchanged ones):
if [[ ${ARG:0:1} == - ]]; then # using a Bash idiom
FLAGS+=("$ARG") # add an element to an array
FILES+=("$ARG")
echo "Issuing: bar \"${FLAGS[#]}\" \"${FILES[#]}\""
bar "${FLAGS[#]}" "${FILES[#]}"
For a quick demo of using arrays in this manner:
$ a=(aaa 'bbb ccc' ddd); for arg in "${a[#]}"; do echo "..${arg}.."; done
Output:
..aaa..
..bbb ccc..
..ddd..
Please see BashFAQ/050 regarding putting commands in variables. The reason that your script doesn't work is because there's no way to quote the arguments within a quoted string. If you were to put quotes there, they would be considered part of the string itself instead of as delimiters. With the arguments left unquoted, word splitting is done and arguments that include spaces are seen as more than one argument. Arguments with "<", ">" or "|" are not a problem in any case since redirection and piping is performed before variable expansion so they are seen as characters in a string.
By putting the arguments (filenames) in an array, spaces, newlines, etc., are preserved. By quoting the array variable when it's passed as an argument, they are preserved on the way to the consuming program.
Some additional notes:
Use lowercase (or mixed case) variable names to reduce the chance that they will collide with the shell's builtin variables.
If you use single square brackets for conditionals in any modern shell, the archaic "x" idiom is no longer necessary if you quote the variables (see my answer here). However, in Bash, use double brackets. They provide additional features (see my answer here).
Use getopts as Let_Me_Be suggested. Your script, though I know it's only an example, will not be able to handle switches that take arguments.
This for ARG in "$#" can be shortened to this for ARG (but I prefer the readability of the more explicit version).
See BashFAQ #50 (and also maybe #35 on option parsing). For the scenario you describe, where you're building a command dynamically, the best option is to use arrays rather than simple strings, as they won't lose track of where the word boundaries are. The general rules are: to create an array, instead of VAR="foo bar baz", use VAR=("foo" "bar" "baz"); to use the array, instead of $VAR, use "${VAR[#]}". Here's a working version of your example script using this method:
#!/bin/bash
# This is clearly wrong
FILES=()
FLAGS=()
for ARG in "$#"; do
echo "foo: Handling $ARG"
if [ x${ARG:0:1} = "x-" ]; then
# Looks like a flag, add it to the flags array
FLAGS=("${FLAGS[#]}" "$ARG") # FLAGS+=("$ARG") would also work in bash 3.1+, as Dennis pointed out
else
# Looks like a file, add it to the files string
FILES=("${FILES[#]}" "$ARG")
fi
done
# Call bar with the flags and files (we don't care that they'll
# have an extra space or two)
CMD=("bar" "${FLAGS[#]}" "${FILES[#]}")
echo "Issuing: ${CMD[*]}"
"${CMD[#]}"
Note that in the echo command I used "${VAR[*]}" instead of the [#] form because there's no need/point to preserving word breaks here. If you wanted to print/record the command in unambiguous form, this would be a lot messier.
Also, this gives you no way to build up redirections or other special shell options in the built command -- if you add >outfile to the FILES array, it'll be treated as just another command argument, not a shell redirection. If you need to programmatically build these, be prepared for headaches.
getopts should be able to handle spaces in arguments correctly ("file name.txt"). Weird characters should work as well, assuming they are correctly escaped (ls -b).