I am using bash and I was wondering if there is a nice way to do a find and replace on a file using bash. In my case I want to turn a placeholder variable into the orignal variable, working with any number of different variables. For example:
"$PLACEHOLDER_ABC" -> $ABC
"$PLACEHOLDER_123" -> $123
"$PLACEHOLDER_qwe" -> $qwe
I am not sure where to start, should I be using find, sed, a while loop, or all of the above?
Something like
sed 's/\$PLACEHOLDER_/\$/g' file
would remove PLACEHOLDER_ everywhere it is found after a dollar sign, effectively truncating the parameter expansion where it occurs.
You may want to have a look at "m4" the macro processor. The manpages are here. It is included in most Linux/Unix distros.
Here is a little example:
define(`hello', `Hello, World')
hello, welcome to m4!
If we run this file through m4 with the command
m4 sample.m4 > sample.txt
it produces the following output:
Hello, World, welcome to m4!
find filename -exec sed -i 's/\$PLACEHOLDER_/\$/g' {} \;
Related
I am trying to find a way to escape the dollar sign within the sed command in a bash script. I have found here tons of answers that say that you need to put four backslashes in a row in order to escape the sign. I have also tried the version with two backslashes, but for some reason, I can't get it to work. Can someone please tell me what I'm doing wrong?
Let's say that you have a file called aaa.txt located in /home/Documents. The file has just only one line of text, which says aaa. I am trying to replace it using this command (please don't tell me that I can reference it in a different way than with line numbers because this is just a reduced example of something else I am doing):
sed -i "1ccd /home/userr/$PARAMETER/$METHOD" "/home/userr/Documents/aaa.txt"
The output that I get is this:
cd /home/userr/\/\
Which is not what I want. I want to have exactly this output, with dollar signs in the string:
cd /home/userr/$PARAMETER/$METHOD
What is the proper form of the string passed to the sed command to achieve this?
You can escape the dollar sign with a backslash:
sed -i "1ccd /home/userr/\$PARAMETER/\$METHOD" "/home/userr/Documents/aaa.txt"
Documentation here: https://www.gnu.org/software/bash/manual/html_node/Escape-Character.html
You need to remove the double quotes so sed will not try and expand it. This sed should work without escaping needed.
$ sed -i '1ccd /home/userr/$PARAMETER/$METHOD' /home/userr/Documents/aaa.txt
I would like to escape a file path that is stored in a variable in a bash script.
I read several threads about escaping back ticks or but it seems not working as it should:
I have this variable:
The variables value is entered during the bash script execution as user parameter
CONFIG="/home/teams/blabla/blabla.yaml"
I would need to change this to: \/home\/teams\/blabla\/blabla.yaml
How can I do that with in the script via sed or so (not manually)?
With GNU bash and its Parameter Expansion:
echo "${CONFIG//\//\\/}"
Output:
\/home\/teams\/blabla\/blabla.yaml
Using the solution from this question, in your case it will look like this:
CONFIG=$(echo "/home/teams/blabla/blabla.yaml" | sed -e 's/[]\/$*.^[]/\\&/g')
echo "/home/teams/blabla/blabla.yaml" | sed 's/\//\\\//g'
\/home\/teams\/blabla\/blabla.yaml
explanation:
backslash is used to set the following letter/symbol as an regular expression or vice versa. double backslash is used when you need a backslash as letter.
Why does that need escaping? Is this an XY Problem?
If the issue is that you are trying to use that variable in a substitution regex, then the examples given should work, but you might benefit by removing some of the "leaning toothpick syndrom", which many tools can do just by using a different match delimiter. sed, for example:
$: sed "s,SOME_PLACEHOLDER_VALUE,$CONFIG," <<< SOME_PLACEHOLDER_VALUE
/home/teams/blabla/blabla.yaml
Be very careful about this, though. Commas are perfectly valid characters in a filename, as are almost anything but NULLs. Know your data.
I am quite new to shell scripting.
I am scraping a website and the scraped text contains a lot of repetitions. Usually they are the menus on a forum, for example. Mostly, I do this in Python, but I thought that sed command will save me reading and printing the input, loops etc. I want to delete thousands of repeated lines from the same single file. I do not want to copy it to another file, because I will end up with 100 new files. The following is a shadow script which I run from the bash shell.
#!/bin/sed -f
sed -i '/^how$/d' input_file.txt
sed -i '/^is test$/d' input_file.txt
sed -i '/^repeated text/d' input_file.txt
This is the content of the input file:
how to do this task
why it is not working
this is test
Stackoverflow is a very helpful community of programmers
that is test
this is text
repeated text is common
this is repeated text of the above line
Then I run in the shell the following command:
sed -f scriptFile input_file.txt
I get the following error
sed: scriptFile line 2: untermindated `s' command
How can I correct the script, and what is the correct syntax of the command I should use to get it work?
Any help is highly appreciated.
assuming you know what your script is doing, it's very easy to put them into a script. in your case, the script should be:
/^how$/d
/^is test$/d
/^repeated text/d
that's good enough.
to make the script alone to be executable is easy too:
#!/usr/bin/env sed -f
/^how$/d
/^is test$/d
/^repeated text/d
then
chmod +x your_sed_script
./your_sed_script <old >new
here is a very good and compact tutorial. you can learn a lot from it.
following is an example from the site, just in case the link is dead:
If you have a large number of sed commands, you can put them into a file and use
sed -f sedscript <old >new
where sedscript could look like this:
# sed comment - This script changes lower case vowels to upper case
s/a/A/g
s/e/E/g
s/i/I/g
s/o/O/g
s/u/U/g
Wouldn't it be easier to do it with egrep followed by a mv, for example
egrep -v 'pattern1|pattern2|pattern3|...' <input_file.txt >tmpfile.txt
mv tmpfile.txt input_file.txt
Each pattern would describe the lines being deleted, much like in sed. You would not end up with additional files, because the mv removes them.
If you have so many pattern, that you don't want to specify them directly on the command line, you can store them in a file use the -f option of egrep.
how do i capture a pattern in a filename and use that to do in linux?
example in a folder contains these files:
BBB137O19_rc.fa
BBB921N08_cleaned.fa
BBB002O19_cc.fa
I would like to capture the front part of the filename and use that to do things like renaming, run a program etc. Apparently, basename is greedy and works for everything before the extension.
thanks in advance
I tried this command but failed
for i in *.fa; base=$(basename $i _*.fa); comb="${base}_ec.txt"; mv ec.txt $comb; done
You can use BASH string manipulations:
s='BBB921N08_cleaned.fa'
echo "${s%%_*}"
BBB921N08
Also, sed can be used:
echo "$i"|sed 's/_.*//'
This removes _ and any character (.) occuring any number of times after it (*).
Sed with its regular expressions is especially useful, if you have more complicated patterns to process.
Could someone explain the meaning of below sed statement?
sed -i "s/PS1\='\\\\u\#[^]]*:/PS1\='\\\\u\#\\\\H:/g" test
First of all, note that PS1 is the bash prompt. See How to: Change / Setup bash custom prompt (PS1) for more references.
sed -i "s/PS1\='\\\\u\#[^]]*:/PS1\='\\\\u\#\\\\H:/g" test
It looks for the text PS1\='\\u\#[^]]*: and replaces it with PS1\='\\u\#\\H: in test file.
sed 's/hello/bye/g' file is the basic sed command that looks for hello and replaces it with bye all along the file (g means "global", so it does every time it finds the text).
While this sed expression shows the result on stdout, if you want the result to update the file, you add the -i option instead.
Then, note that I mentioned that the text looked for is PS1\='\\u\#[^]]*:, while in the sed expression we see PS1\='\\\\u\#[^]]*:. That's why any \ has to be escaped... and the \ character is used to do so.
Regarding the specific pattern looked for:
PS1\='\\u\#[^]]*:
means text like
PS1='\\u\#`
+
any string until the character `]` is found
+
:
So it will match texts like PS1\='\\u\#[hello how are you]:.
It replaces them with PS1\='\\u\#\\H:.