Replace Text with Formatted Text on Linux - linux

I'm looking for a way to replace the text between positions 10 and 17 in a file with a Linux command or script. For example, I'd like to replace the date text 20140101 with 01/01/2014.
I'm hoping this is something I can in a single command from the command line with maybe sed or awk?

Using sed, you can capture the first 9 chars in a capture group that would be placed as is. The remaining would be broken in 3 capture groups and re-arranged as you desire.
Something like:
sed -r 's#(.{9})(.{4})(.{2})(.{2})#\1\3/\4/\2#' file
If you are on a system that does not have GNU sed escape ( ) { } with \.

If all your strings are dates then the best is to use date command:
$ date -d 20140101 +%m/%d/%Y
01/01/2014
$ date -d 20140923 +"%m-%d-%Y %a %b"
09-23-2014 Tue Sep
It is especially great tool to translate number of seconds from 1970 (unix epoch) used in many log files:
$ date --date='#1411199063' +"%m-%d-%Y %H:%M:%S"
09-20-2014 07:44:23

Related

Get last 30 minutes from log file

I have a log file that contain logs as follows
1486307866.155 240207 68.146.231.80 TCP_MISS/200 790 CONNECT clients1.google.com:443 - DIRECT/172.217.6.238 -
1486307866.155 is the time in unix format with corresponds to 2017-02-05 07:17:46 (Format : Y-m-d H:i:s)
I need a unix command that give me the logs within last 30 minutes in the following format and discarding any details that i don't need.
2017-02-05 07:17:46|68.146.231.80|clients1.google.com:443
Using GNU date and GNU awk you can achieve what you want:
awk -v bt=$(date "+%s" -d "30 minutes ago") '$1 > bt {printf("%s|%s|%s\n", strftime("%F %T",$1), $3, $7)} ' yourfile
Explanation:
the date command date "+%s" -d "30 minutes ago" gets the timestamp from 30 minutes ago
the date command is replaced with its output via the command substitution feature $( ... )
the awk option -v passes that timestamp as variable named bt into the awk script
the script prints only those lines from the file having a value in column one ($1) larger than bt in your desired format

Search specific word and delete it from file in linux

I have a file (file1.txt) which contains below text:
mon
tue
tue_day
tuesday
wed
and I want to search for a word "tue" and delete it from this file.
I used
sed -i "/tue/d" file1.txt
but it deletes all the lines containing tue word i.e. line 2,3 and 4. I want to delete the only line 2 which conatins exact same text that i want to remove from file.
could you please suggest?
Just tell sed that you want lines that are exactly "tue". How? Prepending and appending ^ and $ to indicate beginning and end of line:
$ sed '/^tue$/d' file
mon
tue_day
tuesday
wed
To replace with something given in a variable, use double quotes like this:
var="tue"
sed -i "/^$var$/d" file

How to search a specific column in a file for a string in shell?

I am learning shell scripting. I wrote a script that indexes all the files and folders in the current working directory in the following tab separated format,
Path Filename File/Directory Last Modified date Size Textfile(Y/N)
for example, two lines in my file 'index.txt' are,
home/Desktop/C C d Thu Jan 16 01:23:57 PST 2014 4 KB N
home/Desktop/C/100novels.txt 100novels.txt f Thu Mar 14 06:04:06 PST 2013 0 KB Y
Now, i want to write another script to search for files from the previous file that takes in command line parameters, the file name. And also optional parameters as -dir (if to search only for directories), -date DATE (for last modified date), -t (for text files only) etc. How to go about this ?
I know the grep command that searches in a file for a string, but suppose i entered a file name, it would search the entire file for the file name instead of searching only the filename column. How do i do this? Is there any other command to achieve this ?
"i want to search in a speific column for a particular string... how to do that ?"
For example, to search for directories:
awk -F '\t' '$3 == "d"' filename
You can also search by regular expression or substring matching. Please post further questions as you learn.
Single quotes prevent variable substitution (and other forms of expansion) -- see the bash manual. So you have to pass the value by other means:
awk -F'\t' -v s="$search_file" '$2 == s' ./index.txt > final.txt
or, use double quotes, more fragile and harder to read IMO
awk -F'\t' "\$2 == \"$search_file\"" ./index.txt > final.txt
Also, you don't mix single and double quotes. Pick one or the other, depending on the functionality you need.
You can doit using the command awk. For example to print the first column of a file using awk:
awk -F ":" '{print $1}' /etc/passwd
(note that the -F flag is to choose a field separator for the columns, in your case you don't need it). Then you can use grep to filter in that column. Good luck!

Change date format in first column using awk/sed

I have a shell script which is automatically ran each morning which appends that days results to a text file. The file should have todays date on the first column followed by results separated by commas. I use the command date +%x to get the day in the required format (dd/mm/yy). However on one computer date +%x returns mm/dd/yyyy ( any idea why this is the case?). I then sort the data in the file in date order.
Here is a snippet of such a text file
29/11/12,9654.80,194.32,2.01,7.19,-7.89,7.65,7.57,3.98,9625.27,160.10,1.66,4.90,-4.79,6.83,4.84,3.54
03/12/12,5184.22,104.63,2.02,6.88,-6.49,7.87,6.67,4.10,5169.52,93.81,1.81,5.29,-5.45,7.87,5.37,4.10
04/12/12,5183.65,103.18,1.99,6.49,-6.80,8.40,6.66,4.38,5166.04,95.44,1.85,6.04,-6.49,8.40,6.28,4.38
11/07/2012,5183.65,102.15,1.97,6.78,-6.36,8.92,6.56,4.67,5169.48,96.67,1.87,5.56,-6.10,8.92,5.85,4.67
07/11/2012,5179.39,115.57,2.23,7.64,-6.61,8.83,7.09,4.62,5150.17,103.52,2.01,7.01,-6.08,8.16,6.51,4.26
11/26/2012,5182.66,103.30,1.99,7.07,-5.76,7.38,6.37,3.83,5162.81,95.47,1.85,6.34,-5.40,6.65,5.84,3.44
11/30/2012,5180.82,95.19,1.84,6.51,-5.40,7.91,5.92,4.12,5163.98,91.82,1.78,5.58,-5.07,7.05,5.31,3.65
Is it possible to change the date format for the latter four lines to the correct date format using awk or sed? I only wish to change the date format for those in the form mm/dd/yyyy to dd/mm/yy.
It looks like you're using two different flavors (versions) of date. To check which versions you've got, I think GNU date accepts the --version flag whereas other versions, like BSD/OSX will not accept this flag.
Since you may be using completely different systems, it's probably safest to avoid date completely and use perl to print the current date:
perl -MPOSIX -e 'print POSIX::strftime("%d/%m/%y", localtime) . "\n"'
If you are sure you have GNU awk on both machines, you could use it like this:
awk 'BEGIN { print strftime("%d/%m/%y") }'
To fix the file you've got, here's my take using GNU awk:
awk '{ print gensub(/^(..\/)(..\/)..(..,)/, "\\2\\1\\3", "g"); next }1' file
Or using sed:
sed 's/^\(..\/\)\(..\/\)..\(..,\)/\2\1\3/' file
Results:
29/11/12,9654.80,194.32,2.01,7.19,-7.89,7.65,7.57,3.98,9625.27,160.10,1.66,4.90,-4.79,6.83,4.84,3.54
03/12/12,5184.22,104.63,2.02,6.88,-6.49,7.87,6.67,4.10,5169.52,93.81,1.81,5.29,-5.45,7.87,5.37,4.10
04/12/12,5183.65,103.18,1.99,6.49,-6.80,8.40,6.66,4.38,5166.04,95.44,1.85,6.04,-6.49,8.40,6.28,4.38
07/11/12,5183.65,102.15,1.97,6.78,-6.36,8.92,6.56,4.67,5169.48,96.67,1.87,5.56,-6.10,8.92,5.85,4.67
11/07/12,5179.39,115.57,2.23,7.64,-6.61,8.83,7.09,4.62,5150.17,103.52,2.01,7.01,-6.08,8.16,6.51,4.26
26/11/12,5182.66,103.30,1.99,7.07,-5.76,7.38,6.37,3.83,5162.81,95.47,1.85,6.34,-5.40,6.65,5.84,3.44
30/11/12,5180.82,95.19,1.84,6.51,-5.40,7.91,5.92,4.12,5163.98,91.82,1.78,5.58,-5.07,7.05,5.31,3.65
This should work: sed -re 's/^([0-9][0-9])\/([0-9][0-9])\/[0-9][0-9]([0-9][0-9])(.*)$/\2\/\1\/\3\4/'
It can be made smaller but I made it so it would be more obvious what it does (4 groups, just switching month/day and removing first two chars of the year).
Tip: If you don't want to cat the file you could to the changes in place with sed -i. But be careful if you put a faulty expression in you might end up breaking your source file.
NOTE: This assumes that IF the year is specified with 4 digits, the month/day is reversed.
This below command will do it.
Note:No matter how many number of lines are present in the file.this will just change the last 4 lines.
tail -r your_file| awk -F, 'NR<5{split($1,a,"/");$1=a[2]"/"a[1]"/"a[3];print}1'|tail -r
Well i could figure out some way without using pipes and using a single awk statement and this solution does need a tail command:
awk -F, 'BEGIN{cmd="wc -l your_file";while (cmd|getline tmp);split(tmp,x)}x[1]-NR<=4{split($1,a,"/");$1=a[2]"/"a[1]"/"a[3];print}1' your_file
Another solution:
awk -F/ 'NR<4;NR>3{a=$1;$1=$2;$2=a; print $1"/"$2"/" substr($3,3,2) substr($3,5)}' file
Using awk:
$ awk -F/ 'NR>3{x=$1;$1=$2;$2=x}1' OFS="/" file
By using the / as the delimiter, all you need to do is swap the 1st and 2nd fields which is done here using a temporary variable.

Append date to filename in linux

I want add the date next to a filename ("somefile.txt"). For example: somefile_25-11-2009.txt or somefile_25Nov2009.txt or anything to that effect
Maybe a script will do or some command in the terminal window. I'm using Linux(Ubuntu).
The script or command should update the filename to a new date everytime you want to save the file into a specific folder but still keeping the previous files. So there would be files like this in the folder eventually: filename_18Oct2009.txt , filename_9Nov2009.txt , filename_23Nov2009.txt
Info/Summary
With bash scripting you can enclose commands in back ticks or parantheses. This works great for labling files, the following wil create a file name with the date appended to it.
Methods
Backticks -
$ echo myfilename-"`date +"%d-%m-%Y"`"
$(parantheses) - :
$ echo myfilename-$(date +"%d-%m-%Y")
Example Usage:
echo "Hello World" > "/tmp/hello-$(date +"%d-%m-%Y").txt"
(creates text file '/tmp/hello-28-09-2022.txt' with text inside of it)
Note, in Linux quotes are your friend, best practice to enclose the file name to prevent issues with spaces and such in variables.
There's two problems here.
1. Get the date as a string
This is pretty easy. Just use the date command with the + option. We can use backticks to capture the value in a variable.
$ DATE=`date +%d-%m-%y`
You can change the date format by using different % options as detailed on the date man page.
2. Split a file into name and extension.
This is a bit trickier. If we think they'll be only one . in the filename we can use cut with . as the delimiter.
$ NAME=`echo $FILE | cut -d. -f1
$ EXT=`echo $FILE | cut -d. -f2`
However, this won't work with multiple . in the file name. If we're using bash - which you probably are - we can use some bash magic that allows us to match patterns when we do variable expansion:
$ NAME=${FILE%.*}
$ EXT=${FILE#*.}
Putting them together we get:
$ FILE=somefile.txt
$ NAME=${FILE%.*}
$ EXT=${FILE#*.}
$ DATE=`date +%d-%m-%y`
$ NEWFILE=${NAME}_${DATE}.${EXT}
$ echo $NEWFILE
somefile_25-11-09.txt
And if we're less worried about readability we do all the work on one line (with a different date format):
$ FILE=somefile.txt
$ FILE=${FILE%.*}_`date +%d%b%y`.${FILE#*.}
$ echo $FILE
somefile_25Nov09.txt
cp somefile somefile_`date +%d%b%Y`
You can add date next to a filename invoking date command in subshell.
date command with required formatting options invoked the braces of $() or between the backticks (`…`) is executed in a subshell and the output is then placed in the original command.
The $(...) is more preferred since in can be nested. So you can use command substitution inside another substitution.
Solutions for requests in questions
$ echo somefile_$(date +%d-%m-%Y).txt
somefile_28-10-2021.txt
$ echo somefile_$(date +%d%b%Y).txt
somefile_28Oct2021.txt
The date command comes with many formatting options that allow you to customize the date output according to the requirement.
%D – Display date in the format mm/dd/yy (e.g. : 10/28/21)
%Y – Year (e.g. : 2021)
%m – Month (e.g. : 10)
%B – Month name in the full string format (e.g. : October)
%b – Month name in the shortened string format (e.g. : Oct)
%d – Day of month (e.g. : 28)
%j – Day of year (e.g. : 301)
%u – Day of the week (e.g. : 4)
%A – Weekday in full string format (e.g. : Thursday)
%a – Weekday in shortened format (e.g. : Thu)
I use this script in bash:
#!/bin/bash
now=$(date +"%b%d-%Y-%H%M%S")
FILE="$1"
name="${FILE%.*}"
ext="${FILE##*.}"
cp -v $FILE $name-$now.$ext
This script copies filename.ext to filename-date.ext, there is another that moves filename.ext to filename-date.ext, you can download them from here.
Hope you find them useful!!
I use it in raspberry pi, and the first answer doesn't work for me, maybe because I typed wrong or something? I don't know. So I combined the above answers and came up with this:
now=$(date +'%Y-%m-%d')
geany "OptionalName-${now}.txt"
That if you want to use geany or anything else
a bit more convoluted solution that fully matches your spec
echo `expr $FILENAME : '\(.*\)\.[^.]*'`_`date +%d-%m-%y`.`expr $FILENAME : '.*\.\([^.]*\)'`
where first 'expr' extracts file name without extension, second 'expr' extracts extension

Resources