I have a comma delimited file (CSV file) test.csv as shown below.
FHEAD,1,2,3,,,,,,
FDEP,2,3,,,,,,,,
FCLS,3,,,4-5,,,,,,,
FDETL,4,5,6,7,8,
FTAIL,5,67,,,,,,
I wanted to remove the empty columns only from 2nd and 3rd row of the file i.e. were ever the records starts with FDEP and FCLS only in those rows I wanted to remove the empty columns (,,).
after removing the empty columns the same file test.csv should look like
FHEAD,1,2,3,,,,,,
FDEP,2,3
FCLS,3,4-5
FDETL,4,5,6,7,8,
FTAIL,5,67,,,,,,
How can I do this in Unix???
Here's one way to do it, using sed:
sed '/^F\(DEP\|CLS\),/ { s/,\{2,\}/,/g; s/,$// }'
We use a range of /^F\(DEP\|CLS\),/, i.e. the following command will only process lines matching ^F\(DEP\|CLS\),. This regex matches beginning-of-string, followed by F, followed by either DEP or CLS, followed by ,. In other words, we look for lines starting with FDEP, or FCLS,.
Having found such a line, we first substitute (s command) all runs (g flag, match as many times as possible) of 2 or more (\{2,\}) commas (,) in a row by a single ,. This squeezes ,,, down to a single ,.
Second, we substitute , at end-of-string by nothing. This gets rid of any trailing comma.
Related
I have a file which contains two columns (names.csv), values are separated by comma
,
a123456789-anything,a123456789-anything
b123456789-anything,b123456789-anything
c123456789-anything,c123456789-anything
d123456789-anything,d123456789-anything
e123456789-anything,e123456789-anything
e123456777-anything,e123456999-anything
These columns have values with 10 digits, which are unique identifiers, and some extra junk in the values (-anything).
I want to see if the columns have the prefix matched!
To verify the values on first and second column I use:
cat /home/names.csv | parallel --colsep ',' echo column 1 = {1} column 2 = {2}
Which print the values. Because the values are HEX digits, it is cumbersome to verify one by one by only reading. Is there any way to see if the 10 digits of each column pair are exact matches? They might contain special characters!
Expected output (example, but anything that says the columns are matched or not can work):
Matches (including first line):
,
a123456789-anything,a123456789-anything
b123456789-anything,b123456789-anything
c123456789-anything,c123456789-anything
d123456789-anything,d123456789-anything
e123456789-anything,e123456789-anything
Non-matches
e123456777-anything,e123456999-anything
Here's one way using awk. It prints every line where the first 10 characters of the first two fields match.
% cat /tmp/names.csv
,
a123456789-anything,a123456789-anything
b123456789-anything,b123456789-anything
c123456789-anything,c123456789-anything
d123456789-anything,d123456789-anything
e123456789-anything,e123456789-anything
e123456777-anything,e123456999-anything
% awk -F, 'substr($1,1,10)==substr($2,1,10)' /tmp/names.csv
,
a123456789-anything,a123456789-anything
b123456789-anything,b123456789-anything
c123456789-anything,c123456789-anything
d123456789-anything,d123456789-anything
e123456789-anything,e123456789-anything
For instance let say I have a text file:
worker1, 0001, company1
worker2, 0002, company2
worker3, 0003, company3
How would I use sed to take the first 2 characters of the first column so "wo" and remove the rest of the text and attach it to the second column so the output would look like this:
wo0001,company1
wo0002,company2
wo0003,company3
$ sed -E 's/^(..)[^,]*, ([^,]*,) /\1\2/' file
wo0001,company1
wo0002,company2
wo0003,company3
s/ begin substitution
^(..) match the first two characters at the beginning of the line, captured in a group
[^,]* match any amount of non-comma characters of the first column
, match a comma and a space character
([^,]*,) match the second field and comma captured in a group (any amount of non-comma characters followed by a comma)
match the next space character
/\1\2/ replace with the first and second capturing group
I have a huge data set of entries like these:
(21, 2, '23.5R25 ETADT', 'description, with a comma'),
(22, 1, '26.5R25 ETADT', 'Description without a comma'),
(23, 5, '20.5R20.5', 'Another description with ; semicolumn'),
I'm trying to replace every comma in the list with a tab. Excluding the commas within the single quotes. Also excluding the ending commas.
So the examples entries should become:
(21[TAB]2[TAB]'23.5R25 ETADT'[TAB]'description, with a comma'),
(22[TAB]1[TAB]'26.5R25 ETADT'[TAB]'Description without a comma'),
(23[TAB]5[TAB]'20.5R20.5'[TAB]'Another description with ; semicolumn'),
I've got like 6000 rows of data like this. The tabs allow me to tell Excel to import the elements of these entries into different columns.
The Regex I've tried was this: [ ]*,[ ]*
But this Regex selects all the commas, even the ones within the single quotes.
It looks as though each of your lines has 4 elements within parenthesis. And it looks like only the last 2 elements use single quotes. If those assumptions can be made, I've tested the following in Notepad++:
"Find what :" ^\(([^,]*),\s*([^,]*),\s*'([^']*)'\s*,\s*
"Replace with :" \(\1\t\2\t'\3'\t
EDIT:
The search regex is dependent upon the 4 column model with only the last two elements having single quotes. Visually this is how it works:
^\(: Finds an opening parenthesis
([^,]*): Captures non-comma characters which will be all of element 1
,\s*: Matches a comma and any trailing spaces
([^,]*): Captures non-comma characters which will be all of element 2
,\s*: Matches a comma and any trailing spaces
'([^']*)': Captures the string in single quotes which will be all of element 3
\s*,\s*: Matches a comma and all surrounding spaces
Ignore the rest of the string, there are no more commas to be replaced we just want to replace parts of the line we just read in
I want to check for a pattern (only if the pattern starts with) in second column in a CSV file and if that pattern exists then replace something else in same line.
I wrote the following sed command for following csv to change the I to N if the pattern 676 exists in second column. But it checks 676 in the 7th and 9th column also since the ,676 exists. Ideally, I want only the second line to be checked for if the prefix 676 exists. All I want is to check 676 prefixed in second column (pattern not in the middle or end of the second value Ex- 46769777) and then do the change on ,I, to ,N,.
sed -i '/,676/ {; s/,I,/,N,/;}' temp.csc
6768880,55999777,S,I,TTTT,I,67677,yy
6768880,676999777,S,I,TTTT,I,67677,yy
6768880,46769777,S,I,TTTT,I,67677,yy
Expected result required
6768880,55999777,S,I,TTTT,I,67677,yy
6768880,676999777,S,N,TTTT,N,67677,yy
6768880,40999777,S,I,TTTT,I,67677,yy
If you are not bound by sed, awk might be a better option for you. Give this a try :
awk -F"," '{match($2,/^676/)&&gsub(",I",",N")}{print}' temp.csc
match syntax does the matching of second column to numbers that starts with (^) 676. gsub replaces I with N.
Result:
6768880,55999777,S,I,TTTT,I,67677,yy
6768880,676999777,S,N,TTTT,N,67677,yy
6768880,46769777,S,I,TTTT,I,67677,yy
This requires that 676 appear at the beginning of the second column before any changes are made:
$ sed '/^[^,]*,676/ s/,I,/,N,/g' file
6768880,55999777,S,I,TTTT,I,67677,yy
6768880,676999777,S,N,TTTT,N,67677,yy
6768880,46769777,S,I,TTTT,I,67677,yy
Notes:
The regex /^[^,]*,676/ requires that 676 appear after the first appearance of a comma on the line. In more detail:
^ matches the beginning of the line
[^,]* matches the first column
,676 matches the first comma followed by 676
In your desired output, ,I, was replaced with ,N, every time it appeared on the line. To accomplish this, g (meaning global) was added to the substitute command.
I have a 3 column file. I would like to append a third column which is just one word repeated many times. I tried the following
paste file.tsv <(echo 'new_text') > new_file.tsv
But the text 'new_text' only appears on the first line, not every line.
How can I get 'new_text' to appear on every line.
Thanks
sed '1,$ s/$/;ABC/' infile > outfile
This replaces the line end ("$") with ";ABC".