Related
I am working on a script to test new-to-me hard drives in the background (so I can close the terminal window) and log the outputs. My problem is in getting badblocks to print stdout to the log file so I can monitor its multi-day progress and create properly formatted update emails.
I have been able to print stdout to a log file with the following: (flags are r/w, % monitor, verbose)
sudo badblocks -b 4096 -wsv /dev/sdx 2>&1 | tee sdx.log
Normally the output would look like:
Testing with pattern 0xaa: 2.23% done, 7:00 elapsed. (0/0/0 errors)
No new-line character is used, the ^H control command backs up the cursor, and then the new updated status overwrites the previous status.
Unfortunately, the control character is not processed but saved as a character in the file, producing the above output followed by 43 copies of ^H, the new updated stats, 43 copies of ^H, etc.
Since the output is updated at least once per second, this produces a much larger file than necessary, and makes it difficult to retrieve the current status.
While working in terminal, the solution cat sdx.log && echo"" prints the expected/wanted results by parsing the control characters (and then inserting a carriage return so it is not immediately printed over by the next terminal line), but using cat sdx.log > some.file or cat sdx.log | mail both still include all of the extra characters (though in email they are interpreted as spaces). This solution (or ones like it which decode or remove the control character at the time of access still produce a huge, unnecessary output file.
I have worked my way through the following similar questions, but none have produced (at least that I can figure out) a solution which works in real time with the output to update the file, instead requiring that the saved log file be processed separately after the task has finished writing, or that the log file not be written until the process is done, both of which defeat the stated goal of monitoring progress.
Bash - process backspace control character when redirecting output to file
How to "apply" backspace characters within a text file (ideally in vim)
Thank you!
The main place I've run into this in real life is trying to process man pages. In the past, I've always used a simple script that post processes by stripping out the backspace appropriately. One could probably do this sort of thing in 80 character of perl, but here's an approach that handles backspace and cr/nl fairly well. I've not tested extensively, but it produces good output for simple cases. eg:
$ printf 'xxx\rabclx\bo\rhel\nworld\n' | ./a.out output
hello
world
$ cat output
hello
world
$ xxd output
00000000: 6865 6c6c 6f0a 776f 726c 640a hello.world.
If your output starts to have a lot of csi sequences, this approach just isn't worth the trouble. cat will produce nice human consumable output for those cases.
#include <assert.h>
#include <ctype.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
FILE * xfopen(const char *path, const char *mode);
off_t xftello(FILE *stream, const char *name);
void xfseeko(FILE *stream, off_t offset, int whence, const char *name);
int
main(int argc, char **argv)
{
const char *mode = "w";
char *name = strchr(argv[0], '/');
off_t last = 0, max = 0, curr = 0;
name = name ? name + 1 : argv[0];
if( argc > 1 && ! strcmp(argv[1], "-a")) {
argv += 1;
argc -= 1;
mode = "a";
}
if( argc > 1 && ! strcmp(argv[1], "-h")) {
printf("usage: %s [-a] [-h] file [ file ...]\n", name);
return EXIT_SUCCESS;
}
if( argc < 2 ) {
fprintf(stderr, "Missing output file. -h for usage\n");
return EXIT_FAILURE;
}
assert( argc > 1 );
argc -= 1;
argv += 1;
FILE *ofp[argc];
for( int i = 0; i < argc; i++ ) {
ofp[i] = xfopen(argv[i], mode);
}
int c;
while( ( c = fgetc(stdin) ) != EOF ) {
fputc(c, stdout);
for( int i = 0; i < argc; i++ ) {
if( c == '\b' ) {
xfseeko(ofp[i], -1, SEEK_CUR, argv[i]);
} else if( isprint(c) ) {
fputc(c, ofp[i]);
} else if( c == '\n' ) {
xfseeko(ofp[i], max, SEEK_SET, argv[i]);
fputc(c, ofp[i]);
last = curr + 1;
} else if( c == '\r' ) {
xfseeko(ofp[i], last, SEEK_SET, argv[i]);
}
}
curr = xftello(ofp[0], argv[0]);
if( curr > max ) {
max = curr;
}
}
return 0;
}
off_t
xftello(FILE *stream, const char *name)
{
off_t r = ftello(stream);
if( r == -1 ) {
perror(name);
exit(EXIT_FAILURE);
}
return r;
}
void
xfseeko(FILE *stream, off_t offset, int whence, const char *name)
{
if( fseeko(stream, offset, whence) ) {
perror(name);
exit(EXIT_FAILURE);
}
}
FILE *
xfopen(const char *path, const char *mode)
{
FILE *fp = fopen(path, mode);
if( fp == NULL ) {
perror(path);
exit(EXIT_FAILURE);
}
return fp;
}
You can delete the ^H
sudo badblocks -b 4096 -wsv /dev/sdx 2>&1 | tr -d '\b' | tee sdx.log
I have found col -b and colcrt usefull, but none worked perfect for me. These will apply control characters, not just drop them:
sudo badblocks -b 4096 -wsv /dev/sdx 2>&1 | col -b | tee sdx.log
I need to detect corrupted text file where there are invalid (non-ASCII) utf-8, Unicode or binary characters.
�>t�ï¿ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½w�ï¿ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿ï¿ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿ï¿½ï¿½ï¿½ï¿½ï¿½o��������ï¿ï¿½_��������������������o����������������������￿����ß����������ï¿ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½~�ï¿ï¿½ï¿½ï¿ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½}���������}w��׿��������������������������������������ï¿ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½~������������������������������������_������������������������������������������������������������������������������^����ï¿ï¿½s�����������������������������?�������������ï¿ï¿ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½w�������������ï¿ï¿ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿ï¿½}����������ï¿ï¿½ï¿½ï¿½ï¿½y����������������ï¿ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½o�������������������������}��
what I have tried:
iconv -f utf-8 -t utf-8 -c file.csv
this converts a file from utf-8 encoding to utf-8 encoding and -c is for skipping invalid utf-8 characters. However at the end those illegal characters still got printed. Are there any other solutions in bash on linux or other languages?
Assuming you have your locale set to UTF-8 (see locale output), this works well to recognize invalid UTF-8 sequences:
grep -axv '.*' file.txt
Explanation (from grep man page):
-a, --text: treats file as text, essential prevents grep to abort once finding an invalid byte sequence (not being utf8)
-v, --invert-match: inverts the output showing lines not matched
-x '.*' (--line-regexp): means to match a complete line consisting of any utf8 character.
Hence, there will be output, which is the lines containing the invalid not utf8 byte sequence containing lines (since inverted -v)
I would grep for non ASCII characters.
With GNU grep with pcre (due to -P, not available always. On FreeBSD you can use pcregrep in package pcre2) you can do:
grep -P "[\x80-\xFF]" file
Reference in How Do I grep For all non-ASCII Characters in UNIX. So, in fact, if you only want to check whether the file contains non ASCII characters, you can just say:
if grep -qP "[\x80-\xFF]" file ; then echo "file contains ascii"; fi
# ^
# silent grep
To remove these characters, you can use:
sed -i.bak 's/[\d128-\d255]//g' file
This will create a file.bak file as backup, whereas the original file will have its non ASCII characters removed. Reference in Remove non-ascii characters from csv.
Try this, in order to find non-ASCII characters from the shell.
Command:
$ perl -ne 'print "$. $_" if m/[\x80-\xFF]/' utf8.txt
Output:
2 Pour être ou ne pas être
4 Byť či nebyť
5 是或不
What you are looking at is by definition corrupted. Apparently, you are displaying the file as it is rendered in Latin-1; the three characters � represent the three byte values 0xEF 0xBF 0xBD. But those are the UTF-8 encoding of the Unicode REPLACEMENT CHARACTER U+FFFD which is the result of attempting to convert bytes from an unknown or undefined encoding into UTF-8, and which would properly be displayed as � (if you have a browser from this century, you should see something like a black diamond with a question mark in it; but this also depends on the font you are using etc).
So your question about "how to detect" this particular phenomenon is easy; the Unicode code point U+FFFD is a dead giveaway, and the only possible symptom from the process you are implying.
These are not "invalid Unicode" or "invalid UTF-8" in the sense that this is a valid UTF-8 sequence which encodes a valid Unicode code point; it's just that the semantics of this particular code point is "this is a replacement character for a character which could not be represented properly", i.e. invalid input.
As for how to prevent it in the first place, the answer is really simple, but also rather uninformative -- you need to identify when and how the incorrect encoding took place, and fix the process which produced this invalid output.
To just remove the U+FFFD characters, try something like
perl -CSD -pe 's/\x{FFFD}//g' file
but again, the proper solution is to not generate these erroneous outputs in the first place.
To actually answer the question about how to remove only invalid code points, try
iconv -f UTF-8 -t UTF-8//IGNORE broken-utf8.txt >fixed-utf8.txt
(You are not revealing the encoding of your example data. It is possible that it has an additional corruption. If what you are showing us is a copy/paste of the UTF-8 rendering of the data, it has been "double-encoded". In other words, somebody took -- already corrupted, as per the above -- UTF-8 text and told the computer to convert it from Latin-1 to UTF-8. Undoing that is easy; just convert it "back" to Latin-1. What you obtain should then be the original UTF-8 data before the superfluous incorrect conversion.
iconv -f utf-8 -t latin-1 mojibake-utf8.txt >fixed-utf8.txt
See also mojibake.)
This Perl program should remove all non-ASCII characters:
foreach $file (#ARGV) {
open(IN, $file);
open(OUT, "> super-temporary-utf8-replacement-file-which-should-never-be-used-EVER");
while (<IN>) {
s/[^[:ascii:]]//g;
print OUT "$_";
}
rename "super-temporary-utf8-replacement-file-which-should-never-be-used-EVER", $file;
}
What this does is it takes files as input on the command-line, like so: perl fixutf8.pl foo bar baz
Then, for each line, it replaces each instance of a non-ASCII character with nothing (deletion).
It then writes this modified line out to super-temporary-utf8-replacement-file-which-should-never-be-used-EVER (named so it dosen't modify any other files.)
Afterwards, it renames the temporary file to that of the original one.
This accepts ALL ASCII characters (including DEL, NUL, CR, etc.), in case you have some special use for them. If you want only printable characters, simply replace :ascii: with :print: in s///.
I hope this helps! Please let me know if this wasn't what you were looking for.
The following C program detects invalid utf8 characters.
It was tested and used on a linux system.
/*
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <stdio.h>
#include <stdlib.h>
void usage( void ) {
printf( "Usage: test_utf8 file ...\n" );
return;
}
int line_number = 1;
int char_number = 1;
char *file_name = NULL;
void inv_char( void ) {
printf( "%s: line : %d - char %d\n", file_name, line_number, char_number );
return;
}
int main( int argc, char *argv[]) {
FILE *out = NULL;
FILE *fh = NULL;
// printf( "argc: %d\n", argc );
if( argc < 2 ) {
usage();
exit( 1 );
}
// printf( "File: %s\n", argv[1] );
file_name = argv[1];
fh = fopen( file_name, "rb" );
if( ! fh ) {
printf( "Could not open file '%s'\n", file_name );
exit( 1 );
}
int utf8_type = 1;
int utf8_1 = 0;
int utf8_2 = 0;
int utf8_3 = 0;
int utf8_4 = 0;
int byte_count = 0;
int expected_byte_count = 0;
int cin = fgetc( fh );
while( ! feof( fh ) ) {
switch( utf8_type ) {
case 1:
if( (cin & 0x80) ) {
if( (cin & 0xe0) == 0xc0 ) {
utf8_1 = cin;
utf8_type = 2;
byte_count = 1;
expected_byte_count = 2;
break;
}
if( (cin & 0xf0) == 0xe0 ) {
utf8_1 = cin;
utf8_type = 2;
byte_count = 1;
expected_byte_count = 3;
break;
}
if( (cin & 0xf8) == 0xf0 ) {
utf8_1 = cin;
utf8_type = 2;
byte_count = 1;
expected_byte_count = 4;
break;
}
inv_char();
utf8_type = 1;
break;
}
break;
case 2:
case 3:
case 4:
// printf( "utf8_type - %d\n", utf8_type );
// printf( "%c - %02x\n", cin, cin );
if( (cin & 0xc0) == 0x80 ) {
if( utf8_type == expected_byte_count ) {
utf8_type = 1;
break;
}
byte_count = utf8_type;
utf8_type++;
if( utf8_type == 5 ) {
utf8_type = 1;
}
break;
}
inv_char();
utf8_type = 1;
break;
default:
inv_char();
utf8_type = 1;
break;
}
if( cin == '\n' ) {
line_number ++;
char_number = 0;
}
if( out != NULL ) {
fputc( cin, out );
}
// printf( "lno: %d\n", line_number );
cin = fgetc( fh );
char_number++;
}
fclose( fh );
return 0;
}
... I'm trying to detect if a file has corrupted characters. I'm also
interested in deleting them.
This is easy with ugrep and takes just one line:
ugrep -q -e "." -N "\p{Unicode}" file.csv && echo "file is corrupted"
To remove invalid Unicode characters:
ugrep "\p{Unicode}" --format="%o" file.csv
The first command matches any character with -e "." except valid Unicode with -N "\p{Unicode}" that is a "negative pattern" to skip.
The second command matches a Unicode character "\p{Unicode}" and writes it with --format="%o".
I am probably repeating what other have said already. But i think your invalid characters get still printed because they may be valid. The Universal Character Set is the attempt to reference the worldwide frequently used characters to be able to write robust software which is not relying on a special character-set.
So i think your problem may be one of the following both - in assumption that your overall target is to handle this (malicious) input from utf-files in general:
There are invalid utf8 characters (better called invalid byte sequences - for this i'd like to refer to the corresponding Wikipedia-Article).
There are absent equivalents in your current display-font which are substituted by a special symbol or shown as their binary ASCII-equivalent (f.e. - i therefore would like to refer to the following so-post: UTF-8 special characters don't show up).
So in my opinion you have two possible ways to handle this:
Transform the all characters from utf8 into something handleable - f.e. ASCII - this can be done f.e. with iconv -f utf-8 -t ascii -o file_in_ascii.txt file_in_utf8.txt. But be careful transferring from one the wider character-space (utf) into a smaller one might cause a data loss.
Handle utf(8) correctly - this is how the world is writing stuff. If you think you might have to rely on ASCII-chars because of any limitating post-processing step, stop and rethink. In most cases the post-processor already supports utf, it's probably better to find out how to utilize it. You're making your stuff future- and bullet-proof.
Handling utf might seem to be tricky, the following steps may help you to accomplish utf-readyness:
Be able to display utf correctly or ensure that your display-stack (os, terminal and so on) is able to display an adequate subset of unicode (which, of course, should meet your needs), this may prevent the need of a hex-editor in many cases. Unfortunately utf is too big, to come in one font, but a good point to start at is this so-post: https://stackoverflow.com/questions/586503/complete-monospaced-unicode-font
Be able to filter invalid byte sequences. And there are many ways to achieve that, this ul-post shows a plenty variety of these ways: Filtering invalid utf8 - i want to especially point out the 4th answer which suggests to use uconv which allows you to set a callback-handler for invalid sequences.
Read a bit more about unicode.
A very dirty solution in python 3
import sys
with open ("cur.txt","r",encoding="utf-8") as f:
for i in f:
for c in i:
if(ord(c)<128):
print(c,end="")
The output should be:
>two_o~}}w~_^s?w}yo}
Using Ubuntu 22.04, I get more correct answer by using:
grep -axv -P '.*' file.txt
The original answer without the -P, seems to give false positives for a lot of asian characters, like:
<lei:LegalName xml:lang="ko">피씨에이생명보험주식회사</lei:LegalName>
<lei:LegalName xml:lang="ko">린드먼 부품소재 전문투자조합 1</lei:LegalName>
<lei:LegalName xml:lang="ko">비엔피파리바 카디프손해보험 주식회사</lei:LegalName>
These characters do pass the scanning of the isutf8 utility.
Usually I use wc -l to count the lines of a file. However for a file with 5*10^7 lines, I get only 10^7 as an answer.
I've tried everything proposed here here:
How to count lines in a document?
But it takes to much time than wc -l.
Is there any other option?
Anyone serious about speed line counting can just create their own implementation:
#include <stdio.h>
#include <string.h>
#include <fcntl.h>
#define BUFFER_SIZE (1024 * 16)
char BUFFER[BUFFER_SIZE];
int main(int argc, char** argv) {
unsigned int lines = 0;
int fd, r;
if (argc > 1) {
char* file = argv[1];
if ((fd = open(file, O_RDONLY)) == -1) {
fprintf(stderr, "Unable to open file \"%s\".\n", file);
return 1;
}
} else {
fd = fileno(stdin);
}
while ((r = read(fd, BUFFER, BUFFER_SIZE)) > 0) {
char* p = BUFFER;
while ((p = memchr(p, '\n', (BUFFER + r) - p))) {
++p;
++lines;
}
}
close(fd);
if (r == -1) {
fprintf(stderr, "Read error.\n");
return 1;
}
printf("%d\n", lines);
return 0;
}
Usage
a < input
... | a
a file
Example:
# time ./wc temp.txt
10000000
real 0m0.115s
user 0m0.102s
sys 0m0.014s
# time wc -l temp.txt
10000000 temp.txt
real 0m0.120s
user 0m0.103s
sys 0m0.016s
* Code compiled with -O3 natively on a system with AVX and SSE4.2 using GCC 4.8.2.
You could try sed
sed -n '$=' file
The = says to print the line number, and the dollar says to only do it on the last line. The -n says not to do too much else.
Or here's a way in Perl, save this as wc.pl and do chmod +x wc.pl:
#!/usr/bin/perl
use strict;
use warnings;
my $filename = <#ARGV>;
my $lines = 0;
my $buffer;
open(FILE, $filename) or die "ERROR: Can not open file: $!";
while (sysread FILE, $buffer, 65536) {
$lines += ($buffer =~ tr/\n//);
}
close FILE;
print "$lines\n";
Run it like this:
wc.pl yourfile
Basically it reads your file in in chunks of 64kB at a time and then takes advantage of the fact that tr returns the number of substitutions it has made after asking it to delete all newlines.
Try with nl and see what happens...
You can get the line count using awk as well like below
awk 'END {print NR}' names.txt
(OR) Using while .. do .. done bash loop construct like
CNT=0; while read -r LINE; do (( CNT++ )); done < names.txt; echo $CNT
Depends on how you open the file, but probably reading it from STDIN instead would get the fix:
wc -l < file
I'm trying to get the lines with special characters which is not prefixed with \. Below are the special characters:
^$%.*+?!(){}[]|\
I need to check all the above special characters which is not prefixed with \ in 2nd column. I'm trying with awk to complete this, but no luck. I want the output as below.
input.txt
1,ap^ple
2,o$range
3,bu+tter
4,gr(ape
5,sm\(ok\e
6,ra\in
7,p+la\\y
8,wor\+k
output.txt
1,ap^ple
2,o$range
3,bu+tter
4,gr(ape
5,sm\(ok\e
6,ra\in
7,p+la\\y
7th row and 5 row are in output.txt because there is 2 special charcters(one is with backslash another without backslash)
"final" final edit: I wanted to allow "\x" whatever x is, but the OP seems to not want that, so I fixed it too.
After trying to find a "clever" regexp (which choked on "\\" or any impair number of "\", but apparently worked for the rest...)
I re-wrote it in awk to do it in a "state automata" way:
The idea:
If in "normal mode", we encounter a special char other than "\" ? : we print the line!
If in "normal mode", we encounter a "\" ? : we enter "escaped mode", and in that mode, ignore the next char
(but if we don't have a next char, we need to print that line too!)
the script:
awk -F"," '
{
IN_ESCAPED_MODE=0 ;
for (i=1 ; i<=length($2) ; i++)
{ char=substr($2,i,1)
if ( IN_ESCAPED_MODE == 0)
{ if ( index(".^$%*+?!(){}[]|",char) > 0 )
{ print $0 ; break ;
}
if ( index("\\" , char ) > 0 )
{ IN_ESCAPED_MODE=1 ; continue ;
}
}
if ( IN_ESCAPED_MODE == 1)
{ if ( index(".^$%*+?!(){}[]|\\",char) > 0 )
{ IN_ESCAPED_MODE=0 ; continue ;
}
else
{ IN_ESCAPED_MODE=0 ; print $0; break;
}
}
}
if (IN_ESCAPED_MODE == 1)
{
print $0 ; break ;
}
}
' input.txt > output.txt
With this change, you will have the same output as the OP, which prints a line when it contains "\e" for example... Which I find weird: to me "\e" is fine, we can "escape" anything?
With that input:
1,ap^ple
2,o$range
3,bu+tter
4,gr(ape
5,sm\(ok\e
6,ra\in
7,p+la\\y
8,wor\+k
10,\
11,\\
12,\\\
13,.
14,\.
15,..
16,^
17,\^
18,$
19,\$
20,%
21,\%
22,*
23,\*
24,+
25,\+
26,?
27,\?
28,!
29,\!
30,(
31,\(
32,)
33,\)
34,{
35,\{
36,}
37,\}
38,[
39,\[
40,]
41,\]
42,|
43,\|
it outputs:
1,ap^ple
2,o$range
3,bu+tter
4,gr(ape
5,sm\(ok\e
6,ra\in
7,p+la\\y
10,\
12,\\\
13,.
15,..
16,^
18,$
20,%
22,*
24,+
26,?
28,!
30,(
32,)
34,{
36,}
38,[
40,]
42,|
(so it appears to really work this time !)
If you prefer to allow any "\x" and NOT only if "x" is a SPECIAL char:
change the "middle lines":
if ( IN_ESCAPED_MODE == 1)
{ if ( index(".^$%*+?!(){}[]|\\",char) > 0 )
{ IN_ESCAPED_MODE=0 ; continue ;
}
else
{ IN_ESCAPED_MODE=0 ; print $0; break;
}
}
into:
if ( IN_ESCAPED_MODE == 1)
{ IN_ESCAPED_MODE=0 ; continue ;
}
for historical reason : the regexp (which worked in "most" cases but choked in some, for example if there was "\\") :
egrep '[^\][].^$%*+?!(){}[|]|[^\][\][^].^$%*+?!(){}[|\]' input.txt > output.txt
But that one will not display the line 12, for example...
A good read: http://www.regular-expressions.info/charclass.html .... and http://www.gnu.org/software/gawk/manual/html_node/Gory-Details.html (scary ...)
You can try the following:
awk '
{
line=$0
sub(/\\[\^$%.*+?!(){}\[\]|\\]/,"")
if(/[\^$%.*+?!(){}\[\]|\\]/)
print line
}' input.txt
sed '/[]\\^$%.*+?!(){}[|]/ {
h
s/\\[]\\^$%.*+?!(){}[|]/_/g
/[]\\^$%.*+?!(){}[|]/ {
x
p
}
}' YourFile
Depending of shell and sed could be interpreted (especialy the \) differently. Works on my AIX/KSH
So basically I have a file, for the sake of understanding how to write this, that looks like this.
Start:
First line of text
Second line of text
Bin:
Third line of text
Four line of text
What I need to accomplish is write a script that checks for these strings and will output any missing strings.
Based on my assumptions I would assume this would involve an awk or grep that would check if each string and a set of if statements that would say if this doesn't exist then output what string doesn't exist.
Any pointers on how to start this?
Here's what I've tried so far with that is little psuedocode-ish.
`
str1=$(awk '/Start:/' /some/file)
str2=$(awk '/First line of text/' /some/file)
str3=$(awk '/Second line of text/' /some/file)
if $str1 == '' then
print $str1 'does not exist'
elif $str2 == '' then
print $str2 'does not exist'
else $str3 == '' then
print $str3 'does not exist'
fi` `
Something like this should print the missing strings with AWK:
BEGIN {
a[i++]="Start:"
a[i++]="First line of text"
a[i++]="Second line of text"
}
// {
for (s in a) {
if (match($0,a[s]) ) { a[s]="" }
}
}
END {
for (s in a) {
if (a[s] != "") { print a[s] }
}
}