I need to create a batch file [and unfortunately it has to be a batch file :( ] to pull information from a log, put that data into a new file, and then from the new file pull error text into another file. So I should end up with 3 files. The original log, the pulled thread, and the final log.
The thread number and error text is generally random and requires user input so that both the correct thread as well as the correct error are pulled.
Any and all help will be quite appreciated.
My log file might look like this
01/01/01 11:59:58: thread 1234: Start
01/01/01 11:59:58: thread 5678: Start
01/01/01 11:59:58: thread 5678: *: Other Unnecessary information
01/01/01 11:59:58: thread 5678: *: error error
01/01/01 11:59:58: thread 5678: End
01/01/01 11:59:59: thread 1234: *: Other Unnecessary information
01/01/01 11:59:59: thread 1234: *: Other Unnecessary information
01/01/01 11:59:59: thread 1234: End
01/01/01 12:00:00: thread 1234: Start
01/01/01 12:00:00: thread 1234: *: Other userful information
01/01/01 12:00:00: thread 5678: Start
01/01/01 12:00:00: thread 5678: *: Other Unnecessary information
01/01/01 12:00:00: thread 5678: *: Other Unnecessary information
01/01/01 12:00:00: thread 5678: End
01/01/01 12:00:00: thread 1234: *: Other userful information
01/01/01 12:00:00: thread 1234: ERROR ERROR ERROR
01/01/01 12:00:00: thread 1234: *: Other userful information
01/01/01 12:00:00: thread 1234: *: Other userful information
01/01/01 12:00:00: thread 1234: End
01/01/01 12:00:01: thread 1234: Start
01/01/01 12:00:01: thread 1234: *: Other Unnecessary information
01/01/01 12:00:01: thread 1234: *: Other Unnecessary information
01/01/01 12:00:01: thread 1234: End
In this instance I am looking for thread 1234. Once that's in it's own file I want to pull the error text. Along with the lines from Start to End. So my final file would look like whats below.
01/01/01 12:00:00: thread 1234: Start
01/01/01 12:00:00: thread 1234: *: Other userful information
01/01/01 12:00:00: thread 1234: *: Other userful information
01/01/01 12:00:00: thread 1234: ERROR ERROR ERROR
01/01/01 12:00:00: thread 1234: *: Other userful information
01/01/01 12:00:00: thread 1234: *: Other userful information
01/01/01 12:00:00: thread 1234: End
The number of lines above and below the error text can change quite significantly. My preliminary batch file is below.
#echo off
cls
:start
cls
:: Context Menu
echo ____________________________________________________________
echo ^| Please choose an option below. ^|
echo ^|Search logs for specific error? [B] ^|
echo ^|Exit [X] ^|
echo ^|___________________________________________________________^|
:: Directory
set /p choice=" "
if '%Choice%' == 'B' goto :Searchlogs
if '%Choice%' == 'b' goto :Searchlogs
if '%Choice%' == 'X' goto :exit
if '%Choice%' == 'x' goto :exit
if '%Choice%' == ' ' echo "%Choice%" is not a valid option. Please try again.
if not %choice% == set choice =%choice:~0,1%
echo "%Choice%" is not a valid option. Please try again.
TIMEOUT /T -1
goto :start
cls
goto :start
TIMEOUT /T -1
:exit
cls
exit
:Searchlogs
echo Please enter the Thread number in format "1234"
copy \\Servername\log\logfile.log c:\users\%username%\Desktop\logfile.txt
copy \\Servername\log\logfile-1.log c:\users\%username%\logs\logfile-1.txt
set /p threadnumber=" "
find " %threadnumber%:" C:\Users\%username%\Desktop\logfile*.txt >C:\Users\%username%\Desktop\%threadnumber%.txt
echo Please paste in error text.
set /p Errortext=" "
find "%Errortext%" C:\Users\%username%\Desktop\%threadnumber%.txt" >C:\Users\%username%\Desktop\Error.txt
TIMEOUT /T -1
goto :start
#echo off
setlocal EnableDelayedExpansion
:Searchlogs
copy \\Servername\log\logfile.log c:\users\%username%\Desktop\logfile.txt
copy \\Servername\log\logfile-1.log c:\users\%username%\logs\logfile-1.txt
echo Please enter the Thread number in format "1234"
set /p threadnumber=
find " %threadnumber%:" C:\Users\%username%\Desktop\logfile*.txt > C:\Users\%username%\Desktop\%threadnumber%.txt
echo Please paste in error text.
set /p Errortext=
set i=0
set "errorFound="
(for /F "tokens=1-4*" %%a in (C:\Users\%username%\Desktop\%threadnumber%.txt) do (
set /A i+=1
set "line[!i!]=%%a %%b %%c %%d %%e"
if "%%e" equ "End" (
if defined errorFound (
for /L %%i in (1,1,!i!) do echo !line[%%i]!
echo/
)
set i=0
set "errorFound="
) else if "%%e" equ "%Errortext%" (
set errorFound=true
)
)) > C:\Users\%username%\Desktop\Error.txt
Example:
Please enter the Thread number in format "1234"
1234
Please paste in error text.
ERROR ERROR ERROR
01/01/01 12:00:00: thread 1234: Start
01/01/01 12:00:00: thread 1234: *: Other userful information
01/01/01 12:00:00: thread 1234: *: Other userful information
01/01/01 12:00:00: thread 1234: ERROR ERROR ERROR
01/01/01 12:00:00: thread 1234: *: Other userful information
01/01/01 12:00:00: thread 1234: *: Other userful information
01/01/01 12:00:00: thread 1234: End
#ECHO OFF
SETLOCAL
SET /p "findme=Identifier to find (eg thread 1234)"
SET /p "errortext=Error text to find (eg ERROR ERROR ERROR)"
FINDSTR /c:"%findme%" <q23318892.txt >threadfile.txt
SET "block="
SET "errorblock="
FOR /f "delims=" %%a IN (threadfile.txt) DO (
ECHO(%%a|FIND "%findme%: Start" >NUL
IF NOT ERRORLEVEL 1 SET block=Y
IF DEFINED block >>threaderror.txt ECHO(%%a
ECHO(%%a|FIND "%findme%: %errortext%" >NUL
IF NOT ERRORLEVEL 1 SET errorblock=Y
ECHO(%%a|FIND "%findme%: End" >NUL
IF NOT ERRORLEVEL 1 (
IF DEFINED errorblock GOTO done
IF DEFINED block DEL threaderror.txt
SET "block="
SET "errorblock="
)
)
:done
IF EXIST threaderror.txt (TYPE threaderror.txt) ELSE (ECHO threaderror.txt NOT created)
GOTO :EOF
This solution assumes that the first 'error' block is required. You'd need to set up the text to be found and filenames. I used a file named q23318892.txt containing your data for my testing.
fixed to suit further requirement: allow user-input
type file.log | Findstr /c:"Thread 1234" > FileWithThreadInIt.txt
type FileWithThreadInIt.txt | Findstr /c:"Error" >FileWithThreadErrors.txt
type FileWithThreadInIt.txt | Findstr /v /c:"Error" >FileWithThreadNoErrors.txt
type file.log | Findstr /v /c:"Thread 1234" > FileWithOtherThreads.txt
Use variables
Set /p Thread=Enter thread number
type file.log | Findstr /c:"Thread %Thread%" > FileWithThreadInIt.txt
Related
I wrote a massively multithreaded application in perl which basically scans a file- or directory-structure for changes (either using inotify or polling). When it detects changes, it launches subthreads that execute programs with the changed files as an argument, according to a configuration.
This works quite nice so far, with the exception that my application also tries to capture stdout and stderr of the externally executed programs and write them to log files in a structured manner.
I am, however, experiencing an occasional but serious mixup of output here, in the way that every when and then (usually under heavy workload, of course, so that the normal tests always run fine) stdout from a program on thread A gets into the stdout pipe FH of another program running on thread B at the very same time.
My in-thread code to run the externally executed programs and capture the output from them looks like this:
my $out;
$pid = open($out, "( stdbuf -oL ".$cmd." | stdbuf -oL sed -e 's/^/:::LOG:::/' ) 2>&1 |") or xlog('async execution failed for: '.$cmd, LOG_LEVEL_NORMAL, LOG_ERROR);
# catch all worker output here
while(<$out>)
{
if( $_ =~ /^:::LOG:::/ )
{
push(#log, $wname.':::'.$acnt.':::'.time().$_);
} else {
push(#log, $wname.':::'.$acnt.':::'.time().':::ERR:::'.$_);
}
if (time() - $last > 1)
{
mlog($acnt, #log);
$last = time();
#log = ();
}
}
close($out);
waitpid($pid, 0);
push(#log, $wname.':::'.$acnt.':::'.time().':::LOG:::--- thread finished ---');
stdbuf is being used here to suppress buffering delays whereever possible and the sed pipe is being used to avoid the need of handling multiple fds in the reader while still being able to separate normal output from errors.
Captured log lines are being stuffed into a local array by the while loop and every other second contents of that array are handed over to a thread-safe global logging method using semaphores that makes sure nothing gets mixed up.
To avoid unneccesary feedback loops from you: I certainly have made sure (using debug output) that the output really is mixed up on the thread level already and is not a result of locking mistakes later in the output chain!
My Question is: how can it be, that the thread-locally defined $out FH from thread A does receive output that definitely comes from a totally different program running in thread B and therefor should end up in the separately defined thread-local $out FH of thread B? Did I make a grave mistake at some point here or is it just that perl threading is a mess? And, finally, what would be a recommended way to separate the data properly (and preferably in some elegant way)?
Update: due to popular demand I have added the full thread method here:
sub async_command {
my $wname = shift;
my $cmd = shift;
my $acnt = shift;
my $delay = shift;
my $errlog = shift;
my $last = time();
my $pid = 0;
my #log;
my $out;
push(#log, $wname.':::'.$acnt.':::'.$last.':::LOG:::--- thread started ---'.($delay ? ' (sleeping for '.$delay.' seconds)':''));
push(#log, $wname.':::'.$acnt.':::'.$last.':::ERR:::--- thread started ---') if ($errlog);
if ($delay) { sleep($delay); }
# Start worker with output pipe. stdbuf prevents unwanted buffering
# sed tags stdout vs stderr
$pid = open($out, "( stdbuf -oL ".$cmd." | stdbuf -oL sed -e 's/^/:::LOG:::/' ) 2>&1 |") or xlog('async execution failed for: '.$cmd, LOG_LEVEL_NORMAL, LOG_ERROR);
# catch all worker output here
while(<$out>)
{
if( $_ =~ /^:::LOG:::/ )
{
push(#log, $wname.':::'.$acnt.':::'.time().$_);
} else {
push(#log, $wname.':::'.$acnt.':::'.time().':::ERR:::'.$_);
}
if (time() - $last > 1)
{
mlog($acnt, #log);
$last = time();
#log = ();
}
}
close($out);
waitpid($pid, 0);
push(#log, $wname.':::'.$acnt.':::'.time().':::LOG:::--- thread finished ---');
push(#log, $wname.':::'.$acnt.':::'.time().':::ERR:::--- thread finished ---') if ($errlog);
mlog($acnt, #log);
byebye();
}
So... here you can see that #log as well as $out are thread-local variables. The xlog (global log) and mlog-methods (worker logs) actually use Thread::Queue for further processing. I just dont want to use it more than once a second per thread to avoid too much locking overhead.
I have duplicated the push(#log... statements into xlog() calls for debugging. Since the worker name $wname is somewhat tied to the $cmd executed and $acnt is a number unique for each thread, I came to see clearly that there is log output being read from the $out FH that definitely comes from a different $cmd than the one executed in this thread, while $acnt and $wname stay the ones that actually belong to the thread. Also I can see that these log lines then do NOT appear on the $out FH in the other thread where they should be.
I want to get 200 files using multithread, so I modify a TCL example as below.
But the result is strange, the total number of output files is random, about 135. I was confused that how the thread started change the value of variable $thread.
package require Thread
puts "*** I'm thread [thread::id]"
for {set thread 1} {$thread <= 200} {incr thread} {
set thread_ida $thread
tsv::set app global_thread_num $thread_ida
set id [thread::create -joinable {
puts [ tsv::get app global_thread_num ]
set thread_id [ tsv::get app global_thread_num ]
puts "${thread_id}thread_id"
set outFile "./test/${thread_id}"
append outFile ".tmd"
puts $outFile
set FileOut [open $outFile w+]
puts $FileOut "${thread_id}thread_id"
}] ;# thread::create
puts "*** Started thread $id"
lappend threadIds $id
} ;# for
puts "*** Existing threads: [thread::names]"
# Wait until all other threads are finished
foreach id $threadIds {
thread::join $id
}
puts "*** That's all, folks!"
The problem you've got is that these two lines:
puts [ tsv::get app global_thread_num ]
set thread_id [ tsv::get app global_thread_num ]
are not guaranteed to get the same value at all, nor are they at all likely to synchronise with the setting of the shared variable in the outer loop. Threads in Tcl have a reasonable amount of overhead during launch.
Instead, what you should do is make threads with the description of work inside a procedure and then send a simple message to them with the ID to start the real processing; that's much easier to make work.
package require Thread
puts "*** I'm thread [thread::id]"
for {set thread 1} {$thread <= 200} {incr thread} {
set id [thread::create -joinable {
proc DoWork {thread_id} {
# Only one puts here
puts "${thread_id}thread_id"
set outFile "./test/${thread_id}"
append outFile ".tmd"
puts $outFile
set FileOut [open $outFile w+]
puts $FileOut "${thread_id}thread_id"
# Close the channel, please...
close $FileOut
# Thread done, and since we're using joinable threads it should die now
thread::release
}
thread::wait
}] ;# thread::create
puts "*** Started thread $id"
lappend threadIds $id
# Start the work going, passing over the numeric ID in the "message"
thread::send -async $id [list DoWork $thread]
} ;# for
puts "*** Existing threads: [thread::names]"
# Wait until all other threads are finished
foreach id $threadIds {
thread::join $id
}
puts "*** That's all, folks!"
The key things here are that we create a procedure in each thread (DoWork) to receive the message, get the thread to wait for messages with thread::wait, and then launch the work by sending a message in with thread::send -async. The work destroys the thread with thread::release; it needs to do so explicitly otherwise it'll end up back in thread::wait waiting for the next message.
I'd probably use a thread pool in production code, as they're easier to scale to the hardware available in a particular deployment. The DoWork procedure — without the thread::release — would be defined in the pool's -initcmd option. The thread::send -async would be replaced by posting work to the pool, and you'd be waiting for the jobs instead of the threads.
package require Thread
puts "*** I'm thread [thread::id]"
set pool [tpool::create -maxworkers 48 -initcmd {
proc DoWork {thread_id} {
# Only one puts here
puts "${thread_id}thread_id"
set outFile "./test/${thread_id}"
append outFile ".tmd"
puts $outFile
set FileOut [open $outFile w+]
puts $FileOut "${thread_id}thread_id"
# Close the channel, please...
close $FileOut
}
}]
for {set thread 1} {$thread <= 200} {incr thread} {
lappend work [tpool::post -nowait $pool [list DoWork $thread]]
}
# Wait until all work is finished
foreach id $work {
tpool::wait $pool $id
}
puts "*** That's all, folks!"
tpool::release $pool
Hi I have written a script, which was working fine previously with 'snoop' commands. This script forks child in the script to start tcpdump. When i have to stop the dump I kill the child but when i look at the pcap generated in wireshark, it shows the error "The capture file appears to have been cut short in the middle of a packet". My commands are
my $snoopAPP = &startService("tcpdump -w /tmp/app.pcap -i bond0>/dev/null 2>&1" , '');
kill 9, -$snoopAPP;waitpid $snoopAPP, 0;
sub startService(){
#runs a program in background and returns PID which can be used later to kill the process
#arguments are 1, path , 2nd the name of the file
my $processPath = $_[0];chomp($processPath);
if ($_[1] ne ''){$processPath = $processPath . " >$path/$_[1].log";}
print "\nStarting ... \n-- $processPath\n";
my $pid = fork();
die "unable to fork $processPath: $!" unless defined($pid);
if (!$pid) { # child
setpgrp(0, 0);
exec("$processPath");
die "\nunable to exec: $!\n";
exit;
}
print " ---- PID: $pid\n";
return $pid;
}
Another post suggests to wait for tcpdump to exit, which I am doing already, but still it results in the same error message.
Try
kill 15, -$snoopAPP
Signal 9, SIGKILL, is an immediate terminate, and doesn't give the application the opportunity to finish up, so, well, the capture file stands a good chance of being cut short in the middle of a packet.
Signal 15, SIGTERM, can be caught by an application, so it can clean up before terminating. Tcpdump catches it and finishes writing out buffered output.
I have a little issue when my batch writes the time before 10 AM.
What it does is scan the barcode from a user card and writes date, time and that code into a .csv file, so that it can be imported into a excel file.
here is my code:
#echo off
...
:: set /p fname="Digite o nome do arquivo a guardar os logs: "
set fname=refeicao
goto REF
:REF
....
set cod=""
set /p cod="-> "
if /i %cod%=="" goto err
goto ok
:print
echo %date% %time% %cod%>>w:\REFEICAO\%fname%.csv
goto REF
:err
....
goto REF
:OK
...
goto print
:qq
exit
The problem is that when time is earlier than 10AM, it exports with a space before, let's suppose it's 4 AM. It would be like: " 4:00:00,00" instead of "04:00:00,00"
How could i get rid of that space or turn it into a "0"?
[Actual code inserted - minus irrelevant echos etc.]
echo %time: =0%
echo %time: =%
The first line changes spaces to 0, the second one removes spaces. Use the one better fits your problem
I am playing with using flock, a bash command for file locks to prevent two different instances of the code from running more than once.
I am using this testing code:
( ( flock -x 200 ; sleep 10 ; echo "original finished" ; ) 200>./test.lock ) &
( sleep 2 ; ( flock -x -w 2 200 ; echo "a finished" ) 200>./test.lock ) &
I am running 2 subshells (backgrounded). The (flock NUM; ...) NUM>FILE syntax is from flock's man page.
I expect that the first subshell will get an exclusive lock on test.lock, then wait 10 seconds, then print "original finished", all the time holding the lock. The second subshell will start at more or less the same time, wait 2 seconds, then try to get a lock on test.lock, but timeout after 2 seconds. If it gets a lock, then it'll print "a finished". If it doesn't get the lock, that subshell should stop, and nothing should be printed.
Since the first subshell is waiting longer, it will keep the lock for 10 seconds, so the second subshell should not get the lock, and shouldn't finish. i.e. one should see "original finished" printed and not both.
What actually happens is that "a finished" is printed, then "original finished" is printed.
This implies that that the second subshell is either (a) not using the same lock as the first subshell or (b) that it fails to get the lock, but continues to execute or (c) something else.
Why don't those locks work as I expect?
The issue is that, if the flock process fails to get the lock within the timeout, it has no way of killing the parent process (i.e. the shell that spawned it) - all it can do is return a failure return code. You need to check that return code before continuing:
flock <params> && <do other stuff>
so
( ( flock -x 200 ; sleep 10 ; echo "original finished" ; ) 200>./test.lock ) & ( sleep 2 ; ( flock -x -w 2 200 && echo "a finished" ) 200>./test.lock ) &
does what you want.