I have a file of the following format. Each line is a record of three values separated by |. I have written a batch script to validate whether each line has 3 records in it and report if there are any bad records(less or more than 3 values)
VALUE11|VALUE12|VALUE13|
VALUE21|VALUE22|VALUE23|
VALUE31|VALUE32|VALUE33|
PSUEDO CODE:
For each line in the file, DO
(1) Read line to a variable.
(2) Write to a temporary file and get file length.
(3) Remove | from the line.
(4) Write this modified line to temporary file and get file length.
(5) The difference between the two computed lengths gives the value count
Code:
#echo off
setlocal EnableDelayedExpansion
set /A column_count=3
set file_name=%1
for /f "delims=" %%m in (!file_name!) do (
set current_line=%%m
echo.!current_line!>temp_file
:: Get length
for %%s in (temp_file) do set /a curr_line_length=%%~zs
:: Remove |
set "current_line_withoutpipes=!current_line:|=!"
echo.!current_line_withoutpipes!>temp_file
:: Get new length
for %%s in (temp_file) do set /a current_line_withoutpipes_length=%%~zs
:: Diff gives the pipe count
set /A line_column_count=!curr_line_length!-!current_line_withoutpipes_length!
if !column_count! EQU !line_column_count! (
echo Good record
) else (
echo bad record
)
)
endlocal
Problem:
The problem comes when there are ! in the file and DelayedExpansion is set. The characters between two ! are getting truncated when they are set to a variable. In order to over come the problem, I put endlocal and then set DelayedExpansion again. With this change, the problem of ! is resolved but the variables defined before "endlocal" are no longer available after the "endlocal" statement.
Changed Code:
.
.
for /f "delims=" %%m in (!file_name!) do (
endlocal & set current_line=%%m
setlocal EnableDelayedExpansion
echo.!current_line!>temp_file
.
.
Appreciate if someone could help!!
NOTE: The actual file contains hundreds of records.
You could use a setlocal instead of an endlocal in the beginning.
for /f "delims=" %%m in (!file_name!) do (
setlocal DisableDelayedExpansion
set "current_line=%%m"
setlocal EnableDelayedExpansion
echo(!current_line!>temp_file
...
endlocal
endlocal
)
But even this fails when a line begins with a ; and empty lines are dropped too.
For a complete solution you could read Batch files: How to read a file?
Related
I am running Windows 10 Pro using batch files (open to using VBS and PS1 files) and I have a text file automatically exported by software that can look like this:
Sub_Group691_FE7IP11_2017-12-12.sldasm_bin/parts/Loft-Project.sldasm_bin/parts/Loft-Project...
Sub_Group691_FE7IP12_2017-12-12.sldasm_bin/parts/Loft-Project.sldasm_bin/parts/Loft-Project...
Sub_Group691_FE7IP13_2017-12-12.sldasm_bin/parts/Loft-Project.sldasm_bin/parts/Loft-Project...
Each line continues specifying sub-parts after the "..." and could contain "poison" characters. The Sub_Group part is pulled from the filename and can also contain "poison" characters.
What I am looking to do is export just the filename which is right at the beginning of each line, up to and including the first file extension, in this case ".sldasm." Everything to the right of the first instance of .sldasm should be trimmed.
What I have cobbled together so far from research on Stackoverflow is:
#ECHO OFF
SETLOCAL ENABLEDELAYEDEXPANSION
SET "inputfile=C:\Scratch\ASMExport.txt"
SET "outputfile=C:\Scratch\InputFiles.txt"
(
FOR /f "usebackqdelims=" %%a IN ("%inputfile%") DO (
SET "currentline=%%a"
ECHO("!currentline:.sldasm=.sldasm & rem "!"
)
)>"%outputfile%"
GOTO :EOF
My problem lies with the "rem" line, which does not seem to work as intended either because of being within a FOR loop or because of needing to enable delayed expansion. It seems to be parsing the "& rem" as text, which looks to be because of the way delayed expansion works. What I get from the above lines is:
SubGroup691_FE7IP11_2017-12-12.sldasm" & rem "/bin/parts/Loft-Project.sldasm" & rem ""
SubGroup691_FE7IP12_2017-12-12.sldasm" & rem "/bin/parts/Loft-Project.sldasm" & rem ""
SubGroup691_FE7IP13_2017-12-12.sldasm" & rem "/bin/parts/Loft-Project.sldasm" & rem ""
I can use this same line outside the loop and without ENABLEDELAYEDEXPANSION like this:
#ECHO OFF
SETLOCAL
SET "inputstring=Sub_Group691_FE7IP11_2017-12-12.sldasm_bin/parts/Loft-Project.sldasm_bin/parts/Loft-Project"
SET "outputstring=%inputstring:.sldasm=.sldasm & rem "%"
echo %outputstring%
The output to that would give me what I am looking for:
Sub_Group691_FE7IP11_2017-12-12.sldasm
In searching, I am beginning to think that rem cannot be used in this way, and I must move to a token delimiter loop using a bogus delimiter.
I would be content in getting this to work and not worrying about "poison" characters in the filename by being diligent about naming files correctly.
Thank you for your help!
The & rem approach to truncate strings to the right cannot work with delayed expansion (!), it relies on normal/immediate expansion (%). This is because immediate expansion is done before commands (like rem) are recognised, but delayed expansion happens afterwards, so the rem command is never executed.
However, you could in the loop replace every .sldasm by a forbidden character like | and then split the string by a for /F loop, like this:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "InputFile=C:\Scratch\ASMExport.txt"
set "OutputFile=C:\Scratch\InputFiles.txt"
set "Extension=.sdlasm"
> "%OutputFile%" (
for /F usebackq^ delims^=^ eol^= %%A in ("%InputFile%") do (
set "CurrentLine=%%A"
setlocal EnableDelayedExpansion
set "CurrentLine=!CurrentLine:%Extension%=|!"
for /F "delims=|" %%B in ("!CurrentLine!") do (
endlocal
echo(%%B
setlocal EnableDelayedExpansion
)
endlocal
)
)
endlocal
exit /B
Delayed expansion is toggled so that no for variables become expanded when it is enabled, in order to avoid loss of or problems with exclamation marks.
#ECHO OFF
SETLOCAL
SET "sourcedir=U:\sourcedir"
SET "filename1=%sourcedir%\q47788829.txt"
FOR /f "usebackqtokens=1delims=/" %%a IN ("%filename1%") DO (
FOR /f "tokens=1*delims=." %%b IN ("%%a") DO (
FOR /f "tokens=1*delims=_" %%d IN ("%%c") DO (
ECHO %%%%-a=%%~a %%%%-b=%%~b %%%%-c=%%~c %%%%-d=%%~d %%%%-e=%%~e ^<^<
)
)
)
GOTO :EOF
You would need to change the setting of sourcedir to suit your circumstances.
I used a file named q47788829.txt containing your data for my testing.
You should be able to assemble your required report data from the elements %%a..%%e displayed. All a matter of using tokens and delims constructively.
If your input is the text that you wrote, with the same structure, I think this batch should work and extract the names that you want quickly.
CODE:
#echo off
setlocal enabledelayedexpansion
set "inputFile=input.txt"
set "outputFile=result.txt"
for /f "delims=/ tokens=1" %%A in (%inputFile%) do (
set "tempFileName=%%A"
echo !tempFileName:~0,-4!>> %outputFile%
)
You only had to adjust the variables 'inputFile' and 'outputFile' to your needs.
I wrote this code so I can remove a column from a csv file.
#echo off
setlocal enableextensions enabledelayedexpansion
type nul > tmp.txt
SET /A COUNT=0
for /F "tokens=*" %%A in (d.csv) do (
set LINE="%%A"
set /A COUNT+=1
for /F "tokens=1,2,3,4,5,6,7,8,* delims=," %%a in (!LINE!) do (
set row[0]=%%a
set row[1]=%%b
set row[2]=%%c
set row[3]=%%d
set row[4]=%%e
set row[5]=%%f
set row[6]=%%g
set row[7]=%%h
)
echo !row[0]!,!row[2]!,!row[3]!,!row[4]!,!row[5]!,!row[6]! >>tmp.txt
echo.
)
endlocal
Test file:
A1,B1,C1,D1,la la,,1
A2,B2,C2,D2, ,fef 3,
A3,B3,C3,D3,be be ,bo,bo 1
A4,B4,C4,D4,tu tu,tu 7,881
Output file:
A1,C1,D1,la la,1,
A2,C2,D2, ,fef 3,
A3,C3,D3,be be ,bo,bo 1
A4,C4,D4,tu tu,tu 7,881
I don't get why in the output file at the first line the ,, is eliminated and a , added at the end. Also I am wondering if there is a better way to do this.
Thanks!
#ECHO OFF
SETLOCAL ENABLEDELAYEDEXPANSION
(
FOR /f "delims=" %%a IN (q29441490.txt) DO (
SET "line=%%a"
SET "line=!line:,= , !"
FOR /f "tokens=1,2*delims=," %%p IN ("!line!") DO (
SET "line=%%p,%%r"
SET "line=!line: , =,!"
ECHO(!line!
)
)
)>u:\newfile.txt
GOTO :EOF
I used a file named q29441490.txt containing your data for my testing.
Produces u:\newfile.txt
the separators between tokens are delimiter sequences, so ,, is seen as one separator, hence the fields appear moved by one place.
Grab each line, replace each , with , tokenise (you don't say explicitly, but you appear to want to eliminate the second column) so %%q gets the first column and %%r the remainder of the line following the second. Concatenate these, insert the comma and then reverse the substitution.
If you wanted to eliminate another column, then a different tokens element should be specified and the restructure of the line would need to be adjusted.
How could one select a random line of text from a text file and set it in a variable to use?
The Batch program below is Eitan's solution slightly modified to run faster:
#echo off
setlocal EnableDelayedExpansion
set INPUT_FILE="test.txt"
:: # Count the number of lines in the text file and generate a random number
for /f "usebackq" %%a in (`find /V /C "" ^< %INPUT_FILE%`) do set lines=%%a
set /a randnum=%RANDOM% * lines / 32768 + 1, skiplines=randnum-1
:: # Extract the line from the file
set skip=
if %skiplines% gtr 0 set skip=skip=%skiplines%
for /f "usebackq %skip% delims=" %%a in (%INPUT_FILE%) do set "randline=%%a" & goto continue
:continue
echo Line #%randnum% is:
echo/!randline!
Like it is already mentioned here in StackOverflow, among others, %RANDOM% expands to a random number between 0 and 32767.
You can use this mechanism to generate a random line number. However, to make it a valid line number you will have to normalize it by the the number of lines in the input text file.
Here's a simple script that shows how to do it:
#echo off
setlocal enabledelayedexpansion
set INPUT_FILE="test.txt"
:: # Count the number of lines in the text file and generate a random number
set lines=0
for /f "usebackq" %%a in (%INPUT_FILE%) do set /a lines+=1
echo %RANDOM% >nul
set /a randnum=%RANDOM% * !lines! / 32768 + 1
:: # Extract the line from the file
set lines=0
for /f "usebackq tokens=*" %%a in (%INPUT_FILE%) do (
set /a lines+=1
if !lines!==!randnum! set randline=%%a
)
echo Line #!randnum! is:
echo.!randline!
Here's yet another approach. It reads the file name from the command line and uses a FOR /L loop to get to the calculated line number:
#ECHO OFF
FOR /F "" %%I IN ('FIND /C /V "" ^<%1') DO SET /A lines=%%I
SET /A skip=%RANDOM%%%lines
<%1 (
FOR /L %%I IN (1,1,%skip%) DO (
SET /P line=
)
SET line=
SET /P line=
)
ECHO(%line%
The FOR /F loop simply gets the number of lines in the file (the method is borrowed from #Aacini's answer).
A rather simplistic formula then calculates the number of lines to skip in the file.
Next, the file is read. The FOR /L loop merely consumes the specified number of lines using a SET /P instruction. Following the loop, one more SET /P command reads the line that is eventually ECHOed.
The above implementation is just to show the basic idea. It is not without issues, but some of them could easily be resolved:
There's no testing whether the parameter is indeed supplied. If it is absent, the script will break. You could add the necessary check at the beginning of the script like this:
IF "%~1"=="" GOTO :EOF
If there's no parameter, this command terminates the script by sending the control to the end of the script (GOTO :EOF).
The file specified might not exist. Again, you could test that at the beginning, just after verifying that the parameter is supplied, to terminate the script if necessary:
IF NOT EXIST %1 GOTO :EOF
If the file is empty, the lines will be 0 and the subsequent expression using it will run into a division by zero error. Therefore, you'll also need to test the resulting line count (and prevent the script from running further if the count is indeed 0). You can do that by adding the following line just after the FOR /F loop:
IF %lines%==0 GOTO :EOF
Like I said, the formula is somewhat simplistic. It doesn't produce a number greater than 32767, which is the limitation of %RANDOM%. That might well be enough for you, but in case it is not, you could extend the range to 230-1 using two %RANDOM% calls like this:
SET /A skip=(%RANDOM%*32768+%RANDOM%)%%lines
So, here's the same script again, amended to address the above mentioned issues:
#ECHO OFF
IF "%~1"=="" GOTO :EOF
IF NOT EXIST %1 GOTO :EOF
FOR /F "" %%I IN ('FIND /C /V "" ^<%1') DO SET /A lines=%%I
IF %lines%==0 GOTO :EOF
SET /A skip=(%RANDOM%*32768+%RANDOM%)%%lines
<%1 (
FOR /L %%I IN (1,1,%skip%) DO (
SET /P line=
)
SET line=
SET /P line=
)
ECHO(%line%
One other note is that, if you like, you can add messages explaining the reason for the premature termination of the script. Basically, wherever you want to add the message, you'll just need to replace the single
GOTO :EOF
with
(ECHO your message & GOTO :EOF)
For instance:
IF NOT EXIST %1 (ECHO Error! File not found & GOTO :EOF)
I'm working with very large FIX message log files. Each message represents a set of tags separated by SOH characters.
Unlike MQ messages, individual FIX tags (and overall messages) do not feature fixed length or position. Log may include messages of different types (with a different number & sequence of tags).
Sample (of one of many types of messages):
07:00:32 -SEND:8=FIX.4.0(SOH)9=55(SOH)35=0(SOH)34=2(SOH)43=N(SOH)52=20120719-11:00:32(SOH)49=ABC(SOH)56=XYZ(SOH)10=075
So the only certain things are as follows: (1) tag number with equal sign uniquely identifies the tag, (2) tags are delimited by SOH characters.
For specific tags (just a few of them at a time, not all of them), I need to get a list of their distinct values - something like this:
49=ABC 49=DEF 49=GHI...
Format of the output doesn't really matter.
I would greatly appreciate any suggestions and recommendations.
Kind regards,
Victor O.
Option 1
The batch script below has decent performance. It has the following limitations
It ignores case when checking for duplicates.
It may not properly preserve all values that contain = in the value
EDIT - My original code did not support = in the value at all. I lessened that limitation by adding an extra SOH character in the variable name, and changed the delims used to parse the value. Now the values can contain = as long as unique values are differentiated before the =. If the values differentiate after the = then only one value will be preserved.
Be sure to fix the definition of the SOH variable near the top.
The name of the log file is passed as the 1st parameter, and the list of requested tags is passed as the 2nd parameter (enclosed in quotes).
#echo off
setlocal disableDelayedExpansion
:: Fix the definition of SOH before running this script
set "SOH=<SOH>"
set LF=^
:: The above 2 blank lines are necessary to define LF, do not remove.
:: Make sure there are no existing tag_ variables
for /f "delims==" %%A in ('2^>nul set tag_') do set "%%A="
:: Read each line and replace SOH with LF to allow iteration and parsing
:: of each tag/value pair. If the tag matches one of the target tags, then
:: define a tag variable where the tag and value are incorporated in the name.
:: The value assigned to the variable does not matter. Any given variable
:: can only have one value, so duplicates are removed.
for /f "usebackq delims=" %%A in (%1) do (
set "ln=%%A"
setlocal enableDelayedExpansion
for %%L in ("!LF!") do set "ln=!ln:%SOH%=%%~L!"
for /f "eol== tokens=1* delims==" %%B in ("!ln!") do (
if "!!"=="" endlocal
if "%%C" neq "" for %%D in (%~2) do if "%%B"=="%%D" set "tag_%%B%SOH%%%C%SOH%=1"
)
)
:: Iterate the defined tag_nn variables, parsing out the tag values. Write the
:: values to the appropriate tag file.
del tag_*.txt 2>nul
for %%A in (%~2) do (
>"tag_%%A.txt" (
for /f "tokens=2 delims=%SOH%" %%B in ('set tag_%%A') do echo %%B
)
)
:: Print out the results to the screen
for %%F in (tag_*.txt) do (
echo(
echo %%F:
type "%%F"
)
Option 2
This script has almost no limitations, but it significantly slower. The only limitation I can see is it will not allow a value to start with = (the leading = will be discarded).
I create a temporary "search.txt" file to be used with the FINDSTR /G: option. I use a file instead of a command line search string because of FINDSTR limitations. Command line search strings cannot match many characters > decimal 128. Also the escape rules for literal backslashes are inconsistent on the command line. See What are the undocumented features and limitations of the Windows FINDSTR command? for more info.
The SOH definition must be fixed again, and the 1st and 2nd arguments are the same as with the 1st script.
#echo off
setlocal disableDelayedExpansion
:: Fix the definition of SOH before running this script
set "SOH="
set lf=^
:: The above 2 blank lines are necessary to define LF, do not remove.
:: Read each line and replace SOH with LF to allow iteration and parsing
:: of each tag/value pair. If the tag matches one of the target tags, then
:: check if the value already exists in the tag file. If it doesn't exist
:: then append it to the tag file.
del tag_*.txt 2>nul
for /f "usebackq delims=" %%A in (%1) do (
set "ln=%%A"
setlocal enableDelayedExpansion
for %%L in ("!LF!") do set "ln=!ln:%SOH%=%%~L!"
for /f "eol== tokens=1* delims==" %%B in ("!ln!") do (
if "!!"=="" endlocal
set "search=%%C"
if defined search (
setlocal enableDelayedExpansion
>search.txt (echo !search:\=\\!)
endlocal
for %%D in (%~2) do if "%%B"=="%%D" (
findstr /xlg:search.txt "tag_%%B.txt" || >>"tag_%%B.txt" echo %%C
) >nul 2>nul
)
)
)
del search.txt 2>nul
:: Print out the results to the screen
for %%F in (tag_*.txt) do (
echo(
echo %%F:
type %%F
)
Try this batch file. Add the log file name as parameter. e.g.:
LISTTAG.BAT SOH.LOG
It will show all tag id and its value that is unique. e.g.:
9=387
12=abc
34=asb73
9=123
12=xyz
Files named tagNNlist.txt (where NN is the tag id number) will be made for finding unique tag id and values, but are left intact as reports when the batch ends.
The {SOH} text shown in below code is actually the SOH character (ASCII 0x01), so after you copy & pasted the code, it should be changed to an SOH character. I have to substitute that character since it's stripped by the server. Use Wordpad to generate the SOH character by typing 0001 then press ALT+X. The copy & paste that character into notepad with the batch file code.
One thing to note is that the code will only process lines starting at column 16. The 07:00:32 -SEND: in your example line will be ignored. I'm assuming that they're all start with that fixed-length text.
Changes:
Changed generated tag list file into separate files by tag IDs. e.g.: tag12list.txt, tag52list.txt, etc.
Removed tag id numbers in generated tag list file. e.g.: 12=abc become abc.
LISTTAG.BAT:
#echo off
setlocal enabledelayedexpansion
if "%~1" == "" (
echo No source file specified.
goto :eof
)
if not exist "%~1" (
echo Source file not found.
goto :eof
)
echo Warning! All "tagNNlist.txt" file in current
echo directory will be deleted and overwritten.
echo Note: The "NN" is tag id number 0-99. e.g.: "tag99list.txt"
pause
echo.
for /l %%a in (0,1,99) do if exist tag%%alist.txt del tag%%alist.txt
for /f "usebackq delims=" %%a in ("%~1") do (
rem *****below two lines strip the first 15 characters (up to "-SEND:")
set x=%%a
set x=!x:~15,99!
rem *****9 tags per line
for /f "tokens=1,2,3,4,5,6,7,8,9 delims={SOH}" %%b in ("!x!") do (
call :dotag "%%b" %*
call :dotag "%%c"
call :dotag "%%d"
call :dotag "%%e"
call :dotag "%%f"
call :dotag "%%g"
call :dotag "%%h"
call :dotag "%%i"
call :dotag "%%j"
)
)
echo.
echo Done.
goto :eof
rem dotag "{id=value}"
:dotag
for /f "tokens=1,2 delims==" %%p in (%1) do (
set z=0
if exist tag%%plist.txt (
call :chktag %%p "%%q"
) else (
rem>tag%%plist.txt
)
if !z! == 0 (
echo %%q>>tag%%plist.txt
echo %~1
)
)
goto :eof
rem chktag {id} "{value}"
:chktag
for /f "delims=" %%y in (tag%1%list.txt) do (
if /i "%%y" == %2 (
set z=1
goto :eof
)
)
goto :eof
I have a batch file where I need to write a variable into a specific line of a text file and override what is all ready in that line. I have the code to read specific lines from the file maybe I could switch it around to also write?
Reading lines code:
setLocal EnableDelayedExpansion
for /f "tokens=* delims= " %%a in (variables.txt) do (
set /a N+=1
set v!N!=%%a
)
set variable1=!v1!
set variable2=!v2!
set variable3=!v3!
set variable4=!v4!
I've tried to add echo %variable1% > !v4! something like that but it doesn't work.
I figured it out!! Here is the code for anyone else who might ever need it.
#echo off
setlocal enableextensions enabledelayedexpansion
set inputfile=variables.txt
set tempfile=%random%-%random%.tmp
copy /y nul %tempfile%
set line=0
for /f "delims=" %%l in (%inputfile%) do (
set /a line+=1
if !line!==4 (
echo WORDS YOU REPLACE IT WITH>>%tempfile%
) else (
echo %%l>>%tempfile%
)
)
del %inputfile%
ren %tempfile% %inputfile%
endlocal
Another option might be to overwrite the file entirely. Here's the part to do that:
:saveVars
(
ECHO %v1%
ECHO %v2%
ECHO %v3%
ECHO %v4%
ECHO %v5%
) >variables.txt
GOTO :EOF
That is, if the number of lines is fixed and known beforehand. If not, you might want to store the last value of the increment in your example code and, when saving the variables, use it like this:
:saveVars
SETLOCAL EnableDelayedExpansion
(
FOR /L %%i IN (1,1,%N%) DO (ECHO !v%%i!)
) >variables.txt
ENDLOCAL
GOTO :EOF
I'm assuming here that the v1/v2 etc. variables would be used only for synchronising with the file: when it is read, the lines are stored in those variables, and when any of them (variables) gets changed, you just call the saveVars subroutine immediately. Here's an example how you would use it:
…
SET v2=something
CALL :saveVars
…
SET v4=%somevar%
CALL :saveVars
…
If the file is small, the rewriting should be fast enough.
Not absolutely sure I've understood everything correctly, but if you want to substitute something for an existing part of a text file with a batch script, you'll need to write everything (including the changed part) to a new file first, then delete the original and rename the new file to the original name.
I can't really see a point of reading everything into variables, unless I'm missing something. You could simply iterate over the lines writing them one by one into the new file and replacing the specific line's contents with the substitute text along the way:
setLocal EnableDelayedExpansion
>newFile.txt (
for /f "tokens=* delims= " %%a in (variables.txt) do (
set /a N+=1
if !N! == 4 (ECHO substitute text) ELSE ECHO %%a
)
)
del variables.txt
rename newFile.txt variables.txt
If the substitute text must, in turn, be derived from one of the lines, you could do something like this:
setLocal EnableDelayedExpansion
>newFile.txt (
for /f "tokens=* delims= " %%a in (variables.txt) do (
set /a N+=1
if !N! == 1 SET subst_text=%%a
if !N! == 4 (ECHO !subst_text!) ELSE ECHO %%a
)
)
del variables.txt
rename newFile.txt variables.txt