Batch Script - muliple duplicate strings in a row - string

Question

The below script provides the output of each occurence of the token# 1 field but I need add two more conditions.
a. Output should be provided.i.e. only when it is more than one since I have millions of records in a file
b. if there are mulitple strings.i.e. combination of Key fields in a row needs to checked across all the lines for duplicates in a file.
#ECHO OFF
SETLOCAL enabledelayedexpansion
FOR %%c IN ($ #) DO FOR /f "delims==" %%i IN ('set %%c 2^>nul') DO
"SET %%i="
SET /a count=0
FOR /f "tokens=1delims=|" %%i IN (fscif.txt) DO (
SET /a count+=1
IF DEFINED $%%i (SET "$%%i=!$%%i! & !count!") ELSE (SET "$%%i=!count!")
SET /a #%%i+=1 )
FOR /f "tokens=1*delims=$=" %%i IN ('set $ 2^>nul') DO ( ECHO %%i;!#%%i! times;line no %%j
)
For Example:
Original File (Considering token 1 & 3 are key fields)
123|12|Jack
124|23|John
123|14|Jack
125|15|Sam
125|66|Sam
125|66|Sam
Ouput file:
123|Jack;2 times;line no 1 & 3
125|Sam;3 times;line no 4 & 5 & 6

Answer 1

#ECHO OFF
SETLOCAL enabledelayedexpansion
:: Temporary filename
:tloop
SET "temppfx=%temp%\%random%"
IF EXIST "%temppfx%*" GOTO tloop
:: Hold that tempfile name...
ECHO.>"%temppfx%_"
:: a long string of spaces note the end-of-string quote -----here--v
SET "spaces= "
SET /a count=0
(
FOR /f "tokens=1,3 delims=|" %%a IN (fscif.txt) DO (
SET /a count+=1
SET "field1=%%a%spaces%"
SET "field3=%%b%spaces%"
SET "fieldc=%spaces%!count!"
ECHO(!field1:~0,10!!field3:~0,12!^|!fieldc:~-8!^|!count!^|%%a^|%%b
)
)>"%temppfx%1"
:: Now report
SET "key=x"
SET /a count=0
(
FOR /f "tokens=1,3* delims=|" %%a IN ('sort "%temppfx%1" ') DO (
IF "!key!"=="%%a" (
SET "line=!line! %%b"
SET /a count+=1
) ELSE (IF !count! neq 0 CALL :output
SET key=%%a
SET line=%%b
SET "data=%%c"
SET /a count=1
)
)
CALL :output
)>report.txt
del "%temppfx%*"
GOTO :eof
:output
ECHO(!data!;%count% times;line nos %line: = ^& %
GOTO :eof
As I explained earlier, with millions of records, you are likely to run out of environment space. As posted above, I reckon you may still run out because the report of line numbers may be huge - no idea - you are familiar with your real data.
Essentially, the first thing to do is to establish a temporary file.
Starting with the tokens required in the input file - I followed 1 and 3 but no doubt there may be more - just follow the bouncing ball...
The selected fields are padded - on the right for text fields and on the left for the count field using the spaces variable.
Then the tempfile output is generated. I randomly chose a maximum length of 10 for the first field and 12 for the second. These two are combined to give the key field. The leading-filled count field is output as the second column so that after SORTing, the data will appear grouped by key, then line number. The remaining columns of interest are then reproduced.
The data is then sorted as input to the next for/f loop - only tokens 1 (the key), 3 (the raw line number) and "the rest" (the key without the padding) are of interest
Then it's simply a matter of counting matching keys and accumulating the line number in line and reporting when the key changes. One last output is required to report the very last data item, and we're done.

Answer 2

For this ugly batch job I recommend to use sed and uniq from the GNUWin Project:
#echo off&setlocal enabledelayedexpansion
set "inputfile=file"
set "outputfile=out"
set "tempfile=%temp%\%random%"
<"%inputfile%" sed "s/|.*|/|.*|/"|sort|uniq -d>"%tempfile%"
(for /f "usebackqtokens=1-3delims=|" %%i in ("%tempfile%") do (
set /a cnt=0&set "line="
for /f "delims=:" %%a in ('findstr /nr "%%i|%%j|%%k" "%inputfile%"') do set "line=!line!%%a & "&set /a cnt+=1
echo(%%i^|%%k;!cnt! times;line no !line:~0,-3!
))>"%outputfile%"
del "%tempfile%"
type "%outputfile%"
.. output is:
123|Jack;2 times;line no 1 & 3
125|Sam;3 times;line no 4 & 5 & 6

Answer 3

The Batch file below do what you want:
#echo off
setlocal EnableDelayedExpansion
rem Assemble "tokensValues" and "lastToken" variables from the parameters
set letters=0abcdefghijklmnopqrstuvwxyz
set tokensValues=%%!letters:~%1,1!
set lastToken=%1
:nextArg
shift
if "%1" equ "" goto endArgs
set "tokensValues=!tokensValues!#%%!letters:~%1,1!"
set lastToken=%1
goto nextArg
:endArgs
rem Accumulate duplicated strings
set line=0
for /F "tokens=1-%lastToken% delims=|" %%a in (fscif.txt) do (
set /A line+=1
if not defined lines[%tokensValues%] (
set lines[%tokensValues%]=!line!
) else (
set "lines[%tokensValues%]=!lines[%tokensValues%]! & !line!"
)
set /A times[%tokensValues%]+=1
)
rem Show the result
for /F "tokens=2* delims=[]=" %%a in ('set lines[ 2^>NUL') do (
if !times[%%a]! gtr 1 (
set string=%%a
set "string=!string:#=|!"
echo !string!;!times[%%a]! times;line no %%b
)
)
You must provide the number of the desired key fields in the parameters. For example, to consider 1 & 3 as key fields:
prog.bat 1 3
You may provide a maximum of 26 key fields with positions from 1 to 26; this limit may be easily increased up to 52.
This Batch file does not use any external command and works over the original file, so it should run fast. If the file is large, a sort or findstr command over it will take too long (even a simple copy, for that matter).
If we take your example data as representative of the real data, lines variable should store about 2500-3000 lines (that is, number of different lines where the same key fields appear), and with a total environment space of 64 MB I think this program will be capable of process your large files.