I need a method to split files into multiple (or even half) based on KB not on number of lines.
I am a Senior EDI Analyst and wrapped data tends to show up as one single long line. Every "solution" I find splits based on number of lines. I need something that will split based on size.
The end-goal is to "Unwrap" this data, meaning each segment will be on its own line. To do this I need to change the delimiters (as there are "special characters" as delimiters).
I do have a solution for that (see below), but for some reason this will not work on files larger than 10 KB. If you know anything about EDI, that's not very big.
I need to find a solution to split files into smaller files of about 5KB each (then I can use the string replacement and re-combine them myself).
Does anyone have an idea of how I might accomplish this with one, huge line?
(Sorry I have to remove the code I placed here only as AN EXAMPLE because someone flagged this as a duplicate WITHOUT READING IT. Please read above and advise.)
The reason you cannot process files > 10k byte is because batch variables (and command lines) are limited to ~8191 bytes.
You are attacking the problem in an inefficient way. Rather than look for a way to split a file into chunks so that you can use your slow batch "solution", you should be looking for a tool that allows you to work with the large files directly, without resorting to splitting, processing, and re-assembly.
As others have stated, PowerShell, JavaScript, and VBS are all good scripting languages that can solve your problem, and they are native to Windows.
If your files are all less than 1 gigabyte in length, then I suggest you try JREPL.BAT - a regex text processing utility. It is pure script (hybrid batch/JScript) that runs natively on any Windows machine from XP onward - no 3rd party exe file required. Full documentation is available from the command line via jrepl /?, or jrepl /?? for paged help.
To Unwrap a file, translating | into *\r\n (\r is carriage return, and \n a newline):
jrepl "|" "*\r\n" /l /m /x /f "wrappedFileName" /o "unwrappedFileName"
To wrap a file (reverse the process)
jrepl "*\r\n" "|" /l /m /x /f "unwrappedFileName" /o "wrappedFileName"
If you put either command within a batch script, then you must use call jrepl instead of jrepl. This is because JREPL is also a batch script, so control will not return to your script unless you use CALL.
Although your description is extensive, there are multiple points that are not clear. There are too many unrelated details that just deviates from the core point of the problem. If each segment in the line is separated by a | delimiter (you did not explained this point, but it is assumed from the example code) and you want to split the file based on a certain KB size (you did not specified how many KB), then a segment may be splitted in two different files. Also, I don't understand how changing the | delimiters by asterisks may help to solve the problem. After read this question several times, I assumed that the problem is this:
"Split a file that just contain a very long line (with not a single CR+LF pair) into segments delimited by | character, so each segment will be on its own line".
The Batch file below is a solution for this problem:
#echo off
setlocal EnableDelayedExpansion
call :ProcessFile < input.txt > output.txt
goto :EOF
:ProcessFile
set "previous="
:nextChunk
rem Read the next 1023-bytes chunk
set /P "chunk="
if errorlevel 1 goto endOfFile
rem Break segment if previous one ends at a chunk limit
if "!chunk:~0,1!" equ "|" if defined previous (
echo !previous!
set "previous="
)
rem Extract each segment from the chunk and place it on its own line
set "last="
for /F "delims=" %%a in (^"!chunk:^|^=^
% This line separate segments by the given delimiter %
!^") do (
if defined last echo !last!
set "last=!previous!%%a"
set "previous="
)
set "previous=!last!"
goto nextChunk
:endOfFile
rem Show the last segment
if defined previous echo !previous!
exit /B
EDIT: JScript solution added
As others have mentioned, you may also use a solution based on JScript, that is a standard programming language preinstalled in all Windows versions from XP on. In this way, the solution is really simple, because you just need to insert the following two lines in your Batch file:
echo WScript.Stdout.Write(WScript.Stdin.ReadAll().replace(/\^|/g,"\r\n")) > replace.js
cscript //nologo replace.js < input.txt > output.txt
This is a very simple, but powerful method that you may use in other similar replace operations; just read the corresponding documentation.
Split file into 5kB chunks:
set file="x.edb"
set max=5000
REM Findstr line limit 8k
REM Workaround: wrap in an archive to generate CRLF pairs for chunks > 8kB
for %i in (%file%) do (
set /a num=%~zi/%max% >nul &REM No. of chunks
set /a last=%~zi%%max% >nul &REM size of last chunk
if %last%==0 set /a num=num-1 &REM ove zero byte chunk
set size=%~zi
)
ren %file% %file%.0
for /l %i in (1 1 %num%) do (
set /a s1=%i*%max% >nul
set /a s2="(%i+1)*%max%" >nul
set /a prev=%i-1 >nul
echo Writing %file%.%i
type %file%.!prev! | (
(for /l %j in (1 1 %max%) do pause)>nul& findstr "^"> %file%.%i)
FSUTIL file seteof %file%.!prev! %max% >nul
)
if not %last%==0 FSUTIL file seteof %file%.%num% %last% >nul
echo Done.
Tested on Win 10
im generating a CSV from an XLS file with VBA, after that I am filtering the CSV with Batch. My filter looks like this:
for %%a in (*.csv) do (
for /f "usebackq tokens=1-10 delims=, eol=^" %%1 in ("%%a") do (
if %%4 EQU Req_Category ECHO %%1,%%2,%%3,%%4,%%5,%%6,%%7,%%8,%%9 >> "%%a"_JIRA.csv
if %%4 EQU Requirement ECHO %%1,%%2,%%3,%%4,%%5,%%6,%%7,%%8,%%9 >> "%%a"_JIRA.csv
)
)
This works fine if the CSV File has no empty lines.
In rare occasions the XLS -> CSV converting generates empty lines or CRs in the CSV.
SW_Fn-289,4.1.1.1,Controling Hardware PCB,Heading,,,,,IgnoreTesting,
SW_Fn-291,4.1.1.1.0-1,"
Date : 07.03.1777
The SystemDesignSpecification is stored in SVN path
http://sblablablabla.xlsm
",Requirement,Lab1 (B-Sample),,Released,Accepted,IgnoreTesting,
SW_Fn-4281,4.1.1.1.0-2,"
Date : 123.123.123
Path : https://apath.com
",Requirement,R1,,New,New,IgnoreTesting,
SW_Fn-166,4.2,Compliance Requirements,Heading,,,,,IgnoreTesting,
SW_Fn-286,4.2.1,Resource Usage,Heading,,,,,IgnoreTesting,
Every line in the CSV should start with an ID: SW_Fn-Example.
Does every one have an idea how can bring the info on one line with a batch function?
I need to get the file to look like this (before filtering):
SW_Fn-289,4.1.1.1,Controling Hardware PCB,Heading,,,,,IgnoreTesting,
SW_Fn-291,4.1.1.1.0-1,"Date : 07.03.1777 TheSystemDesignSpecificationisstored in SVN path http://sblablablabla.xlsm",Requirement,Lab1 (B-Sample),,Released,Accepted,IgnoreTesting,
SW_Fn-4281,4.1.1.1.0-2," Date : 123.123.123 Path : https://apath.com",Requirement,R1,,New,New,IgnoreTesting,
SW_Fn-166,4.2,Compliance Requirements,Heading,,,,,IgnoreTesting,
SW_Fn-286,4.2.1,Resource Usage,Heading,,,,,IgnoreTesting,
There shouldnt be a line that does not start with SW_Fn-blabla. If a line starts with something else, then it should be a part of the previous line that has an Sw_Fn-blabla.
Then my filter will work to produce this:
SW_Fn-291,4.1.1.1.0-1,"Date : 07.03.1777 TheSystemDesignSpecificationisstored in SVN path http://sblablablabla.xlsm",Requirement,Lab1 (B-Sample),,Released,Accepted,IgnoreTesting,
SW_Fn-4281,4.1.1.1.0-2," Date : 123.123.123 Path : https://apath.com",Requirement,R1,,New,New,IgnoreTesting,
Thanks in advance
try this:
#echo off
for %%a in (*.csv) do (
for /f "delims=" %%b in (%%a) do (
for /f "tokens=4 delims=," %%c in ("%%b") do (
if "%%c"=="Requirement" echo %%b >>%%~na_JIRA%%~xa
if "%%c"=="Req_Category" echo %%b >>%%~na_JIRA%%~xa
)
)
)
read and handle each line complete to overcome the consecutive-delimiter-issue mentioned by Magoo (use another for to check Token4, but don't bother to disassemble and reassemble the complete line)
Aak! don't use numerics for the metavariable (%%1) - it's highly unreliable. Use an alphabetic character.
Batch treats a string of delimiters as a single delimiter and you have nominated commas and spaces as delimiters, so
SW_Fn-166,4.2,Compliance Requirements,Heading,,,,,IgnoreTesting,
would appear as
SW_Fn-166,4.2,Compliance,Requirements,Heading,IgnoreTesting,,,,
You haven't shown what you expect as output. Do you only want the lines that begin SW_Fn- or do you want all lines that don't start SW-Fn appended to the last line that did?
#ECHO Off
SETLOCAL
SET "sourcedir=U:\sourcedir"
SET "destdir=U:\destdir"
SET "filename1=%sourcedir%\q36475816.csv"
SET "outfile=%destdir%\outfile.txt"
SET "line="
(
FOR /f "usebackqdelims=" %%a IN ("%filename1%") DO (
SET "newpart=%%a"
IF DEFINED line CALL :test
IF DEFINED line CALL SET "line=%%line%% %%a"
IF NOT DEFINED line SET "line=%%a"
)
IF DEFINED line ECHO(%line%
)>"%outfile%"
GOTO :EOF
:: Test new data " Accumulate data into line or output & start a new line
:test
SET "newpart=%newpart:"=x%"
IF NOT "%newpart:~0,6%"=="SW_Fn-" goto :eof
echo(%line%
SET "line="
GOTO :eof
You would need to change the settings of sourcedir and destdir to suit your circumstances.
I used a file named q36475816.csv containing your data for my testing.
Produces the file defined as %outfile%
Note that your posted data contains unbalanced quotes in the Fn-4281 item. It's always better to use actual data rather than "somewhere close".
Read each line. If we've already accumulated part of a line, check whether the first few characters are the target. If they are, output the line as constructed and clear line.
If line is clear after this operation, set it to the line read (which must startwith the target, otherwise accumulate the line.
In the :test procedure, remove quotes before testing so that it doesn't break the syntax. Obviously, if the first few characters contains a quote, it doesn't fit the target so the test will correctly detect "no fit"
Your file is actually valid CSV format. Quoted CSV fields may contain any of the following:
comma
quote literal, escaped as ""
newline (either LF or CRLF)
You don't have commas or quotes within your fields, but you do have newlines that are giving your code serious problems.
But that is only one potential problem. Another issue is FOR /F treats consecutive delimiters as a single delimiter, so if any of your desired keeper lines have any empty fields, then your output will be completely wrong.
Batch is inherently far from ideal for any kind of text processing, but for CSV it is especially bad for all but the most simplest problems. If you really want to use batch, you could use ParseCSV.bat to properly parse your CSV and read it using FOR /F in a reliable manner. But there are better options.
PowerShell has an Import-Csv cmdlet. I'm not sure of its capabilities, but if it supports newlines within fields, then you could develop a really slick solution with that.
Another option is my JREPL.BAT regular expression text processor. The following code looks nasty, but it will very efficiently produce your desired output in one step:
jrepl "((?:[\s\S]*?,){3}(?:(Req_Category,|Requirement,)|.*?,)(?:.*?,){4}.*?),[^,\n]*\n?" "$2?$1.replace(/\r\n/g,' ')+'\r\n':''" /m /j /f input.csv /o output.csv
You would need to use CALL JREPL if you put the command within another batch script.
My JREPL solution relies on the fact that none of your input fields contain quoted commas. If it did contain quoted commas, then a JREPL solution would be even more complicated.
This solution works by using the /M multiline option so that I can match across line-breaks.
The search matches each 10 field collection (your 10th field seems to be always empty), regardless of line breaks. $1 contains the first 9 fields (without the trailing comma). $2 contains the 4th field if and only if it matches "Req_Category" or "Requirement". The replacement javascript expression tests if $2 is defined, and if it is, then the whole search expression is replaced with $1 after all newlines have been replaced by spaces, and then a newline is appended. IF $2 is not defined then the whole search expression is replaced with an empty string. Simple in concept, but kind of nasty to develop ;-)
A slight simplification allows you to preserve the original fields containing newlines, and still do the filtering you desire.:
jrepl "((?:[\s\S]*?,){3}(?:(Req_Category,|Requirement,)|.*?,)(?:.*?,){4}.*?),[^,\n]*\n?" "$2?$1+'\r\n':''" /m /j /f input.csv /o output.csv
For something so simple that can easily be done with find replace in notepad, I can't see why it is so hard to do in a command line as it is just one step in the entire procedure that I would like to get down to a single run. The output from the first part lists the local path to all the websites in the webserver as the each local path c:/ etc. every site has the same 59 characters before the part that matters.
To make this a usable link, I need to then add a different string in the same position as the old one with the correct http://. etc. to the balance of the line to make it a working hyperlink.
The final step needs to convert any single "\’s" that are left to a "/". Normally there is only one
All of this can be done in notepad++ using find and replace but it takes 3 runs to achieve the end result the original text file is nothing special, no skipped lines, everyone is identical in layout.
The same 59 characters need to be chopped off (it could even be by Number and not by comparing the text, just shorten by 59 characters if that is easier. The replacement text string is always exactly the same that just gets appended to each line. And for the final touch of replacing every \ with a / to make it fully web-compatible there is only one occurrence on each line.
I have seen many find and replace batch-files that seem to be overkill for such a simple task.
Take each line, count fifty nine characters forward, chop off the 59 and add in the replacement text in its place.
Then change the only backslash in the line to a forward slash and it’s done
Does anyone know a simpler easier way to do this
This uses a helper batch file called repl.bat - download from: https://www.dropbox.com/s/qidqwztmetbvklt/repl.bat
Place repl.bat in the same folder as the batch file or in a folder that is on the path.
Just change http://www.domain.com/ to what you need to prefix the lines with.
type "file.txt" | repl "\\" "/" | repl "^.{59}" "http://www.domain.com/" >"newfile.txt"
The two \\ are intentional as it is a regular expression.
#ECHO OFF
SETLOCAL
(
FOR /f "delims=" %%a IN (Q21495128.txt) DO (
SET "line=%%a"
CALL SET "line=%%line:\=/%%"
CALL SET "line=replacement text%%line:~59%%"
FOR /f "tokens=1*delims==" %%x IN ('set line') DO ECHO %%y
)
)>newfile.txt
GOTO :EOF
where Q21495128.txt was my test source file worked for me.
I have .txt files (1 or more) in a directory that I want my batch file to read their filename, get 20 characters starting from the 4th and creating a new directory in the folder. Here is my code:
for /f %%i in ('dir /b *.TXT') do (
set filename1=%%i
set folder1=%filename1:~4,20%
mkdir %folder1%
)
When I run this program the 1st time, I get a syntax not correct error for the line 3rd line (set folder1=....) and no folder is created. I tried running it a 2nd time and 2 folders were created (one named "~4" and one named "20"). On the 3rd run, the folder was correctly created! If I close the command prompt and open it again, it also needs to run 3 times before it creates the folder.
I've also tried using "for /r . $$i in (*.TXT) with no luck. As I understand, the problem is that the line with "set folder1=..." does not get the proper filename. I've also tried using %%~i or %%~ni, I've tried outputing the filename (which seems to always get the correct string) to a text file and then reading from that file, again with no luck. I don't know what else to try. Is it because %%i stores the file itself and not a string with the filename?
A sample file is named "REG_18004247K_20120208_A.TXT" and I want a folder to be created with the name "18004247K_20120208_A".
I am not at all familiar with batch programming (I'm only working with batch files for like 2 weeks) and I'm guessing the problem might be something really simple. Any help would be appreciated.
Kyriacos
%%i does stores a string with a filename. It is actually the only "variable" that works as expected here.
The key problem is that you are using environment variable substitution (of filename1 and of folder1) inside a loop, hoping that it will be expanded in each iteration of the loop.
However, environment variables are expanded before execution of the whole loop begins.
So, upon the first batch execution, filename1 is not defined and you get
an error, folder1 is not set, and you probably also see a folder
called %folder1% created.
However, the scripts defines filename1 at this time, and the second execution gets further to define folder1 correctly (although line 4 was
already expanded using the incorrect value and fun happens).
The third execution
finally gets to see the right value of folder1; but this would obviously not
work for more than one file, as the loop logic is dysfunctional.
This older answer explains this problem with great other examples and special cases.
Your batch can be fixed by setlocal enabledelayedexpansion at the beginning of the script, initializing the variables to empty strings before the loop, and switching to the !...! syntax when expanding them: !filename1:~4,20!, and !folder1!.
setlocal enabledelayedexpansion
for /r %%i in (*) do (
set filename1=%%i
set folder1=!filename1:~4,20!
mkdir !folder1!
)
Because ... delayed expansion takes some time to store actual variable.
!...! rather than %...% echoes fast allocating variables to DOS.