I recently found the post Find if substring is in string (not in a file) where it is stated that considering
#setlocal enableextensions enabledelayedexpansion
#echo off
set str1=%1
if not x%str1:bcd=%==x%str1% echo It contains bcd
endlocal
then
the x before the two sides of the equality is to ensure that the string bcd works okay. It also protects against certain "improper" starting characters.
However, I haven't found any explanation about the actual effect of this x. So what is the difference between x"%string%" and "%string%"?
That is simply a very bad coded string comparison. The x on both sides makes it possible to compare the two strings even if %str1:bcd=% or %str1% is substituted by Windows command processor on parsing entire command line by an empty string before execution of command IF.
But the batch file execution is nevertheless exited immediately by cmd.exe because of a syntax error in case of value of environment variable str1 contains a space character or "&<>|.
Enclosing an argument string in double quotes results in getting all characters except percent sign and with enabled delayed environment variable expansion also the exclamation mark interpreted as literal character including space which is outside a double quoted string interpreted as argument string separator.
So much better is:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
if "%~1" == "" goto EndBatch
set "str1=%~1"
if not "%str1:bcd=%" == "%str1%" echo It contains bcd
:EndBatch
endlocal
The first argument of the batch file is compared first without surrounding double quotes with an empty string. So if the batch file is started without any argument or with just "" as first argument string, Windows command processor executes the command GOTO resulting in restoring previous environment pushed on stack with command SETLOCAL and exits the batch file.
Otherwise the batch file is called really with an argument string. This argument string is assigned to environment variable str1 with removing surrounding double quotes if there are one. So on calling batch file with argument test the value test is assigned to environment variable str1 and on calling it with "another test" the value another test without the double quotes is assigned to str1. And even on calling the batch file with wrong coded argument string "bcd test (missing second ") just bcd test is assigned to the environment variable str1.
The IF condition compares the value of environment variable str1 with all occurrences of bcd removed with the unmodified variable value. The double quotes around the two strings make it possible to compare the two strings even on containing space or ampersand or the redirection operators <>|. The command IF includes the double quotes on comparing the two strings.
So is this code now safe?
No, it is not in case of somebody calls the batch file invalid with test_bcd" as argument string on which first double quote is missing. In this case the first IF command line executed by cmd.exe is:
if "test_bcd"" == "" goto EndBatch
The trailing " of the wrong specified argument string is not removed by cmd.exe and cause a syntax error on this command line on execution as it can be seen on running the batch file from within a command prompt window with first line modified to #echo on.
One solution without using delayed environment variable expansion is:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "str1=%~1"
if not defined str1 goto EndBatch
set "str1=%str1:"=%"
if not defined str1 goto EndBatch
if not "%str1:bcd=%" == "%str1%" echo It contains bcd
:EndBatch
endlocal
This code makes sure that str1 does not contain any double quote before executing the IF command comparing the strings.
Another solution is using delayed environment variable expansion:
#echo off
setlocal EnableExtensions EnableDelayedExpansion
set "str1=%~1"
if not "!str1:bcd=!" == "!str1!" echo It contains bcd
endlocal
That looks better as the above code without usage of delayed environment variable expansion. But it does not work as expected if the the argument string is for example "!Hello!" because in this case the if not condition is also true and output is therefore the message It contains bcd although the string !Hello! does not contain bcd.
The solution is:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "str1=%~1"
setlocal EnableDelayedExpansion
if not "!str1:bcd=!" == "!str1!" echo It contains bcd
endlocal
endlocal
Delayed expansion is not enabled on assigning the argument string to environment variable str1 which results in getting the exclamation marks in string "!Hello!" interpreted as literal characters. Then delayed expansion is enabled for making the string comparison with using delayed environment variable expansion which avoids that the command line itself is modified by cmd.exe before execution of IF command.
For understanding the used commands and how they work, open a command prompt window, execute there the following commands, and read entirely all help pages displayed for each command very carefully.
call /? ... explains %~1, not so good as done here.
echo /?
endlocal /?
goto /?
if /?
set /?
setlocal /?
See also:
How does the Windows Command Interpreter (CMD.EXE) parse scripts? ... a long answer every batch file writer should read carefully from top to bottom.
forfiles - FALSE vs. false (doesn't work with lower case?) ... this answer is about argument processing of command IF.
Symbol equivalent to NEQ, LSS, GTR, etc. in Windows batch files ... this answer explains in detail how string comparisons are done by command IF.
Why is no string output with 'echo %var%' after using 'set var = text' on command line? ... this answer explains why set "variable=value" should be used in general and not other variants.
Single line with multiple commands using Windows batch file ... explains how & and || outside a double quoted argument string are interpreted by cmd.exe.
Microsoft article about Using command redirection operators explains how <>|& outside a double quoted argument string are interpreted by cmd.exe.
The addition of x (or any other alphabetic character) in front of a string ensures that the relational statement is syntactically valid even when/if the string is empty.
Suppose str1 is an empty string. Then the comparison %str1:bcd=%==%str1% after the substitution degenerates to ==, which is syntactically invalid.
However, with an x in front, the comparison becomes x==x and can be evaluated. Naturally, adding the same prefix to each of the two strings does not affect their (in)equality.
im generating a CSV from an XLS file with VBA, after that I am filtering the CSV with Batch. My filter looks like this:
for %%a in (*.csv) do (
for /f "usebackq tokens=1-10 delims=, eol=^" %%1 in ("%%a") do (
if %%4 EQU Req_Category ECHO %%1,%%2,%%3,%%4,%%5,%%6,%%7,%%8,%%9 >> "%%a"_JIRA.csv
if %%4 EQU Requirement ECHO %%1,%%2,%%3,%%4,%%5,%%6,%%7,%%8,%%9 >> "%%a"_JIRA.csv
)
)
This works fine if the CSV File has no empty lines.
In rare occasions the XLS -> CSV converting generates empty lines or CRs in the CSV.
SW_Fn-289,4.1.1.1,Controling Hardware PCB,Heading,,,,,IgnoreTesting,
SW_Fn-291,4.1.1.1.0-1,"
Date : 07.03.1777
The SystemDesignSpecification is stored in SVN path
http://sblablablabla.xlsm
",Requirement,Lab1 (B-Sample),,Released,Accepted,IgnoreTesting,
SW_Fn-4281,4.1.1.1.0-2,"
Date : 123.123.123
Path : https://apath.com
",Requirement,R1,,New,New,IgnoreTesting,
SW_Fn-166,4.2,Compliance Requirements,Heading,,,,,IgnoreTesting,
SW_Fn-286,4.2.1,Resource Usage,Heading,,,,,IgnoreTesting,
Every line in the CSV should start with an ID: SW_Fn-Example.
Does every one have an idea how can bring the info on one line with a batch function?
I need to get the file to look like this (before filtering):
SW_Fn-289,4.1.1.1,Controling Hardware PCB,Heading,,,,,IgnoreTesting,
SW_Fn-291,4.1.1.1.0-1,"Date : 07.03.1777 TheSystemDesignSpecificationisstored in SVN path http://sblablablabla.xlsm",Requirement,Lab1 (B-Sample),,Released,Accepted,IgnoreTesting,
SW_Fn-4281,4.1.1.1.0-2," Date : 123.123.123 Path : https://apath.com",Requirement,R1,,New,New,IgnoreTesting,
SW_Fn-166,4.2,Compliance Requirements,Heading,,,,,IgnoreTesting,
SW_Fn-286,4.2.1,Resource Usage,Heading,,,,,IgnoreTesting,
There shouldnt be a line that does not start with SW_Fn-blabla. If a line starts with something else, then it should be a part of the previous line that has an Sw_Fn-blabla.
Then my filter will work to produce this:
SW_Fn-291,4.1.1.1.0-1,"Date : 07.03.1777 TheSystemDesignSpecificationisstored in SVN path http://sblablablabla.xlsm",Requirement,Lab1 (B-Sample),,Released,Accepted,IgnoreTesting,
SW_Fn-4281,4.1.1.1.0-2," Date : 123.123.123 Path : https://apath.com",Requirement,R1,,New,New,IgnoreTesting,
Thanks in advance
try this:
#echo off
for %%a in (*.csv) do (
for /f "delims=" %%b in (%%a) do (
for /f "tokens=4 delims=," %%c in ("%%b") do (
if "%%c"=="Requirement" echo %%b >>%%~na_JIRA%%~xa
if "%%c"=="Req_Category" echo %%b >>%%~na_JIRA%%~xa
)
)
)
read and handle each line complete to overcome the consecutive-delimiter-issue mentioned by Magoo (use another for to check Token4, but don't bother to disassemble and reassemble the complete line)
Aak! don't use numerics for the metavariable (%%1) - it's highly unreliable. Use an alphabetic character.
Batch treats a string of delimiters as a single delimiter and you have nominated commas and spaces as delimiters, so
SW_Fn-166,4.2,Compliance Requirements,Heading,,,,,IgnoreTesting,
would appear as
SW_Fn-166,4.2,Compliance,Requirements,Heading,IgnoreTesting,,,,
You haven't shown what you expect as output. Do you only want the lines that begin SW_Fn- or do you want all lines that don't start SW-Fn appended to the last line that did?
#ECHO Off
SETLOCAL
SET "sourcedir=U:\sourcedir"
SET "destdir=U:\destdir"
SET "filename1=%sourcedir%\q36475816.csv"
SET "outfile=%destdir%\outfile.txt"
SET "line="
(
FOR /f "usebackqdelims=" %%a IN ("%filename1%") DO (
SET "newpart=%%a"
IF DEFINED line CALL :test
IF DEFINED line CALL SET "line=%%line%% %%a"
IF NOT DEFINED line SET "line=%%a"
)
IF DEFINED line ECHO(%line%
)>"%outfile%"
GOTO :EOF
:: Test new data " Accumulate data into line or output & start a new line
:test
SET "newpart=%newpart:"=x%"
IF NOT "%newpart:~0,6%"=="SW_Fn-" goto :eof
echo(%line%
SET "line="
GOTO :eof
You would need to change the settings of sourcedir and destdir to suit your circumstances.
I used a file named q36475816.csv containing your data for my testing.
Produces the file defined as %outfile%
Note that your posted data contains unbalanced quotes in the Fn-4281 item. It's always better to use actual data rather than "somewhere close".
Read each line. If we've already accumulated part of a line, check whether the first few characters are the target. If they are, output the line as constructed and clear line.
If line is clear after this operation, set it to the line read (which must startwith the target, otherwise accumulate the line.
In the :test procedure, remove quotes before testing so that it doesn't break the syntax. Obviously, if the first few characters contains a quote, it doesn't fit the target so the test will correctly detect "no fit"
Your file is actually valid CSV format. Quoted CSV fields may contain any of the following:
comma
quote literal, escaped as ""
newline (either LF or CRLF)
You don't have commas or quotes within your fields, but you do have newlines that are giving your code serious problems.
But that is only one potential problem. Another issue is FOR /F treats consecutive delimiters as a single delimiter, so if any of your desired keeper lines have any empty fields, then your output will be completely wrong.
Batch is inherently far from ideal for any kind of text processing, but for CSV it is especially bad for all but the most simplest problems. If you really want to use batch, you could use ParseCSV.bat to properly parse your CSV and read it using FOR /F in a reliable manner. But there are better options.
PowerShell has an Import-Csv cmdlet. I'm not sure of its capabilities, but if it supports newlines within fields, then you could develop a really slick solution with that.
Another option is my JREPL.BAT regular expression text processor. The following code looks nasty, but it will very efficiently produce your desired output in one step:
jrepl "((?:[\s\S]*?,){3}(?:(Req_Category,|Requirement,)|.*?,)(?:.*?,){4}.*?),[^,\n]*\n?" "$2?$1.replace(/\r\n/g,' ')+'\r\n':''" /m /j /f input.csv /o output.csv
You would need to use CALL JREPL if you put the command within another batch script.
My JREPL solution relies on the fact that none of your input fields contain quoted commas. If it did contain quoted commas, then a JREPL solution would be even more complicated.
This solution works by using the /M multiline option so that I can match across line-breaks.
The search matches each 10 field collection (your 10th field seems to be always empty), regardless of line breaks. $1 contains the first 9 fields (without the trailing comma). $2 contains the 4th field if and only if it matches "Req_Category" or "Requirement". The replacement javascript expression tests if $2 is defined, and if it is, then the whole search expression is replaced with $1 after all newlines have been replaced by spaces, and then a newline is appended. IF $2 is not defined then the whole search expression is replaced with an empty string. Simple in concept, but kind of nasty to develop ;-)
A slight simplification allows you to preserve the original fields containing newlines, and still do the filtering you desire.:
jrepl "((?:[\s\S]*?,){3}(?:(Req_Category,|Requirement,)|.*?,)(?:.*?,){4}.*?),[^,\n]*\n?" "$2?$1+'\r\n':''" /m /j /f input.csv /o output.csv
For something so simple that can easily be done with find replace in notepad, I can't see why it is so hard to do in a command line as it is just one step in the entire procedure that I would like to get down to a single run. The output from the first part lists the local path to all the websites in the webserver as the each local path c:/ etc. every site has the same 59 characters before the part that matters.
To make this a usable link, I need to then add a different string in the same position as the old one with the correct http://. etc. to the balance of the line to make it a working hyperlink.
The final step needs to convert any single "\’s" that are left to a "/". Normally there is only one
All of this can be done in notepad++ using find and replace but it takes 3 runs to achieve the end result the original text file is nothing special, no skipped lines, everyone is identical in layout.
The same 59 characters need to be chopped off (it could even be by Number and not by comparing the text, just shorten by 59 characters if that is easier. The replacement text string is always exactly the same that just gets appended to each line. And for the final touch of replacing every \ with a / to make it fully web-compatible there is only one occurrence on each line.
I have seen many find and replace batch-files that seem to be overkill for such a simple task.
Take each line, count fifty nine characters forward, chop off the 59 and add in the replacement text in its place.
Then change the only backslash in the line to a forward slash and it’s done
Does anyone know a simpler easier way to do this
This uses a helper batch file called repl.bat - download from: https://www.dropbox.com/s/qidqwztmetbvklt/repl.bat
Place repl.bat in the same folder as the batch file or in a folder that is on the path.
Just change http://www.domain.com/ to what you need to prefix the lines with.
type "file.txt" | repl "\\" "/" | repl "^.{59}" "http://www.domain.com/" >"newfile.txt"
The two \\ are intentional as it is a regular expression.
#ECHO OFF
SETLOCAL
(
FOR /f "delims=" %%a IN (Q21495128.txt) DO (
SET "line=%%a"
CALL SET "line=%%line:\=/%%"
CALL SET "line=replacement text%%line:~59%%"
FOR /f "tokens=1*delims==" %%x IN ('set line') DO ECHO %%y
)
)>newfile.txt
GOTO :EOF
where Q21495128.txt was my test source file worked for me.
I am using Install Anywhere 2012 and would like to be able to parse a batch or shell script for a give value and have that value stored in an IA variable. For instance, if I have the following shell file:
MY_VAR1=123
MY_VAR2=a\b\c
ECHO $MY_VAR1
I would like to pass in the path to the file and the variable name (ex. MY_VAR1) and have the result, 123, stored in an IA variable of my choosing (lets say $OUTPUT$). I could achieve this through writing some java custom code but was wondering if there was an alternative approach built into IA that would make this much easier. The variable will not be initialized when I need to figure out its value so essentially just echoing it's value or something similar will not work. Any help would be greatly appreciated!
example in Windows batch:
#echo off &setlocal
for /f "tokens=2delims==" %%a in ('findstr "MY_VAR1" "ShellFile"') do set "output=%%a"
if defined output (echo MY_VAR1: %output%) else echo MY_VAR1 not found!
On Linux/Unix, you could use perl or awk (both are standard utilities in most distros). Python or Ruby are also candidates, but may not be installed on your target system. You could even write your own targeted parser using Lex and Yacc and ship it with your installer. However, for your needs, that's surely overkill.
Here's an example of a possible awk solution in an Execute Script/Batch File Action:
#!/bin/bash
awk '
# Process lines that begin with our variable name,
# preceded by optional spaces or tabs.
/^[ \t]*$TARGET_VARIABLE_NAME$=.+/ {
# Split the current line on "=" into
# an array called result
split($0, result, "=")
value = result[1]
# Look for trailing comments and remove them.
offset = index(value, "#")
if (offset > 0) {
value = substr(value, 1, offset - 1)
}
# Remove any possible leading spaces and quotes.
# Note that the single-quote is escaped. That escape
# is for bash, not for awk. I am doing this from
# memory and do not have access to IA right now.
# you may have to play with the escaping.
gsub(/^[\'" ]*/, "", value)
# Remove any possible trailing spaces and quotes.
# See above regarding the escaped single-quote.
gsub(/[\'" ]*$/, "", value)
# send "value" to stdout
print value
}
' < $SHELL_INPUT_FILE$
The print value line (near the end) sends value to stdout.
In the Execute Script/Batch File Action settings you can designate variables that receive the stdout and stderr streams produced by the script action. By default, the stdout stream is stored in $EXECUTE_STDOUT$. You can change this to a variable name of your choosing.
In the example, above, $TARGET_VARIABLE_NAME$ and $SHELL_INPUT_FILE$ are InstallAnywhere variables that hold the name of the variable to find and the name of the file to parse, respectively. These variables will be replaced by their values before the Action executes.
Assume we have a script called /home/fred/hello.sh, which contains the following code:
#!/bin/bash
WIFE='Wilma'
NEIGHBOR="Barney Rubble"
echo "Hello to $WIFE and $NEIGHBOR from $PWD"
Before the Execute Script/Batch File Action runs, stuff the name of the script file into $SHELL_INPUT_FILE$ (/home/fred/hello.sh). Then set the value of $TARGET_VARIABLE_NAME$ to the variable you wish to find (say, NEIGHBOR). After the action completes, $EXECUTE_STDOUT$ in InstallAnywhere will contain Barney Rubble.
You can build on this idea to parse arbitrarily complex files in an Execute Script/Batch File Action. Just make your awk (or perl/Ruby/Python) script as complex as needed.
NOTE: when scripting Unix shell scripts in InstallAnywhere ALWAYS check the "Do not replace unknown variables" option. If you don't, InstallAnywhere will quietly convert anything that looks vaguely like an InstallAnywhere variable into blanks... It's very annoying.
For a Windows solution, find a standalone Windows version of awk or perl and include it with your installation. Then extend the above solution to work for batch files.
You'd want to create two Execute Script/Batch File Actions, one with a rule for Linux/Unix and one with a rule for Windows. You'd have to install the Windows awk or perl executable before calling this action though. Also, you'd need to fully qualify the path to the awk/perl executable. Finally, the actual script will need to be sensitive to differences in batch syntax versus shell syntax.
Below is an awk script modified to look for batch variable definitions. The pattern changes and you won't have to worry about embedded comments:
$PATH_TO_AWK_EXE$ '
# This pattern looks for optional spaces, the word SET
# with any capitalization, the target variable, more
# optional spaces and the equals sign.
/^[ \t]*[Ss][Ee][Tt][ \t]*$TARGET_VARIABLE_NAME$[ \t]*=.+/ {
# Split the current line on "=" into
# an array called result
split($0, result, "=")
value = result[1]
# No trailing comments in Batch files.
# Remove any possible leading spaces and quotes.
# Note that the single-quote is escaped. That escape
# is for bash, not for awk. I am doing this from
# memory and do not have access to IA right now.
# you may have to play with the escaping.
gsub(/^[\'" ]*/, "", value)
# Remove any possible trailing spaces and quotes.
# See above regarding the escaped single-quote.
gsub(/[\'" ]*$/, "", value)
# send "value" to stdout
print value
}
' < $SHELL_INPUT_FILE$
Above, the IA variable $PATH_TO_AWK_EXE$ points to the location where awk was installed. It would be set as some combination of $USER_INSTALL_FOLDER$, possibly other directory names, and the name of the awk.exe file. $PATH_TO_AWK_EXE$ can later be used to remove the awk executable, if desired.
You can try to get variable from output of script
"Execute Script" -> Store process's stdout in: $EXECUTE_OUTPUT$
Than you can use $EXECUTE_OUTPUT$ as variable after that