Unexpected results while replacing single characters in a text file - string

Question

My batch file:
#ECHO off
(FOR /f "delims=" %%i in (source.txt) DO (
SET "line=%%i"
setlocal enabledelayedexpansion
SET "line=!line:Ć=F!"
SET "line=!line:Ç=G!"
SET "line=!line:Ň=R!"
SET "line=!line:Ô=T!"
ECHO.!line!
endlocal
))>"output.txt"
My source.txt file:
ĆÇŇÔ
Expected output.txt file:
FGRT
Current output.txt file:
FFRR
My question is: what's wrong here?

Answer 1

If source.txt is not saved as Unicode, your issue may be related to the codepage at the time you run your loop.
The following example switches to codepage 1252, West European Latin, (as also suggested in the comments by Gerhard), if not that already. Although I'd assume codepage 850, Multilingual (Latin I) should work equally well. (Just change to the codepage required by replacing 1252 on lines 7, and 8, as necessary).
#Echo Off
SetLocal EnableExtensions DisableDelayedExpansion
If Not Exist "source.txt" GoTo :EOF
For /F "Delims==" %%G In ('2^> NUL Set _cp') Do Set "%%G="
For /F Tokens^=* %%G In ('"%SystemRoot%\System32\chcp.com"'
) Do For %%H In (%%G) Do Set "_cp=%%~nH"
If Not %_cp% Equ 1252 (Set "_cpc=TRUE"
"%SystemRoot%\System32\chcp.com" 1252 1> NUL)
(For /F UseBackQ^ Delims^=^ EOL^= %%G In ("source.txt") Do (
Set "line=%%G"
SetLocal EnableDelayedExpansion
Set "line=!line:Ć=F!"
Set "line=!line:Ç=G!"
Set "line=!line:Ň=R!"
Set "line=!line:Ô=T!"
Echo=!line!
EndLocal)) 1> "output.txt"
If Defined _cpc "%SystemRoot%\System32\chcp.com" %_cp% 1> NUL
Please note that using a For loop like this, will remove any blank lines from the output

Answer 2

The answer (as suggested by #Gerhard and #Compo): it was the wrong code page.
Below is my current working batch code if someone else will ever be in the same need (to convert inverted ATASCII characters in ATARI BASIC code).
It converts defined set of characters (you can add more / delete some - just modify strings and the total number of characters) and make comments more visible by adding lines at the start and at the end of each one.
#ECHO off
rem --------------------------------------------------
rem CHECK FOR THE SOURCE FILE
rem --------------------------------------------------
IF "%~1"=="" GOTO :End
rem --------------------------------------------------
rem SET THE CODE PAGE
rem --------------------------------------------------
CHCP 1252 > NUL
rem --------------------------------------------------
rem DEFINE THE SET OF CHARACTERS TO CONVERT
rem --------------------------------------------------
SET "input_set_of_chars= Ł¤§¨cŞ«¬­R°+˛ł´u¸ą»Ľ˝ľÁÂĂÄĹĆÇČÉĘËĚÍÎĎĐŃŇÓÔŐÖ×ŘŮÚř"
SET "output_set_of_chars= #$'()*+,-.0123456789;<=>ABCDEFGHIJKLMNOPQRSTUVWXYZx"
SET "number_of_chars=52"
rem --------------------------------------------------
rem CONVERT EACH LINE OF THE SOURCE FILE
rem --------------------------------------------------
(FOR /f "delims=" %%i in (%~1) DO (
SET "line=%%i"
CALL :ConvertASCII
))> "%~n1-converted%~x1"
GOTO :End
rem --------------------------------------------------
rem START OF THE CONVERT SUBROUTINE
rem --------------------------------------------------
:ConvertASCII
SETLOCAL enableDelayedExpansion
rem --------------------------------------------------
rem IF THE LINE IS NOT A COMMENT THEN DO NOT CONVERT
rem --------------------------------------------------
IF "!line!"=="!line: REM =!" GOTO :LoopEnd
rem --------------------------------------------------
rem MAKE COMMENT LINE A LITTLE MORE VISIBLE
rem --------------------------------------------------
SET "line=!line: REM = REM ----------!----------"
rem --------------------------------------------------
rem CONVERT ALL DEFINED CHARACTERS
rem --------------------------------------------------
SET "counter=0"
:LoopStart
SET "input_char=!input_set_of_chars:~%counter%,1!"
SET "output_char=!output_set_of_chars:~%counter%,1!"
SET "line=!line:%input_char%=%output_char%!"
SET /a counter+=1
IF "!counter!"=="!number_of_chars!" GOTO :LoopEnd
GOTO :LoopStart
rem --------------------------------------------------
rem ECHO THE CURRENT LINE AND EXIT SUBROUTINE
rem --------------------------------------------------
:LoopEnd
ECHO.!line!
ENDLOCAL
EXIT /b 0
:End

Answer 3

As per your comment, it is actually not really a code page issue, because you have got ATASCII encoding in your *.bas files. For converting such files to avoid inverted glyphs, I would use a language that can easily read the file in binary mode and subtract 0x80 from every byte whose value is greater than or equal to 0x80.
Anyway, if you do want to replace the characters left over from your already performed conversion process (Ć, Ç, Ň, Ô, with codes 0x8F, 0x80, 0xD5, 0xE2, resp., as per your active code page 852), I would do it the following way, applying code page 437 during any conversion activities, because this defines the character set of the original IBM PC, also known as OEM font, where there should not occur any unwanted character conversion in the background:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
rem // Define constants here:
set "_ROOT=%~dp0." & rem // (full path to target directory)
set "_SOURCE=source.txt" & rem // (name of source file)
set "_RETURN=return.txt" & rem // (name of return file)
set "_FILTER=^[0-9][0-9]* *REM " & rem /* (`findstr` search expression to filter
rem for specific lines; `^` means all) */
rem // Store current code page:
for /F "tokens=2 delims=:" %%P in ('chcp') do for /F %%C in ("%%P") do set "$CP=%%C"
rem // Set code page to OEM in order to avoid unwanted character conversions:
> nul chcp 437
rem /* Specify character replacements; the `forfiles` command supports substitution
rem of hex codes like `0xHH`, so you can specify special characters by their code
rem in order to avoid having to embed them into this script, which might in turn
rem lead to problems due to dependencies on the current code page; each `0x22`
rem represents a `"` to enclose each replacement expression within quotes; each
rem of the following replacement expression is `"` + char. + `=` + char. +`"`: */
for /F "delims=" %%R in ('
forfiles /P "%~dp0." /M "%~nx0" /C ^
"cmd /C echo 0x220x8F=F0x22 0x220x80=G0x22 0x220xD5=R0x22 0x220xE2=T0x22"
') do set "RPL=%%R"
rem // Change into target directory:
pushd "%_ROOT%" && (
rem // Write into return file:
> "%_RETURN%" (
rem /* Read from source file, temporarily precede each line with line number
rem followed by a `:` in order to not lose blank lines: */
for /F "delims=" %%L in ('findstr /N "^" "%_SOURCE%"') do (
rem // Store current line:
set "LINE=%%L"
rem // Toggle delayed expansion to avoid troubles with `!`:
setlocal EnableDelayedExpansion
rem // Remove temporary line number prefix:
set "LINE=!LINE:*:=!"
rem // Filter for lines that are subject to the replacements:
cmd /V /C echo(^^!LINE^^!| > nul findstr /R /I /C:"!_FILTER!" && (
rem // Perform replacements one after another:
for %%R in (%RPL%) do if defined LINE set "LINE=!LINE:%%~R!"
)
rem // Return resulting line:
echo(!LINE!
endlocal
)
)
rem // Return from target directory:
popd
)
rem // Restore former code page:
if defined $CP > nul chcp %$CP%
endlocal
exit /B
This approach performs the character replacements only in lines that begin with: a decimal number, followed by one or more SPACEs, followed by REM in a case-insensitive manner, followed by a SPACE.
Here is now a script that truly converts ATASCII characters in REM comments in your Atari Basic (*.bas) file, using certutil for converting the binary character codes:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
rem // Define constants here:
set "_TARGET=%~dp0." & rem // (full path to target directory)
set "_SOURCE=source.txt" & rem // (name of source file)
set "_RETURN=return.txt" & rem // (name of return file)
set "_FILTER=^[0-9][0-9]* *REM " & rem /* (`findstr` search expression to filter
rem for specific lines; `^` means all) */
set "_TEMPFN=%TEMP%\%~n0_%RANDOM%" & rem // (path and base name of temporary file)
rem // Change into target directory:
pushd "%_TARGET%" && (
rem // Write into return file:
> "%_RETURN%" (
rem /* Read from source file, temporarily precede each line with line number
rem followed by a `:` in order to not lose blank lines: */
for /F "delims=" %%L in ('findstr /N "^" "%_SOURCE%"') do (
rem // Store current line:
set "LINE=%%L"
rem // Toggle delayed expansion to avoid troubles with `!`:
setlocal EnableDelayedExpansion
rem // Remove temporary line number prefix:
set "LINE=!LINE:*:=!"
rem // Filter for lines that are subject to the replacements:
cmd /V /C echo(^^!LINE^^!| > nul findstr /R /I /C:"!_FILTER!" && (
rem // Found a line, hence write it to temporary file:
(> "!_TEMPFN!.tmp" echo(!LINE!) && (
rem // Convert temporary file to hex dump file:
> nul certutil -f -encodehex "!_TEMPFN!.tmp" "!_TEMPFN!.cnv" 4 && (
rem // Write to temporary file:
(> "!_TEMPFN!.tmp" (
rem // Read hex dump file line by line:
for /F "usebackq tokens=*" %%T in ("!_TEMPFN!.cnv") do (
rem // Reset buffer, loop through hex values:
set "BUFF= " & for %%H in (%%T) do (
rem // Determine new hex value, append it to buffer:
set "HEX=%%H" & set /A "FIG=0x!HEX:~,1!-0x8"
if !FIG! lss 0 (
rem // Value was < 0x80, hence keep it:
set "BUFF=!BUFF! !HEX!"
) else (
rem // Value was >= 0x80, hence subtract 0x80:
set "BUFF=!BUFF! !FIG!!HEX:~1!"
)
)
echo(!BUFF:~2!
)
)) && (
> nul certutil -f -decodehex "!_TEMPFN!.tmp" "!_TEMPFN!.cnv" 4 && (
type "!_TEMPFN!.cnv"
) || echo(!LINE!
) || echo(!LINE!
) || echo(!LINE!
) || echo(!LINE!
) || (
rem // Return resulting line:
echo(!LINE!
)
endlocal
)
)
rem // Clean up temporary files:
del "%_TEMPFN%.tmp" "%_TEMPFN%.cnv"
rem // Return from target directory:
popd
)
endlocal
exit /B