I am working my way through Learn You a Haskell for great good. I am currently on the files and streams section of Chapter 9. For some reason, when I try to pipe code into one of the example Haskell programs, I do not get the same output as the book. Using ConEmu for Linux commands on Windows. For example, I have the program that only prints out strings that are less than 10 characters with the code below (short_lines.hs):
main = interact $ unlines . filter ((<10) . length) . lines
I am going to be passing this file (short_long.txt):
i'm short
so am i
i am a loooooooooong line!!!
yeah i'm long so what hahahaha!!!!!!
short line
loooooooooooooooooooooooooooong
short
Here is the command:
cat short_long.txt | runhaskell short_lines.hs
Here is my output:
so am i
short
The book says that the output is the following:
i'm short
so am i
short
I believe this has to do with the handling of the newline character but I can't figure this out since lines should have removed the newline characters before filtering. It works with manual input but not with piping. Why am I getting a different output? Am I doing something wrong? I tried removing trailing newline characters in Atom editor but it didn't change anything. Any help on why I am not getting the expected result and what I could do to get the expected result would be greatly appreciated. Thank you!
The default newline mode for stdin is nativeNewline, which chooses its behavior based on what it believes your OS to be. I suspect that it has (wrongly) decided you are on a Unix system and it therefore should not do CRLF conversion; thus when given a Windows-style file each line has a trailing '\r' character. Try using
import System.IO
main = do
hSetNewlineMode stdin universalNewlineMode
interact $ unlines . filter ((<10) . length) . lines
to force CRLF conversion and see if that gets you the expected results.
I can reproduce your problem on my Unix system by converting a text file to DOS mode before giving it to your program. Having done so, my suggested fix gets the desired behavior.
I found out that I can change the line ending style from Windows-CRLF to Unix-LF on Atom editor. Currently it is located on the bottom and simply says CRLF or LF. You can click on it to choose a different line style. For this book, that is what I will use for simplicity's sake. However, I believe that amalloy's answer is a better long-term universal approach to IO.
This question already has answers here:
Can I programmatically "burn in" ANSI control codes to a file using unix utils?
(2 answers)
Closed 6 years ago.
I have an output file from a testing script (which I cannot alter), the output looks great in the terminal thanks to the encoding, which displays the output in nice colours.
However when I vim the file, I get the following:
^[[1m0024^[[0m, ^[[36munknown.10^[[0m --> ^[[32mUNKNOWN^[[0m
I would rather the file contained:
0024, unknown.10 --> UNKNOWN
There are a couple of similar questions on stackover flow, but so far I have not found a solution that works for me.
Any help would be greatly appreciated!
Many thanks!
Additional info:
I don't want to conceal the colour characters, I would like to remove them from the file.
The output goes into an evidence file, and then that file is pushed up to a GIT for the team to review. It is difficult to the GIT UI with those colour codes :(
To remove color control character, you may use the following sed command:
sed 's/\x1b\[[^\x1b]*m//g' file
As indicated in here, the a color code is composed of <Esc>[FormatCodem.
The escape character is \x1b in hexadecimal (sometimes noted as \e or \033).
The command looks for the sequence escape followed by square bracket \x1b\[ until the character m, if found it deletes it.
Everything in between these 2 characters is allowed except the escape character itself [^\x1b]*. This allows to have the shortest regex.
If you can't remove them from the tool producing the output, you could still remove them afterwards with the following sed command :
sed -r 's/\^\[\[[0-9]{1,2}m//g'
Example :
$ echo """^[[1m0024^[[0m, ^[[36munknown.10^[[0m --> ^[[32mUNKNOWN^[[0m""" | sed -r 's/\^\[\[[0-9]{1,2}m//g'
0024, unknown.10 --> UNKNOWN
I would like to be able to change the way Maxima generates the LaTeX code (in general). For example, I have the following code in Maxima:
I then exported the code to LaTeX, and I immediately get an error saying:
! Package inputenc Error: Unicode char \u8:− not set up for use with LaTeX.
See the inputenc package documentation for explanation.
Type H <return> for immediate help.
You can check out the LaTeX code generated through this gist on GitHub.
I would like at least not only to not get any errors, but also to change a little bit the style of the LaTeX code generation to adapt it to certain circumstances. For example, I would like to be able to insert a break line (or more) after the outputs...
Is it possible to do? Are there any alternatives?
You can put the following line in the preamble of the LaTeX output:
\DeclareUnicodeCharacter{2212}{-}
which tells LaTeX what to do with the Unicode hyphen (character 2212 according to this table).
WxMaxima should generate this declaration itself -- it is a bug that it does not.
Most likely something happened when you exported the code. To fix the existing file you can follow the accepted answer to this question.
https://tex.stackexchange.com/questions/83440/inputenc-error-unicode-char-u8-not-set-up-for-use-with-latex
But try exporting again and see if the error was accidental.
Update:
Add
\DeclareUnicodeCharacter{00A0}{ }
to your preamble.
Well, the gistfile1.txt at line 54 contains − characters instead of -. I wonder how those characters were generated. What commands did you enter in wxMaxima to generate gistfile1.txt?
Anyway, I find that if I replace those characters with ordinary hyphens, it avoids the error you reported. So perhaps the problem is to avoid generating those characters in the first place.
EDIT: This answer is off the mark. See my other answer for a real solution.
Had some spam issues on my server and, after finding out and removing some Perl and PHP scripts I'm down to checking what they really do, although I'm a senior PHP programmer I have little experience with Perl, can anyone give me a hand with the script here:
http://pastebin.com/MKiN8ifp
(It was one long line of code, script was called list.pl)
The start of the script is:
$??s:;s:s;;$?::s;(.*); ]="&\%[=.*.,-))'-,-#-*.).<.'.+-<-~-#,~-.-,.+,~-{-,.<'`.{'`'<-<--):)++,+#,-.{).+,,~+{+,,<)..})<.{.)-,.+.,.)-#):)++,+#,-.{).+,,~+{+,,<)..})<*{.}'`'<-<--):)++,+#,-.{).+:,+,+,',~+*+~+~+{+<+,)..})<'`'<.{'`'<'<-}.<)'+'.:*}.*.'-|-<.+):)~*{)~)|)++,+#,-.{).+:,+,+,',~+*+~+~+{+<+,)..})
It continues with precious few non-punctuation characters until the very end:
0-9\;\\_rs}&a-h;;s;(.*);$_;see;
Replace the s;(.*);$_;see; with print to get this. Replace s;(.*);$_;see; again with print in the first half of the payload to get this, which is the decryption code. The second half of the payload is the code to decrypt, but I can't go any further with it, because as you see, the decryption code is looking for a key in an envvar or a cookie (so that only the script's creator can control it or decode it, presumably), and I don't have that key. This is actually reasonably cleverly done.
For those interested in the nitty gritty... The first part, when de-tangled looks like this:
$? ? s/;s/s;;$?/ :
s/(.*)/...lots of punctuation.../;
The $? at the beginning of the line is the pre-defined variable containing the child error, which no doubt serves only as obfuscation. It will be undefined, as there can be no child error at this point.
The questionmark following it is the start of a ternary operator
CONDITION ? IF_TRUE : IF_FALSE
Which is also added simply to obfuscate. The expression returned for true is a substitution regex, where the / slash delimiter has been replaced with colon s:pattern:replacement:. Above, I have put back slashes. The other expression, which is the one that will be executed is also a substitution regex, albeit an incredibly long one. The delimiter is semi-colon.
This substitution replaces .* in $_ - the default input and pattern-searching space - with a rather large amount of punctuation characters, which represents the bulk of the code. Since .* matches any string, even the empty string, it will simply get inserted into $_, and is for all intents and purposes identical to simply assigning the string to $_, which is what I did:
$_ = q;]="&\%[=.*.,-))'-,-# .......;;
The following lines are a transliteration and another substitution. (I inserted comments to point out the delimiters)
y; -"[%-.:<-#]-`{-}#~\$\\;{\$()*.0-9\;\\_rs}&a-h;;
#^ ^ ^ ^
#1 2 3
(1,2,3 are delimiters, the semi-colon between 2 and 3 is escaped)
The basic gist of it is that various characters and ranges -" (space to double quote), and something that looks like character classes (with ranges) [%-.:<-#], but isn't, get transliterated into more legible characters e.g. curly braces, dollar sign, parentheses,0-9, etc.
s;(.*);$_;see;
The next substitution is where the magic happens. It is also a substitution with obfuscated delimiters, but with three modifers: see. s does nothing in this case, as it only allows the wildcard character . to match newline. ee means to evaluate the expression twice, however.
In order to see what I was evaluating, I performed the transliteration and printed the result. I suspect that I somewhere along the line got some characters corrupted, because there were subtle errors, but here's the short (cleaned up) version:
s;(.*);73756220656e6372797074696f6e5f6 .....;; # very long line of alphanumerics
s;(..);chr(hex($1));eg;
s;(.*);$_;see;
s;(.*);704b652318371910023c761a3618265 .....;; # another long line
s;(..);chr(hex($1));eg;
&e_echr(\$_);
s;(.*);$_;see;
The long regexes are once again the data containers, and insert data into $_ to be evaluated as code.
The s/(..)/chr(hex($1))/eg; is starting to look rather legible. It is basically reading two characters at the time from $_ and converting it from hex to corresponding character.
The next to last line &e_echr(\$_); stumped me for a while, but it is a subroutine that is defined somewhere in this evaluated code, as hobbs so aptly was able to decode. The dollar sign is prefixed by backslash, meaning it is a reference to $_: I.e. that the subroutine can change the global variable.
After quite a few evaluations, $_ is run through this subroutine, after which whatever is contained in $_ is evaluated a last time. Presumably this time executing the code. As hobbs said, a key is required, which is taken from the environment %ENV of the machine where the script runs. Which we do not have.
Ask the B::Deparse module to make it (a little more) readable.
^M is the dos carriage return that's left after each line when you move a file from a Windows box to a *NIX box. I know how to remove it. I am curious to know is there any other reason besides aesthetics that it should be removed from a PHP script.
The PHP script runs fine with it in. Normally, I would remove it without hesitation, but don't want to have my name next to each line in an svn blame command. (besides the point).
Question: Is there a reason in regards to functionality of why it should be remove other than aesthetics? It doesn't seem to break anything to keep it in. (Give me a good reason plz)
All in all, it should be fine. Other languages are picky about their line endings; I've seen it cause issues in Perl scripts, for example. But for PHP, i've never seen it matter much.
One occasion where it could conceivably matter is in multi-line strings, where the extra chars would make it through to the output. This might matter if your output is not HTML or XML. But JS shouldn't be particular about extraneous CRs, and HTML and XML will generally treat any whitespace the same as a single space (or in many cases, disregard whitespace altogether). Textareas and <pre> elements and such might end up with extra whitespace in them. That's about the only issue i can think of.