I've some problem when working on remote files (perl scripts) with Komodo IDE. There is (as far as I know) no way to change ftp transfer mode from binary to ASCII, which result in "^M" character at the end of every line. My setup is Linux server, and Windows client. Is there any way to solve this issue without nessecity of correcting saved file on Linux every time. This behaviour disqualify Komodo IDE, which was my favourite IDE until now.
The "^M" you observe has nothing to do with your file being ASCII, but line ending format (carriage return and line feed characters.)
I have not verified this, but here's a link showing how to save files in Komodo using a different line ending method. Saving files in DOS mode is not needed anymore, since most editors recognize UNIX file format nowadays.
Add switch -w to your Perl shebang.
Related
I'm trying to send file using curl to an endpoint and save the file to the machine.
Sending curl from Linux and saving it on the machine works well,
but doing the same curl from Windows is adding ^M character to every end of line.
I'm printing the file before saving it and can't see ^M. Only viewing the file on the remote machine after saving it shows me ^M.
A simple string replacement doesn't seem to work.
Why is ^M being added? How can I prevent this?
Quick Answer: That's a carriage return. They're a harmless but mildly irritating artifact of how Windows encodes text files. You can strip them out of your files with dos2unix. You can configure most text editors to use "Unix Line Endings" or "LF Line Endings" to prevent them from appearing in new files that you create from Windows PCs in the future.
Long Answer (with some historical trivia):
In a plain text file, when you create a new line (by pressing enter/return), a "line break" is embedded in the file. On Unix/Linux, this is a single character, '\n', the "line feed". On Windows, this is two sequential characters, '\r\n', the "carriage return" followed by the "line feed".
When physical teletype terminals, which behaved much like typewriters, were still in use, the "line feed" character meant "move the paper up to the next line" and the "carriage return" character meant "slide the carriage all the way over so the typing head is on the far left". From the very beginning, nearly all teletype terminals supported implicit carriage return; i.e., triggering a line feed would automatically trigger a carriage return. The developers working on what later evolved into Windows decided that it would be best to include explicit carriage returns, just in case (for some reason) the teletype does not perform one implicitly. The Unix developers, on the other hand, chose to work with the assumption of implicit carriage return.
The carriage return and line feed are ASCII Control Characters which means they do not have a visible representation as standalone printable characters, instead they affect the output cursor itself (in this case, the position of the output cursor).
The "^M" you see is a stand-in representation for the carriage return character, used by programs that don't fully "cook" their output (i.e., don't apply the effects of some ASCII Control Characters). (Other control characters have other representations starting with "^", and the "^" character is also used to represent the "ctrl" keyboard key in some Unix programs like nano.)
You can use dos2unix to convert the line endings from Windows-style to Unix-style.
$ curl https://example.com/file_with_crlf.txt | dos2unix > file.txt
On some distros, this tool is included by default, on others it can be installed via the package manager (e.g., on Ubuntu, sudo apt install dos2unix). There also exists a package, unix2dos, for the inverse.
Most "smart" text editors for coding (Sublime, Atom, VS Code, Notepad++, etc.) will happily read and write with either Windows-style or Unix-style line endings (this might require changing some configuration options). Often, the line-endings are auto-detected by scanning the contents of a file, and usually new files are created with the Operating System's native line endings (by default). Even the new version of Notepad supports Unix-style line endings. On the other hand, some Unix tools will produce strange results in the presence of Windows-style line breaks. If your codebase will be used by people on both Unix and Windows operating systems, the nice thing to do is to use Unix-style line endings everywhere.
Git on Windows also has an optional mode that checks out all files with Windows-style line breaks, but checks them back in with Unix-style line breaks.
Side Notes (interesting, but not directly related to your question):
What the carriage return actually does (on a modern virtual terminal, be it Windows or Unix) is move the output cursor to the beginning of the line. If you use the carriage return without a line feed, you can "overwrite" part of a string that has already been printed.
$ printf "dogdog" ; printf "\rcat\n"
catdog
Some Unix programs use this to asynchronously update part of the last line of output, to implement things like a live-updating progress indicator. For example, curl, which shows download progress on stdout if the file contents are piped elsewhere.
Also: If you had a tool that interpreted Windows-style line endings as literally as possible, and you fed it a string with Unix-style line endings such as "hello\nworld", you would get output like this:
hello
world
Fortunately, such implementations are extremely rare and, in general, the vast majority of Windows tools can render Unix-style line-endings identically to Windows-style line endings without any problem.
I develop and use Git on both Windows and Linux. When using IDEs or Git on Windows, I'm frequently prompted on whether to save files with CR LF or not.
I am doing mainly C# and JavaScript ES6 development which involves code that contains multi-line strings.
What reasons are there to save files with CR LF on Windows? Are CR-LFs mostly of historical significance? I have not yet noticed a drawback to working with UNIX \n line endings on Windows.
Windows batch files can malfunction when saved with newline endings because the goto command works by jumping to the appropriate offset in the script — but it is not computed correctly unless the lines end with carriage-return/line-feed.
Some Windows programs don't properly handle '\n' without '\r' but any decent editor or for that matter any decent program should handle them identically. But CRLF is traditionally the sanctioned way to do line endings on Windows and you might have compatibility issues if you don't.
My Perforce-based project supports both Linux and Cygwin platforms with the same shell scripts (e.g. build_project.sh). But Perforce defaults line endings for text files to the local platform (Docs). This causes \r\n newlines in the .sh scripts, which fail on Cygwin.
Some of the ideas I've thought of so far:
Is there a way to make Cygwin accept \r\n files? (Without having to run dos2unix, the files fetch as read-only).
Is there a way to set specific files to be text, but with Unix line endings for everyone? (I am guessing, "no", but thought I'd check.)
Of course I can set the entire workspace's line endings to \n (unix). But this makes the Windows clients unhappy with their .bat files being \n instead of \r\n. Also if the setting is per workspace (I can't recall), then a workspace setup is slightly harder for the new Windows user as they must set that option.
Set the .sh files to be "binary", but then we lose the text diffs on those files. Is there a workaround for this? Is this the common (good) hack?
This is a fairly minor nit, but I suspect that some of you have a BKM for this pattern.
Thanks.
EDIT: Craig's answer in this question seems to suggest that using Unix line endings will just leave files with \r\n's alone if they are originally submitted that way.
EDIT: To force bash (i.e. Cygwin) to accept files with \r\n endings, one can specify set -o igncr in the script. This is nice if one expects Cygwin users to that might not be very Unix literate (my case) or when we can't globally impose the trigger in the solution below for some other reason.
I believe that when you install Cygwin you can configure it to use Windows line endings. Leaving that aside, though:
If you use the "unix" LineEnd for absolutely everyone, then all of your text files will have their own internally-consistent line endings (but will not be necessarily consistent with the client platforms). This works by virtue of the fact that the Windows files will end up having the \r as part of the content of the line, so when being synced out in "unix" format they'll have \r\n endings.
The thing to watch out for is mixing and matching LineEnd settings when doing this -- if somebody with a "win" or "local" LineEnd syncs that same file, now they have \r\r\n endings! So if you want to go with the per-file line ending plan, make sure EVERYONE uses "unix" as their LineEnd. This is pretty easy to do with a trigger, e.g.:
Triggers:
form-in client "sed -i %quote%s/LineEnd:.*/LineEnd: unix/%quote% %formfile%"
I am on Windows 7 running PowerShell version 2.0.
I used the following PowerShell code (from this answer: https://stackoverflow.com/a/65148/360840) to modify my PowerShell profile:
"`nNew-Alias which get-command" | add-content $profile
As expected, it appended the line in question to my profile BUT afterwards, my Vim sessions started showing ^M characters at the end of each line. Very strange! Any clue on what caused it and how to fix it? I tried set fileformat=dos in Vim but it did nothing.
Here's my Vim version information:
(Not sure how to make the picture larger - this is not its normal size.)
There are some options to solve the problem in vim-faq 14.13. I am seeing a lot of ^M symbols in my file. I tried setting the 'fileformat' option to 'dos' and then 'unix' and then 'mac'. None of these helped. How can I hide these symbols?
It is possible that your file was in unix file format (all lines ending in <CR>), and you appended lines with dos file format (lines ending in <CR><NL>, what makes Vim confused.
So, I ran into this issue, and the solution was slightly different. I've checked my file format, and it's unix, you can check yours with
:set fileformat?
And it displays all of the ^M characters I don't want... The interesting thing, is that in the VIM manual as pointed to above at: http://vimhelp.appspot.com/vim_faq.txt.html#faq-14.13 states that the ^M line ending is from the DOS file format and that "If the file has some lines ending with and some lines ending with followed by a , then the fileformat is set to 'unix'."
So, given this it would seem that you would want to remove all of the ^M endings, and keep the endings, but removing the endings with :%s/\r$// immediately resolved the issue for me, and the file format still says 'unix'.
That's what worked for me. As a side-note, in order to provide a more google-able reference, I would like to note that I ran into this while using LocomotiveCMS. I was pulling my CSS file back down to work on locally, using Wagon. When I got it back it was full of ^M characters, the file still works fine, it's just ugly.
The problem was that the file became corrupted. In such a case, Vim starts displaying ^M characters to alert you to the problem. The fix was to manually remove these characters (%s/^M//g).
Still not clear to me, because these characters are SUPPOSED TO BE in the file as Windows file format uses them. But at least the problem was solved.
EDIT: Please see the exchange of comments w/ dash-tom-bang, which provides clarification.
I have a couple files that were recently edited on windows and via Cpanel's file editor and now show up double spaced (as in an extra line CR/LF between each line). Vim is telling me (via :set ff?) the file format is unix (and I'm working on a Mac). If I show special characters via :set list all the lines just end in $. I tried setting the format via :e ++ff=mac which appears to remove all line breaks in the currently edited document and when I write the file and re-open it's back to being double spaced. I also tried searching and replacing ^M and various \r\n combinations. I know I'm missing something simple but can someone shed some light on what is going on? Is this even a line ending issue?
It appears to be a line ending issue.
The Vim wiki has this to say on the subject:
http://vim.wikia.com/wiki/File_format#Terminator_after_last_line
I, however, for expediency, when faced with a line ending problem, use BBEdit on my Mac to change them to Unix (I share, on the LAN, my eight Linux boxes with a Macbook Pro so I use a directory in Dropbox to transfer files across. scp will do the same job).
Unless you have a copy of BBEdit lying about, you can download Barebones's free Text Wrangler & it'll do the same job. Only works on a Mac obviously...