MSVC compilers support command files which are used to pass command line options. This is primarily due to the restriction on the size of the command line parameters that can be passed to the CreateProcess call.
This is less of an issue on Linux systems but when executing cygwin ports of Unix applications, such as gcc, the same limits apply.
Therefore, does anyone know if gcc/g++ also support some type of command file?
Sure!
#file
Read command-line options from file. The options read are inserted
in place of the original #file option. If file does not exist, or
cannot be read, then the option will be treated literally, and not
removed.
Options in file are separated by whitespace. A whitespace
character may be included in an option by surrounding the entire
option in either single or double quotes. Any character (including
a backslash) may be included by prefixing the character to be
included with a backslash. The file may itself contain additional
#file options; any such options will be processed recursively.
You can also jury-rig this type of thing with xargs, if your platform has it.
Related
I have the following unanswered questions, and I am looking for documentation that explains the PowerShell core requirement for LF vs CRLF line ending in Linux environments.
1- Can PWSH Core handle files with CRLF?
2- When I run a ps1 file with an LF line ending, can it call another ps1 file with mixed CRLF line endings?
3- Is the PWSH line ending requirement documented and consistent across all Linux distributions?
I am asking the above questions since PowerShell is born in Windows environment and I expect it can somehow tolerate LF vs. CRLF discrepancies.
A link to online documentation would be a great help. I did search and surprisingly, I could not find any.
Just preserving the content of the link mentioned in the comments (source):
Line endings
Windows editors, including ISE, tend to use a carriage return followed by linefeed (\r\n or `r`n) at the end of each line. Linux editors use linefeed only (\n or `n).
Line endings are less important if the only thing reading the file is PowerShell (on any platform). However, if a script is set to executable on Linux, a shebang must be included and the line-ending character used must be linefeed only.
For example, a created as follows named test.ps1 must use \n to end lines:
#!/usr/bin/env powershell
Get-Process
The first line is the shebang and lets Linux know which parser to use when executing the shell script.
Once created, chmod may be used to make the script executable outside of PowerShell:
chmod +x test.ps1
According to this, PowerShell can handle both - CRLF and LF - even in the same file. The only exception is the shebang: it has to end with LF.
stackprotector's community wiki answer, inspired by a link provided by Ben Voigt, provides the gist of an answer to your questions.
Let me complement it by answering your questions one by one:
I am looking for documentation that explains the PowerShell core requirement for LF vs CRLF line ending in Linux environments.
As of this writing, there appears to be no such documentation:
The conceptual about_Special_Characters help topic discusses `r (CR) and `n (LF) separately, but not explicitly in terms of their potential function as newline characters / sequences (line breaks; the topic uses the term "new line" to refer to a LF character, specifically).
1- Can PWSH Core handle files with CRLF?
Yes - both PowerShell editions - the legacy, ships-with-Windows, Windows-only Windows PowerShell edition (whose latest and final version is v5.1), as well as the cross-platform, install-on-demand, PowerShell (Core) edition (v6+) - treat CRLF and LF newlines interchangeably - both with respect to reading source code and reading files[1] - even if the two newline formats are mixed in a single file.
While not documented as such, PowerShell has always worked this way, and, given its commitment to backward compatibility, this won't change (which in this case is definitely a blessing, given that PowerShell (Core) is now cross-platform and must be able to handle files with LF-only newlines on Unix-like platforms).
The following examples demonstrate this - they work the same on all supported platforms ("`n" creates a LF-only newline, "`r`n" a CRLF newline):
# Read a file with mixed newline formats.
PS> "one`ntwo`r`nthree" > temp.txt; (Get-Content temp.txt).Count; Remove-Item temp.txt
3
# Execute a script with mixed newline formats.
PS> "'one'`n'two'`r`n'three'" > temp.ps1; . ./temp.ps1; Remove-Item temp.ps1
one
two
three
2- When I run a ps1 file with an LF line ending, can it call another ps1 file with mixed CRLF line endings?
Yes - this follows from the above.
However, special considerations apply to shebang-line-based PowerShells scripts on Unix-like platforms:
Such stand-alone shell scripts - which needn't an arguably shouldn't have a .ps1 extension - are first read by the system on Unix-like platforms, and therefore require the shebang line - by definition the first line - to be terminated with a LF-only ("`n") newline, given that only LF by itself is considered a newline on Unix-like platforms.[2] All remaining lines are then read only by PowerShell, and any mix of CLRF and LF is then accepted, as usual; e.g.:
# Run on any Unix-like platform - note that `n alone must end the first line.
PS> "#!/usr/bin/env pwsh`n'one'`r`n'two'" > temp; chmod a+x temp; ./temp; Remove-Item temp
one
two
In practice, not least due to the not insignificant startup cost of pwsh, the PowerShell (Core) CLI, but also due to several bugs as of PowerShell 7.2.6, stand-alone shebang-line-based PowerShell scripts - which are primarily useful for being called from outside PowerShell - are rare.
3- Is the PWSH line ending requirement documented and consistent across all Linux distributions?
No, it isn't documented.
Yes, as implied by the above, it is consistent, not just across Linux distributions, but across all supported platforms.
[1] Even CR-only newlines - as used in long-obsolete legacy mac OS versions, which should therefore be avoided nowadays - are recognized in PowerShell source code and by Get-Content, but not by Measure-Object -Line, for instance.
[2] If the first line ends in CRLF, the CR (\r) is retained as part of the line and therefore as part of the target executable path or the last option passed to it, which breaks the invocation - see this answer for a real-life manifestation of this problem.
Being as cannonical as a I can be:
I worked on PowerShell v2-v3.
Newlines of either form were always meant to be interchangeable within PowerShell.
There was a good amount of Unix influence in the language. Being able to support both forms of newlines was near and dear to many a team members' heart. Being able to support it from the get-go prevented the possibility of making a script that couldn't run just because it was copied to mac or linux and saved in the wrong editor.
The importance of this feature has been proved again and again, and has obviously become mission critical now that PowerShell Core is a thing (because the scenario listed above is far more common). I'd wager that as long as PowerShell is a language, this will be the behavior of PowerShell scripts.
As far as shebang files go, this isn't really an exception to this rule. With a shebang file, Unix is reading the file line by line and then sending it to the interpreter. A carriage return is outside of it's range of expectations, not PowerShell's.
Hope this helps shed some light on things.
I'm not asking about general syntactic rules for file names. I mean gotchas that jump out of nowhere and bite you. For example, trying to name a file "COM<n>" on Windows?
From: http://www.grouplogic.com/knowledge/index.cfm/fuseaction/view_Info/docID/111.
The following characters are invalid as file or folder names on Windows using NTFS: / ? < > \ : * | " and any character you can type with the Ctrl key.
In addition to the above illegal characters the caret ^ is also not permitted under Windows Operating Systems using the FAT file system.
Under Windows using the FAT file system file and folder names may be up to 255 characters long.
Under Windows using the NTFS file system file and folder names may be up to 256 characters long.
Under Window the length of a full path under both systems is 260 characters.
In addition to these characters, the following conventions are also illegal:
Placing a space at the end of the name
Placing a period at the end of the name
The following file names are also reserved under Windows:
aux,
com1,
com2,
...
com9,
lpt1,
lpt2,
...
lpt9,
con,
nul,
prn
Full description of legal and illegal filenames on Windows: http://msdn.microsoft.com/en-us/library/aa365247.aspx
A tricky Unix gotcha when you don't know:
Files which start with - or -- are legal but a pain in the butt to work with, as many command line tools think you are providing options to them.
Many of those tools have a special marker "--" to signal the end of the options:
gzip -9vf -- -mydashedfilename
As others have said, device names like COM1 are not possible as filenames under Windows because they are reserved devices.
However, there is an escape method to create and access files with these reserved names, for example, this command will redirect the output of the ver command into a file called COM1:
ver > "\\?\C:\Users\username\COM1"
Now you will have a file called COM1 that 99% of programs won't be able to open, and will probably freeze if you try to access.
Here's the Microsoft article that explains how this "file namespace" works. Basically it tells Windows not to do any string processing on the text and to pass it straight through to the filesystem. This trick can also be used to work with paths longer than 260 characters.
The boost::filesystem Portability Guide has a lot of good info.
Well, for MSDOS/Windows, NUL, PRN, LPT<n> and CON. They even cause problems if used with an extension: "NUL.TXT"
Unless you're touching special directories, the only illegal names on Linux are '.' and '..'. Any other name is possible, although accessing some of them from the shell requires using escape sequences.
EDIT: As Vinko Vrsalovic said, files starting with '-' and '--' are a pain from the shell, since those character sequences are interpreted by the application, not the shell.
My Perforce-based project supports both Linux and Cygwin platforms with the same shell scripts (e.g. build_project.sh). But Perforce defaults line endings for text files to the local platform (Docs). This causes \r\n newlines in the .sh scripts, which fail on Cygwin.
Some of the ideas I've thought of so far:
Is there a way to make Cygwin accept \r\n files? (Without having to run dos2unix, the files fetch as read-only).
Is there a way to set specific files to be text, but with Unix line endings for everyone? (I am guessing, "no", but thought I'd check.)
Of course I can set the entire workspace's line endings to \n (unix). But this makes the Windows clients unhappy with their .bat files being \n instead of \r\n. Also if the setting is per workspace (I can't recall), then a workspace setup is slightly harder for the new Windows user as they must set that option.
Set the .sh files to be "binary", but then we lose the text diffs on those files. Is there a workaround for this? Is this the common (good) hack?
This is a fairly minor nit, but I suspect that some of you have a BKM for this pattern.
Thanks.
EDIT: Craig's answer in this question seems to suggest that using Unix line endings will just leave files with \r\n's alone if they are originally submitted that way.
EDIT: To force bash (i.e. Cygwin) to accept files with \r\n endings, one can specify set -o igncr in the script. This is nice if one expects Cygwin users to that might not be very Unix literate (my case) or when we can't globally impose the trigger in the solution below for some other reason.
I believe that when you install Cygwin you can configure it to use Windows line endings. Leaving that aside, though:
If you use the "unix" LineEnd for absolutely everyone, then all of your text files will have their own internally-consistent line endings (but will not be necessarily consistent with the client platforms). This works by virtue of the fact that the Windows files will end up having the \r as part of the content of the line, so when being synced out in "unix" format they'll have \r\n endings.
The thing to watch out for is mixing and matching LineEnd settings when doing this -- if somebody with a "win" or "local" LineEnd syncs that same file, now they have \r\r\n endings! So if you want to go with the per-file line ending plan, make sure EVERYONE uses "unix" as their LineEnd. This is pretty easy to do with a trigger, e.g.:
Triggers:
form-in client "sed -i %quote%s/LineEnd:.*/LineEnd: unix/%quote% %formfile%"
I need to parse the $PATH environment variable in my application.
So I was wondering what escape characters would be valid in $PATH.
I created a test directory called /bin:d and created a test script called funny inside it. It runs if I call it with an absolute path.
I just can't figure out how to escape : in $PATH I tried escaping the colon with \ and wrapping it into single ' and double " quotes. But always when I run which funny it can't find it.
I'm running CentOS 6.
This is impossible according to the POSIX standard. This is not a function of a specific shell, PATH handling is done within the execvp function in the C library. There is no provision for any kind of quoting.
This is the reason why including certain characters (anything not in the "portable filename character set" - colon is specifically called out as an example.) is strongly recommended against.
From SUSv7:
Since <colon> is a separator in this context, directory names that might be used in PATH should not include a <colon> character.
See also source of GLIBC execvp. We can see it uses the strchrnul and memcpy functions for processing the PATH components, with absolutely no provision for skipping over or unescaping any kind of escape character.
Looking at the function
extract_colon_unit
it seems to me that this is impossible. The : is unconditionally and
inescapably used as the path separator.
Well, this is valid at least for bash. Other shells may vary.
You could try mounting it
mount /bin:d /bind
PATH=/bind
According to http://tldp.org/LDP/abs/html/special-chars.html single quotes should preserve all special characters, so without trying it, I would think that '/bin:d' would work (with)in $PATH.
Assume we have a file /root/file.ini.
In Ubuntu's shell, we can show the content with this command,
less /root\\file.ini
However, in debian's shell, the command below will report that the file does not exist.
Does anybody happen to know how to make linux support "\\" as a path delimiter? I need to solve it because we have a software, which tries to access a file using "\\". It works fine in ubuntu, but not in debian.
Thanks
Linux cannot support \ as a path delimiter (though perhaps it might be able to with substantial changes to the kernel). This is because \ is a valid file name character. In fact the only characters not allowed as part of a file name are / and \0 (the null character).
If this seems to be working under ubuntu, then I would check for the existence of a file called root\file.ini in /
I believe you will probably find it easier to make your program platform independent.
I found this forum post which states / is a platform independent path delimiter in ANSI C any that file operations will automatically convert / to actual path delimiter used on the host OS.
have you tried "\\\\" (4 backslashes) first and third one for escaping and second and the last one to rule them all?