My application uses fopen to open a lot of files. While in linux opening and reading thousand of files doesn't even take a second; in cygwin it takes more than 5 seconds.
I think it is because path conversion functions in cygwin dlls. 'open' function is a bit faster. If I use -mno-cygwin it becomes very fast but I can't use it.
Is there an easy way to make cygwin dlls just open files; without any linux-windows conversion?
It depends on how the system was mounted in the Cygwin environment.
$ mount
C:/cygwin/bin on /usr/bin type ntfs (binary,auto)
C:/cygwin/lib on /usr/lib type ntfs (binary,auto)
C:/cygwin on / type ntfs (binary,auto)
C: on /cygdrive/c type ntfs (binary,posix=0,user,noumount,auto)
D: on /cygdrive/d type iso9660 (binary,posix=0,user,noumount,auto)
The mount option "binary" makes it so CRLF <-> LF conversions are not performed on files read from the volume. This is default.
Some things you can do to speed up a Cygwin prompt are the following:
Add the following lines to your ~/.bashrc:
# eliminate long Windows pathnames from the PATH
export PATH='/bin:/usr/bin:/usr/local/bin'
# check the hash before searching the PATH directories
shopt -s checkhash
# do not search the path when .-sourcing a file
shopt -u sourcepath
Disconnect your network drives.
Disable your antivirus, or otherwise exclude Cygwin's folders from its scans.
Thorough antivirus programs scan files for malware as they're opened by programs, and this means it'll be working overtime if your script is opening thousands of files.
Use the option --cache-file="$HOME/.config.cache" when running autotools configure scripts.
This will create a file that holds prerecorded configure discoveries, most of which are usable between software builds. (This is also a good idea when using Linux).
Since the shell seems to be the bottleneck of the Cygwin system, a huge script that relies on starting a large number of processes will take forever and this will cut down on the number of processes it needs to start.
Set up Cygwin's sshd and stop using Windows Command Prompt in favor of PuTTY.
PuTTY responds better to changing text on the screen, as it was built for the more mature CLI interface of *NIX.
Related
Host system: Windows Server 2008 32-bit
Installed: Cygwin
I don't know when this problem started, but one of my Rails gems uses the command which to determine the location of a system-installed executable. In my circumstance, it returns /cygdrive/c/Windows/System32/pngcrush - and the file is inaccessible.
In both cygwin terminal and Windows cmd I get the following:
ls -la /cygdrive/c = No such file or directory
mount =
C:/cygwin/bin on /usr/bin type ntfs (binary,auto)
C:/cygwin/lib on /usr/lib type ntfs (binary,auto)
C:/cygwin on / type ntfs (binary,auto)
B: on /cygdrive/b type smbfs (binary,posix=0,user,noumount,auto
C: on /cygdrive/c type ntfs (binary,posix=0,user,noumount,auto)
Running cd / in Windows terminal brings me to C:\
cd /cygdrive = The system cannot find the path specified
FURTHER EDIT:
I can access the drives using /c for example, but just not /cygdrive/c which is what which returns.
I found the answer on superuser.com; moving C:\cygwin\bin to the front of the PATH variable fixed the issue.
This fix was necessary, but not sufficient, for me; one more fix was needed. I was getting the error running Bourne shell scripts which referred to other files on the localFS. All the paths checked out. It wasn't until I used Sys Internals procmon to troubleshoot that I noticed an extra char at the end of the path that wasn't resolving. Viewing the shell script in Notepad++ with View/Show Symbol/Show All Characters revealed Windows style [CR][LF] line ends, not Unix [LF]. Cygwin's sh.exe was including the [CR] on the end of the path, resulting in file not found. I appied cygwin's d2u to convert line ends, and the problem was fixed.
Recent NTFS and Windows implement symlinks:
NTFS junction point can be used as directory symlink since NTFS 3.0 (Windows 2000) using linkd or junction tools.
NTFS symbolic link can also be used as symlink (for both file and directory) since Windows Vista using mklink tool.
But on Cygwin 1.7 (installed on Windows 7), ln -s creates a text file.
on Cygwin:
$ ln -s -v target mylink
`mylink' -> `target'
on MinGW (or your favorite editor):
$ cat mylink
!<symlink>ÿþt a r g e t
Is it possible to tell Cygwing to use NTFS junction point or NTFS symbolic link?
other question: Is this available on MinGW?
⸻⸻ Short answer ⸻⸻
Define environment variable:
CYGWIN=winsymlinks:nativestrict
As pointed out by mwm you may also have to go to the settings or to run bash as Administrator. See the Notes section.
⸻⸻ Long answer ⸻⸻
Default Cygwin symlinks are just regular files
By default Cygwin creates text files as workaround for Windows symlink flaw.
These files are not really symlinks.
Almost all Windows programs do not considers these files as symlinks.
Native symlinks are available on recent Windows versions
Recent NTFS and Windows implement symlinks:
NTFS junction point can be used as directory symlink
since NTFS 3.0 (Windows 2000) using linkd or junction tools.
NTFS symbolic link can also be used as symlink
(for both file and directory) since Windows Vista using mklink tool.
Cygwin can create native NTFS symlinks
Simplified extract of the Cygwin documentation:
Symbolic links
[...]
Cygwin creates symbolic links potentially in multiple different ways:
The default symlinks are plain files containing a magic cookie
followed by the path to which the link points. [...]
The shortcut style symlinks are Windows .lnk [...] created
if the environment variable CYGWIN [...] is set to contain
the string winsymlinks or winsymlinks:lnk. [...]
Native Windows symlinks are only created on Windows Vista/2008 and later,
and only on filesystems supporting reparse points.
Due to to their weird restrictions and behaviour, they are only created
if the user explicitely requests creating them.
This is done by setting the environment variable CYGWIN
to contain the string winsymlinks:native or winsymlinks:nativestrict.
[...]
On the NFS filesystem, Cygwin always creates real NFS symlinks.
Configuring Cygwin
Cygwin User's Guide presents variable CYGWIN and option winsymlinks:
The CYGWIN environment variable is used to configure many global settings [...].
It contains the options listed below, separated by blank characters. [...]
[...]
[...]
[...]
[...]
winsymlinks:{lnk,native,nativestrict} -
if set to just winsymlinks or winsymlinks:lnk, Cygwin creates symlinks
as Windows shortcuts with a special headerand the R/O attribute set.
If set to winsymlinks:native or winsymlinks:nativestrict,
Cygwin creates symlinks as native Windows symlinks on filesystems
and OS versions supporting them. If the OS is known not to support
native symlinks (Windows XP, Windows Server 2003), a warning message
is produced once per session.
The difference between winsymlinks:native and
winsymlinks:nativestrict is this: If the filesystem supports native
symlinks and Cygwin fails to create a native symlink for some reason,
it will fall back to creating Cygwin default symlinks with
winsymlinks:native, while with winsymlinks:nativestrict
the symlink(2) system call will immediately fail.
CYGWIN=winsymlinks:native always creates a link but uses a Cygwin fall-back when target does not exists
on Cygwin:
$ export CYGWIN="winsymlinks:native"
$ ln -s -v target mylink
`mylink' -> `target'
$ echo content > target
on MinGW:
$ cat mylink
content
People using both Windows and Cygwin programs may have issues when a symlink is created as a dummy file (Cygwin fallback when target is missing)...
CYGWIN=winsymlinks:nativestrict always uses native-Windows symlink but fails when target does not exist
on Cygwin:
$ export CYGWIN="winsymlinks:nativestrict"
$ rm -f a b
$ ln -sv a b
ln: failed to create symbolic link `b': No such file or directory
$ touch b
$ ln -sv a b
ln: failed to create symbolic link `b': File exists
$ rm b
$ touch a
$ ln -sv a b
`b' -> `a'
Because nativestrict requires the target exists before the symlink creation, some commands/scripts may fail when creating a link.
Notes
Since Windows 10 build 14972, native NTFS symlinks are available in a non-elevated shell by enabling the Developer Mode in the Developer Settings.
Reference: https://blogs.windows.com/windowsdeveloper/2016/12/02/symlinks-windows-10/
In older versions, only administrators have the ability to create native NT symlinks
so under Windows UAC, the Cygwin terminal emulator (mintty)
should be run with elevated privileges
(right-click the shortcut and choose Run as Administrator
or set the mintty shortcut property, Advanced → Run as Administrator).
Special thanks to Guria, Spooky and Gene Pavlovsky for their contributions.
The accepted answer is right, two little side notes.
If you only care about the symlinks you create yourself on the command line, install cygutils-extra package, it includes a winln command, which has the same syntax as ln, but creates native Windows links. Create an alias: alias ln=winln (only works in interactive shell), or even replace the ln file with winln (works in shell scripts as well) - but it might get overwritten the next time coreutils package is updated.
I've only found out it's possible to use native symlinks when I already had Cygwin installed, and added some symlinks by myself as well. So after I set CYGWIN=winsymlinks:native as my system environment variable, I wanted to convert all the existing non-native links to native. Here's what I did.
Just in case, back up your entire Cygwin directory first.
Find all symlinks and save the list to /links file:
cd /; find . -regextype egrep -regex './(dev|proc|mnt|cygdrive)' -prune -o -type l -print >links
Review links.
Create a tar archive with all the links: tar c --files-from=links >links.tar
Extract the tar archive: tar x --files-from=links <links.tar
Since native symlinks are now enabled, tar will overwrite the old Cygwin's symlinks with native symlinks.
Clean up: rm -f links links.tar
P.S. At first I used CYGWIN=winsymlinks:nativestrict, but then I found out that in this mode, ln -s target link fails if target doesn't exist. By contrast, native will create a Cygwin (non-native) symlink link pointing to the nonexistent target - this matches the behavior of ln on UNIX systems. In rare cases, nativestrict can break some programs or scripts, for example Gentoo run-crons script uses a lockfile which is a symlink pointing to the PID of the running process. In nativestrict mode the script stopped working, because it could no longer create the lockfile. Note: run-crons is a crontab helper script on Gentoo Linux, adding support for cron.{hourly,daily,weekly,monthly}/ dirs, it works very well with Cygwin.
Since #olibre answer didn't work for me. I just created a shell function.
: '
mklink - Create NTFS (Windows) links that is usable by Windows and Cygwin
Usage: mklink [/D | /H | /J] <link-path> <target-path>
Options:
/D Directory Symbolic Link
/H Hardlink
/J Directory Junction (you should prefer /D)
With no options, it creates a NTFS file symlink.
'
mklink () {
if [ "$#" -ge "3" ]; then
cmd /c mklink "$1" "$(cygpath --windows --absolute "$2")" "$(cygpath --windows --absolute "$3")"
else
cmd /c mklink "$(cygpath --windows --absolute "$1")" "$(cygpath --windows --absolute "$2")"
fi
}
Do note you need administrator permissions (for Cygwin) to run the above without problems.
Note that I am unaware whether there's any difference between symlinking to an absolute path versus symlinking to a relative path using CMD's mklink. On Linux, those 2 have different behaviours if you ever decide to move the symlink or move the target file, or move both.
I guess the easiest way is to
grant SeCreateSymbolicLinkPrivilege from Local Group Policy editor (gpedit.msc, on path by default, non-home versions)
create script named ln on path (batch or bash), implementation
similar to above described shell function
profit
You were probably looking for a way to get to another destination in catalogue tree using MSYS. There is a way. You should create a shell script ("*.sh" file) which contains line:
cd "/drive_letter/SubCatalogue/SubFolder/..."
Here is my problem,
In Windows I am making a zip file in which there is a text .sh file which is supposed to be executed in Linux.
The user on the other end opens the zip file in Linux and tries to execute the .sh file but the execute permission is gone. So the user has to do it manually ( like explained here:add execute permission.
How can I in Windows make the .sh executable and add it to a zip file so that when the zip file opens in linux the .sh file still retains its execute permission ( so that user doesn't have to do it manually)
As far as I know the permission system in Linux is set up in such a way to prevent exactly what you are trying to accomplish.
I think the best you can do is to give your Linux user a custom unzip one-liner to run on the prompt:
unzip zip_name.zip && chmod +x script_name.sh
If there are multiple scripts that you need to give execute permission to, write a grant_perms.sh as follows:
#!/bin/bash
# file: grant_perms.sh
chmod +x script_1.sh
chmod +x script_2.sh
...
chmod +x script_n.sh
(You can put the scripts all on one line for chmod, but I found separate lines easier to work with in vim and with shell script commands.)
And now your unzip one-liner becomes:
unzip zip_name.zip && source grant_perms.sh
Note that since you are using source to run grant_perms.sh, it doesn't need execute permission
The ZIP file format does allow to store the permission bits, but Windows programs normally ignore it.
The zip utility on Cygwin however does preserve the x bit, just like it does on Linux.
If you do not want to use Cygwin, you can take a source code and tweak it so that all *.sh files get the executable bit set.
Or write a script like explained here
This is possible using the Info-Zip open-source Zip utilities. If unzip is run with the -X parameter, it will attempt to preserve the original permissions. If the source filesystem was NTFS and the destination is a Unix one, it will attempt to translate from one to the other. I do not have a Windows system available right now to test the translation, so you will have to experiment with which group needs to be awarded execute permissions. It'll be something like "Users" or "Any user"
Use my windows command line utility zip_exec.zip to set the executable flag for linux/unix and mac (tested on files created with Windows Explorer and 7zip). The cpp source is also available. I searched the internet a lot before making my own utility. It can be modified to set any file attribute.
This is not possible. Linux permissions and windows permissions do not translate. They are machine specific. It would be a security hole to allow permissions to be set on files before they even arrive on the target system.
I have a folder /cygwin/d/myfolder/
And everytime I save files there, from cygwin if i do an ls -la I see that the files are given permission 000. That actually causes me quite a bit of problem as I rsync this folder to my server and none of the files are accessible. How can I get the files to automatically get a reasonable permission?
Have a read through the answers at this link:
http://cygwin.1069669.n5.nabble.com/vim-and-file-permissions-on-Windows-7-td61390.html
The solution there worked for me also:
Edit /etc/fstab and add this line at the end of the file:
none /cygdrive cygdrive binary,noacl,posix=0,user 0 0
Then close all Cygwin processes, open a new terminal and ls -l on your files again.
Explanation:
By default, Cygwin uses the filesystem's access control lists (ACLs) to implement real POSIX permissions. Some Windows-native program or process may create or modify the ACLs such that Cygwin computes the POSIX permissions as 000. With the noacl mount option, Cygwin ignores filesystem ACLs and only fakes a subset of permission bits based on the DOS readonly attribute.
Check to make sure that your umask is set correctly with the umask command. If your umask is say 0777 that subtracts from the permissions of new files and will end up with 000 permissions. There's probably several other possibilities to consider beyond that.
If your id is not set up correctly in /etc/passwd and /etc/group that can also cause ls to show unexpected results. Check the permissions of the folder. Also check the Windows permissions with the getfacl command. Maybe also check the mount command.
In above answer, solution was proposed:
Edit /etc/fstab and add this line at the end of the file:
none /cygdrive cygdrive binary,noacl,posix=0,user 0 0
And in that answer there was this comment:
When I try this, all my files are -rw-r--r-- no matter what chmod() I do. I can't mark the files as executable; it just reverts to 0644. (umask==0022)
I had this same problem, but it manifested in inability to execute DOS batch files (*.bat) when running Cygwin ksh or mksh. I stumbled across this website: http://pipeline.lbl.gov/code/3rd_party/licenses.win/cygwin-doc-1.4/html/faq/ which contains this helpful advice:
Note that you can use mount -x to force Cygwin to treat all files under the mount point as executable. This can be used for individual files as well as directories. Then Cygwin will not bother to read files to determine whether they are executable.
So then cross-referencing with this page - https://cygwin.com/cygwin-ug-net/using.html#mount-table - with its advice:
cygexec - Treat all files below mount point as cygwin executables.
I added cygexec to fourth field of my fstab. This did it. My .bat is now executable inside ksh/mksh, which is necessary since I'm running a Jenkins job that calls a Korn shell stack 3 files deep, that I have no modifiable control over. I just needed the .bat to run!
Update: the solution above wasn't quite what I needed, on further testing. It resulted in some executables such as javac and cl to behave oddly (the utilities would print their usage and exit). I think what I needed instead of 'cygexec' was just 'exec'. As the same page notes:
exec - Treat all files below mount point as executable.
On my Win7 PC files were usually
----------+ 1 David None 69120 Jun 17 13:17 mydoc.txt
I tried all of above no luck
Turned out I still had some old historical mount entries in my .zshrc
I deleted these and Bob's your Uncle problem gone away!
I'm pretty new to programming for Linux environments, so I don't exactly know what to search for in order to answer this question for myself. I need to understand how applications set the shell to accept a certain command to start them. For example, you can start Firefox from the command line by executing the command: firefox.
I don't know where this is defined. Makefile? Configure script? In the source code itself?
Any resources / reading on Linux programming tidbits like these would be greatly appreciated!
Thank you.
Firefox is launched by the command "firefox" because there is an executable file in one of the folders in the $PATH environment variable called "firefox".
Are you talking about the PATH variable? It seems like you are.
In linux, you should be able to type: "echo $PATH" (without quotes) and get a ":"-separated list of locations where programs are located (like firefox).
If you need to add something to your path, you should be able to do:
export PATH=$PATH:/another/directory
In your shell (which is most likely bash)
You can also type:
which firefox
To display the location of the firefox executable.
Typically the shell is going to have an environment variable called $PATH set. This is just an ordered list of all the directories to look when somebody types in a command. As soon as it finds an executable file (by which I mean a file for which you have execute permissions, not a file ending in .exe) with the same name as whatever was typed, it will run that file. Common directories in $PATH might be /bin, /usr/local/bin, ~/bin, etc.
So, when you type 'firefox', the shell looks through all the directories in $PATH until it finds /usr/local/bin/firefox, which it then runs. To make your own programs run the same way, you'll either need to put them (or a symbolic link to them) in a directory that is likely to be in every user's path (/usr/local/bin/ is a good choice), or you'll need to get your users to add your program's directory to their $PATH.
For a more complete description, see the Wikipedia article about the $PATH variable.
As an alternative to the modification of $PATH mentioned earlier, you could also copy or link your executable in one of the directories already in your $PATH. more specifically, /usr/local/bin/ is available on most UNIX system for pretty much this purpose (installing software outside the default package management of the operating system).
It has to be in the path as everyone else mentioned, but you might also need to make it executable with something like this:
chmod +x /path/to/file
And if it's a script there's usually a shebang at the top that tells the os what to use to execute it:
#! /usr/bin/python
Often, large packages are installed in /opt with a wrapper script or link somewhere in the PATH. For example, on my system, Google Picasa is installed in /opt/google/picasa and there is a symlink at /usr/bin/picasa to /opt/google/picasa/3.0/picasa
Firefox is at /usr/bin/firefox on my system and that's a symlink to /usr/bin/firefox-3.0 which is itself a symlink to /usr/lib/firefox-3.0.11/firefox.sh - That shell file fumbles around until it finally runs /usr/lib/firefox-3.0.11/firefox (unless it finds a reason to do something else). That, finally, is a binary executable. So /usr/lib is where firefox is installed, for me.
You can use this command to find out where a program is:
type -a firefox
Then, you can find out what kind of file it is using this:
file /usr/bin/firefox
Also see the Filesystem Heirarchy Standard for more information about recommended locations for files and programs.