I've been requested to run a code on a file in a file system I had to create.
(I created the fs using mkfs and then mounted it with another directory: /home/may/new_place (the original fs appears on my desktop - 8.6GB filesystem)
My question is, can you even create a file in a filesystem? I can't even transfer a file into it, so can't execute my code.
I'm really new to this.. thank you all
(P.S. I'm using linux xubunto OS)
are you able to touch any files -- with touch command -- also open a sample file -- write echo $HOSTNAME - save it - chmod u+x and run it -- see if you are able to execute the file
Related
Running Ubuntu 18.04 LTS. I've put a script in the following location:
/usr/share/initramfs-tools/scripts/init-bottom/dothis
dothis is set as +x. I've run sudo update-initramfs -u which appears to update the initramfs contents just fine. I've looked at /boot/grub/grub.cfg and see the expected initrd file configured.
Running lsinitramfs on the file DOES show the script as having been added.
I cannot however find any evidence that dothis is being run on boot. As per documentation online, the root filesystem SHOULD be in place by the time the /init-bottom scripts are run, which should be the only thing required for my 'hello world' to work, as it outputs to a file on the root file system.
Is there some other step that needs to be done in order to get this script to run, or show whether it's being run? I've tried to simply output text to a file in /tmp but this is not showing up either.
Since you did not provide more details, I have to guess. Does your dothis script write to /tmp inside the initramfs? The root file system is mounted at /root and the initramfs uses switch_root to switch to it (as last step of the initramfs). Then all files written to /tmp are lost. If you let your script write to /root/root/log, this file will appear in /root/log on the booted system.
To further debug your issue, you can boot your system with the kernel parameter break=bottom. Then you get a shell where you can run your script manually. For even faster debug cycles, you can use a virtual machine.
I'm watching a folder using incrontab with the command in the incrontab -e editor:
/media/pi/VDRIVE IN_CLOSE_WRITE sudo mv $#/$# /media/pi/VDRIVE/ready/$#
The watched folder is relieving a file over the network from another machine—the file shows up OK and appears to trigger the incrontab job presumably once the copy process has closed the file, but the mv command results in a 0 bytes file in the destination folder with the correct name.
All run as root.
It seems that there is a bug in Samba on OSX which results in two events when writing to a shared folder on the network. This makes incrontab pretty unworkable when working with OSX computers (more recent OS 10.7 up).
So when OSX writes a file to the Linux samba share, there are two events, the first one triggers the mv action before the file has finished actually writing. Its a bug in OSXs SAMBA implementation.
In the end I used inotify to write events to a log file (of which there are always two), then scanned the file for two instances of the event before performing the action.
Another strategy was to use LSOF on a cron routine that will just ignore any files open for writing.
I am hoping that a more experienced set of eyes will find something obvious that I am missing or will be able to help me work around the errors that mv and rsync are producing. Up for the challenge?
Basic idea:
I have a bash script in which I am automating the move of files from one directory to another.
The problem:
When I run the script, periodically I get the following error from the mv command:
mv: cannot stat `/shares/directory with spaces/test file.txt': No such file or directory. The error code from the vm command is 1. Even more odd, the file move actually succeeds sometimes.
In addition, I have a branch of logic in the script that will alternately use rsync to move/copy specific files (from the same local file system source and destination as the mv command mentioned above). I get a similar error related to the stat() system call:
rsync: link_stat "/shares/directory with spaces/test file.txt" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9]
This error does not always manifest itself when the script is run. Sometimes it completes the file move without complaint, while other times it will return the error consistently when the script is run successive times.
There is one additional ingredient you should be aware of (and I am growing to suspect this as a key ingredient in my grief): the directory /shares/ is a directory that is being monitored by an installation of Dropbox -- meaning it is watched and mirrored by an installation of Dropbox. At this point, I am unable to determine if dropboxd is somehow locking the file, or the like, such that it cannot be stat-ed. To be clear, the files are eventually freed from this state without further intervention and are mv-able.
The code:
mv -v --no-clobber "${SOURCEPATH}${item}" "${DESTINATIONPATH}${item}"
More info:
The following might, or might not, be relevant:
mount indicates the filesystem is ext4
Presumably, ownership and permissions shouldn't be an issue as the script is being run by root. Especially if the file system is not fuse-based.
The base "directory" in the path (e.g. /shares/) is a symlink to another directory on the same file system.
The flavor of Linux is Debian.
Troubleshooting:
In trying to eliminate any issues with the variable expansion or their contents, I tried hardwiring the bash script like such:
mv -v --no-clobber "/shares/directory with spaces/test file.txt" "/new destination/directory with spaces/test file.txt" after verifying via ls -al that "test file.txt" existed. For reference the permissions were: -rw-r--r--
Unfortunately, this too results in the same error.
Other possible issues I could think of and what I have done to try to rule them out:
>> possible issue: slow HDD (or drive is in low power mode) or external USB drive
>> findings: The drives are all local SATA disks set to not park heads. In addition, even when forcing a consistent read from the file system, the same error happens
>> possible issue: non-Linux, NFS or fuse-based file system
>> findings: nope, source and destination are on the same local file system and mount indicates the file system is ext4
>> possible issue: white space or other unprintable chars in the file path
>> findings: verified that the source and destination paths where properly wrapped in quotes
>> possible issue: continuation issues after escaped newline (space after \ in wrapped command)
>> findings: made sure the command was all on one line, still the same error
>> possible issue: globbing (use of * in specifying the files to move)
>> findings: nope, each file is specified directly by path and name
>> possible issue: path confusion from the use of local path
>> findings: nope, file paths are fully qualified starting from /
>> possible issue: the files are not actually in the path specified
>> findings: nope, verified the file existed right prior to executing the script via ls -al
>> possible issue: somehow the --no-clobber of mv was causing issues
>> findings: nope, tried it without, same error
>> possible issue: only files created via Dropbox sync to the file system are problematic
>> findings: nope, created a local file directly via touch new-local-file.txt and it too produced the same stat() error
My analysis:
The fact that mv and rsync produce similar stat() errors leads me to believe:
there is some systemic underlying boundary case (e.g. file permissions/ownership or file busy) that is not accounted for in the bash script; or
the same bug is plaguing me in both the mv and the rsync scenarios.
Desired outcomes:
1. The root cause of the intermittent errors can be identified.
2. The root cause can be resolved or worked around.
3. The bash script can be improved to gracefully handle when the error occurs.
So, with a lot more troubleshooting I found an errant rsync statement some 200 lines earlier in the script that was conditionally executed (thus the seeming inconsistent behavior). That rsync --archive ... statement was being passed /shares/ as its source directory, therefore it effected the /shares/directory with spaces/ subdirectory. That subdirectory was the ${SOURCEPATH} of the troubling mv command mentioned in my post above.
Ultimately, it was a missing --dry-run flag on the rsync --archive ... statement that causing the trampling of the files that the script later expected to pass to mv to process.
Thanks for all who took the time to read my post. Though I am bummed to have spent my and your time on what turned out to be a bug in my script, it is reassuring to know that:
- computers are not irrational
- I am not insane
- there is not some nefarious, deep rooted bug in the linux file system
For those that stumble upon this post in the future because you are experiencing an error of cannot stat, please read my troubleshooting notes above. Much research went into that list. One of those might be your issue. If not, keep debugging, there is an explanation. Good luck!
In my work, I use 2 Linux servers.
The first one is used for web-crawling and create it as a text file.
The other one is used for analyzing the text file from the web crawler.
So the issue is that when a text file is created on web-crawling server,
it needs to be transferred automatically on the analysis server.
I've used shell programming guides referring some tips,
and set up the crawling server to be able to execute the scp command without requiring the password (By using ssh-keygen command, Add ssh-key on authorized_keys file located in /root/.ssh directory)
But I cannot figure out how to programmatically transfer the file when it is created.
My job position is just data analyze (Not programming)
So, the lack of background programming knowledge is my big concern
If there is a way to trigger the scp to copy the file when it is created, please let me know.
You could use inotifywait to monitor the directory and run a command every time a file is created in the directory. In this case, you would fire off the scp command. IF you have it set up to not prompt for the password, you should be all set.
inotifywait -mrq -e CREATE --format %w%f /path/to/dir | while read FILE; do scp "$FILE"analysis_server:/path/on/anaylsis/server/; done
You can find out more about inotifywait at http://techarena51.com/index.php/inotify-tools-example/
Here is my problem,
In Windows I am making a zip file in which there is a text .sh file which is supposed to be executed in Linux.
The user on the other end opens the zip file in Linux and tries to execute the .sh file but the execute permission is gone. So the user has to do it manually ( like explained here:add execute permission.
How can I in Windows make the .sh executable and add it to a zip file so that when the zip file opens in linux the .sh file still retains its execute permission ( so that user doesn't have to do it manually)
As far as I know the permission system in Linux is set up in such a way to prevent exactly what you are trying to accomplish.
I think the best you can do is to give your Linux user a custom unzip one-liner to run on the prompt:
unzip zip_name.zip && chmod +x script_name.sh
If there are multiple scripts that you need to give execute permission to, write a grant_perms.sh as follows:
#!/bin/bash
# file: grant_perms.sh
chmod +x script_1.sh
chmod +x script_2.sh
...
chmod +x script_n.sh
(You can put the scripts all on one line for chmod, but I found separate lines easier to work with in vim and with shell script commands.)
And now your unzip one-liner becomes:
unzip zip_name.zip && source grant_perms.sh
Note that since you are using source to run grant_perms.sh, it doesn't need execute permission
The ZIP file format does allow to store the permission bits, but Windows programs normally ignore it.
The zip utility on Cygwin however does preserve the x bit, just like it does on Linux.
If you do not want to use Cygwin, you can take a source code and tweak it so that all *.sh files get the executable bit set.
Or write a script like explained here
This is possible using the Info-Zip open-source Zip utilities. If unzip is run with the -X parameter, it will attempt to preserve the original permissions. If the source filesystem was NTFS and the destination is a Unix one, it will attempt to translate from one to the other. I do not have a Windows system available right now to test the translation, so you will have to experiment with which group needs to be awarded execute permissions. It'll be something like "Users" or "Any user"
Use my windows command line utility zip_exec.zip to set the executable flag for linux/unix and mac (tested on files created with Windows Explorer and 7zip). The cpp source is also available. I searched the internet a lot before making my own utility. It can be modified to set any file attribute.
This is not possible. Linux permissions and windows permissions do not translate. They are machine specific. It would be a security hole to allow permissions to be set on files before they even arrive on the target system.