File can't be accessed on Linux cluster - linux

I am trying to run a job on a Linux cluster.
I am trying to read a file /storage1/fs1/Active/LP6005120-DNA_B02.bam.
I did chmod 755 /storage1/fs1/Active/LP6005120-DNA_B02.bam before submitting the job that tries to access this file. It ran for the first time, but when I try to submit the same job again, I am getting error saying ERROR, cannot read /storage1/fs1/Active/LP6005120-DNA_B02.bam. How do I resolve this issue?

Related

Writing command to persistent storage is hanging in Azure App Service for Linux Containers

We have a custom container image that we are trying to run within Azure App Service. The image has a shell script as an entry point, which executes a lot of commands some of which are file IO related. We need to turn on Persisten Storage by changing the configuration WEBSITES_ENABLE_APP_SERVICE_STORAGE = true.
The container is failing to start and by viewing the logs, we are seeing that the process hangs during the following statement:
echo "start"
tar cf - --one-file-system -C /usr/src/folder . | tar xf -
echo "end"
The statement is basically compressing a folder and extracting it in the /home/site/wwwroot directory, which is the one that is persisted within the App Service.
We can see start being logged in the logs, but we can't see end, which leads us to believe that the process is hanging on the tar statement, but we are seeing some of the files and folders being copied.
If we set WEBSITES_ENABLE_APP_SERVICE_STORAGE = false everything works as expected... of course until a restart of the container, where we loose all of our changes. Hence why we need Persistent Storage. Furthermore, this works ok on my local machine pointing to an external volume.
What gives?
After a lot of troubleshooting I figured the issue was the command was taking a long time to execute because files were being moved to another network location and the container was restarting before the command had time to finish. We changed the WEBSITES_CONTAINER_START_TIME_LIMIT to give it enough time to finish the command and everything started working as expected.

Gitlab Executor TTY

I'm trying to leverage Gitlab Runner to execute builds on remote machine. I have the runner and am using the shell executor to kick off builds locally. Unfortunately, I'm receiving the following error when I kick off something:
Using Shell executor...
standard in must be a tty
ERROR: Job failed: exit status 1
I suspect this is occurring because the user that I'm passing in with the --user parameter can access tty. Any thoughts on how to get this working? The shell type is bash.
Check first if GitLab is doing a sudo once connected to the remote machine, to change user.
If that is the case, with visudo (on that remote machine), do:
Defaults:[username] !requiretty
If not, as in here, check if this is a JVM initialization issue.

Pyspark write files to local on yarn cluster mode

I am trying to run my pyspark code. My destination directory is a local directory. The user with which I am submitting spark-submit command is the super user and has all privileges to read the file from hdfs and write the files to local.
The job is running without any error but there is no output directory or files getting created.
I have set the HADOOP_USER_NAME as super user in my spark code to avoid permission issue as well.
Can someone please help
If you are running in YARN cluster mode then the YARN ApplicationMaster is actually running on a node so will be writing out local to the node. If you find which node it was then you should find your output directory and files there.

Download file in jenkins job

I have a jenkins job which will execute node application. This job is configured to run on docker only during execution.
Is it possible to download file from node application everytime when job gets executed?
I tried using nodejs plugins to save and download file. File is getting saved in local but not able to download.
If your docker container runs some job and creates a file as the output of the job, and you want it available outside the container after the job is done, my suggestion is that you create the file in a location that is mapped to a host folder via the volume option. Run your docker container as follows:
sudo docker -d -v /my/host/folder:/my/location/inside/container mynodeapp:latest
Ensure that your node application writes the output file to the location /my/location/inside/container. When the job is completed, the output file can be accessed on the host-machine at /my/host/folder.

Trouble starting postgresql

I just got a copy of a VM running SLES10 in which I have Postgresql. I try to start it using the command "postgresql start" but it returns an error message: "could not start postmaster". On the old machine it works fine. The only things I did since copying the machine was to change its IP Address and name, install VMWare Tools and delete the postgresql logs directory /.../postgres/pg_log (I recreated it afterwards). I tried restarting the machine but I still get the error. What can I do to determine the cause of the problem?
What can I do to determine the cause of the problem?
Check out the Postgres and system log files
My bet is that there is some problem with accessing the data directory e.g. the account running the postmaster does not have read/write access to that directory.
Edit
You can try to start Postgres "manually" (as opposed to the system provided scripts) by running:
pg_ctl -D /path/to/data/dir start
then you might be able to see error messages on the console. Make sure you are running this as the postgres user

Resources