I have an issue with Puppet and want to understand it better. When I run puppet agent -t on my client it runs and recognises an Exec command which is triggered from a refresh event.
The Exec doesn't execute at all and I have noticed that a little later in the /var/log/messages there is another input which says
Unscheduling all events on Stage [main]
What I need to know is, is it this Unscheduling that is stopping my Exec command from running?
Related
I use Gitlab for doing Continuous Integration and Development and all of a sudden I get this error message "There has been a runner system failure, please try again"
There's no real error message or error code.
I've tried restarting the gitlab runner, using gitlab-runner restart, I've done a reboot of the server its running on but I keep getting this error message on Gitlab whenever I push a code change.
After a couple of hours, I realized the issue is that the server that Gitlab Runner is running on has no space left.
I logged into the server in question. Looked at the Gitlab log file using the following command:
journalctl -u gitlab-runner
And it showed me the following logs:
May 21 08:20:41 gitlab-runner[18936]: Checking for jobs... received job=178911 repo_url=https://.......git runner=f540b942
May 21 08:20:41 gitlab-runner-01 gitlab-runner[18936]: WARNING: Failed to process runner builds=0 error=open /tmp/trace543210445: no space left on device executor=docker runner=f540b942
To fix this issue I ran docker conatiner prune which clears out stopped containers.
Alternatively you could use docker system prune which would remove all unused objects.
See https://linuxize.com/post/how-to-remove-docker-images-containers-volumes-and-networks/ for more information about those docker commands.
Afterwards, I no longer got the error on Gitlab when pushing changes.
I'm trying to leverage Gitlab Runner to execute builds on remote machine. I have the runner and am using the shell executor to kick off builds locally. Unfortunately, I'm receiving the following error when I kick off something:
Using Shell executor...
standard in must be a tty
ERROR: Job failed: exit status 1
I suspect this is occurring because the user that I'm passing in with the --user parameter can access tty. Any thoughts on how to get this working? The shell type is bash.
Check first if GitLab is doing a sudo once connected to the remote machine, to change user.
If that is the case, with visudo (on that remote machine), do:
Defaults:[username] !requiretty
If not, as in here, check if this is a JVM initialization issue.
I'm trying to get airflow working to better orchestrate an etl process. When I make changes to a dag in my dags folder, I often have to restart the scheduler with
airflow scheduler
before the changes are visible in the UI. I would like to run the scheduler as a daemon process with
airflow scheduler -D
but we I try to do so, I get a message saying
[2018-10-17 14:13:54,769] {jobs.py:580} ERROR -
Cannot use more than 1 thread when using sqlite. Setting max_threads to 1
I think this error pops up because the scheduler is already running as a daemon. However, when I try to find out where the scheduler is being run with
lsof -i
I don't get any results.
Question: Why am I not able to restart the scheduler with airflow scheduler -D. Why does the scheduler restart with airflow webserver? How do I successfully kill the process that is preventing me to run airflow scheduler -D?
Run ps aux | grep airflow and check if airflow webserver or airflow scheduler processes are running. If they are kill them and rerun using airflow scheduler -D
You need to clear out the airflow-scheduler.pid file at $AIRFLOW_HOME. The stale pid file from the daemon will prevent you to start another scheduler process.
If you just restart your webserver, the dag changes gets reflected in UI. No need to restart scheduler for the same.
Applicable for 1.8 and 1.10.3. Cant comment for latest 1.10.10.
I know docker only listens to pid 1 and in case that pid exits (or turns into a daemon) it thinks the program exited and the container is shut down.
When apache-spark is started the ./start-master.sh script how can I kept the container running?
I do not think: while true; do sleep 1000; done is an appropriate solution.
E.g. I used command: sbin/start-master.sh to start the master. But it keeps shutting down.
How to keep it running when started with docker-compose?
As mentioned in "Use of Supervisor in docker", you could use phusion/baseimage-docker as a base image in which you can register scripts as "services".
The my_init script included in that image will take care of the exit signals management.
And the processes launched by start-master.sh would still be running.
Again, that supposes you are building your apache-spark image starting from phusion/baseimage-docker.
As commented by thaJeztah, using an existing image works too: gettyimages/spark/~/dockerfile/. Its default CMD will keep the container running.
Both options are cleaner than relying on a tail -f trick, which won't handle the kill/exit signals gracefully.
Here is another solution.
Create a file spark-env.sh with the following contents and copy it into the spark conf directory.
SPARK_NO_DAEMONIZE=true
If your CMD in the Dockerfile looks like this:
CMD ["/spark/sbin/start-master.sh"]
the container will not exit.
tail -f -n 50 /path/to/spark/logfile
This will keep the container alive and also provide useful info if you run -it interactive mode. You can run -d detached and it will stay alive.
I have this in my event log for one of my nodes in the puppet dash board:
Changed (1)
Service[openstack-keystone] (/etc/puppetlabs/puppet/modules/keystone/manifests/init.pp:129)
Property Message
ensure ensure changed 'stopped' to 'running'
But how can I see what actually command puppet is using to change the service's state from stopped to running?
And how can I change it, if I don't think puppet is doing the correct thing?
You can run puppet agent -t --debug to manually start a puppet run and see the commands being run.
To change the commands, you can consider specifying the provider or the start, stop, status, and restart commands on the service resource. Check out the type reference for more information on the service type's parameters.
1)If you want to see the background work of puppet means how it is applying the catalog.
step1) Stop the puppet master and client daemon. eg:/etc/init.d/puppetmaster stop.
step2) Run the puppet master and puppet agent as a foreground process to see the
- puppet master --no-daemonize (run master as foreground process)
- puppet master --debug --no-daemonize (To debug the puppet master)
- puppet agent --no-daemonize (run agent as foreground)
- puppet agent --debug --no-daemonize (run as foreground and debug)
2) If you think puppet is not doing this properly, you can write your own DSL with puppet types and provider or go with the EXEC to execute commands.Even though if you feel that it is not working as yo then you can write the script to execute on agent nodes.