*I followed https://www.anchormen.nl/spark-docker/ and
When i executed the command ,apache startup fails saying "--" is not recognized.
I am very new to spark.Requesting help from our trusted community members.
bash /opt/spark/sbin/start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /opt/spark/logs/spark--org.apache.spark.deploy.master.Master-1-e6b8f9219a40.out
failed to launch: nice -n 0 /opt/spark/bin/spark-class org.apache.spark.deploy.master.Master --host e6b8f9219a40 --port 7077 --webui-port 8080
nohup: can't execute '--': No such file or directory
full log in /opt/spark/logs/spark--org.apache.spark.deploy.master.Master-1-e6b8f9219a40.out
*
Only difference from the article is that i used alpine linux,which am restricted to.
to verify i tried to cat the log file..and got the same error. alpine linux in docker is not recognizing "--" .am i doing something wrong?
[SOLVED] thanks Robert.
if anyone looking for answer add the following in your docker-file
RUN apk update && apk upgrade && apk add curl ca-certificates tar supervisor bash procps coreutils
Your problem is here in this line
The thing is that the alpine image comes with busybox, which is a replacement of many commands including nohup and ps. So alpine comes with a non-gnu nohup which cannot handle that --, and a ps which cannot handle -p.
So, install coreutils and procps packages before any call to apache spark scripts in order to have the version of nohup and ps that you need.
In Dockerfile or container command line:
RUN apk --update add coreutils procps
Related
First of all, let me state I'm not the most virtuous of Linux users, so bear with me...
Below is a brief run-down of all the steps I took. Ultimately the question/issue is is that it seems impossible for me to get a proper docker-compose installation downloaded.
Followed instructions to install docker https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html
sudo curl -L "https://github.com/docker/compose/releases/download/1.26.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
Tried 4 variations of the above command to try to install docker-compose. As shown in the URLs below.
https://www.codegrepper.com/code-examples/php/how+to+install+docker+compose+in+ec2
https://portal.cloud303.io/forum/aws-1/question/i-want-to-install-docker-compose-on-an-amazon-linux-2-ec2-instance-9
https://acloudxpert.com/how-to-install-docker-compose-on-amazon
https://gist.github.com/npearce/6f3c7826c7499587f00957fee62f8ee9
When typing "docker-compose", "sudo docker-compose" etc. All it will say is
"Line 1: Not: command not found".
It seems to be the issue that the docker-compose file is only 9kb in size. Because this is what I get back every time I use the above mentioned docker-compose install sequences.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 9 100 9 0 0 58 0 --:--:-- --:--:-- --:--:-- 58
This issue is sort of addressed here:
https://github.com/docker/compose/issues/6268
Where it is said that the OS is not supported or that we're running a 32bit instance, but of which seem to be strange because all the above tutorials are specifically for AWS EC2 Linux 2.
fwiw 'uname -m' returns aarch64.
So, does anyone have an idea of how to get a full-sized version of docker-compose instead of the 9kb file?
Thanks!
followed the link to install the docker-compose link
Basically, thee are only two steps as below:
sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose version
Amazon Linux is "just" the OS, but compiled binaries also depend on your processor architecture. For instance, EC2 t3a series are based on x86_64 architecture wheras the new t4g types are of aarch64 (which is the reason I run into as similar issue as you and ended up here). uname -m returns that identifier, in your case apparently aarch64.
Since the docker-compose github repo contains no released binary for this architeture (as opposed to x86_64), the resolved full download URL returns a "Not found" body instead of the expected binary, thus the described error message when you try to execute it.
As you already found out, there a multiple threads which discuss this issue. The only thing that eventually worked for me was to install docker-compose manually via python's packaga manager pip. Since this approach involves a couple of compilation steps, you need to add various additional OS packages and may face other mostly dependency related errors in the process. But the following steps eventually worked for me:
uname -s -m
1> Linux aarch64
cat /etc/system-release
1> Amazon Linux release 2 (Karoo)
sudo yum install -y python37 \
python3-devel.$(uname -m) libpython3.7-dev \
libffi-devel openssl-devel
# need gcc and friends
sudo yum groupinstall -y "Development Tools"
# make sure pip is up2date
sudo python3 -m pip install -U pip
python3 -m pip install docker-compose
docker-compose --version
1> docker-compose version 1.27.4, build unknown
Hope it works for you as well. Good Luck!
You can start from the scratch on Amazon Linux ec2 instance for installing Docker by following the step:
sudo yum update -y
sudo amazon-linux-extras install docker
sudo yum install docker
sudo service docker start
sudo usermod -a -G docker ec2-user
Then logout from the instance and login again to verify the installation of Docker
docker info
To install Docker-compose follow the below steps:
sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose version
I tried everything here mentioned, in the end, it was a problem with the symlink. The official docker page gave me the solution.
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
If it doesn't work after that command, check if you can find docker in your bin folder, e.g. with: ls /usr/local/bin/
If you can see docker-compose there, you almost made it.
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
An easy way to check if docker-compose is there and Linux able to find it, is to use `which docker-compose, also if it is linked correctly, you will get the path to docker-compose as a response.
This worked for me on a AWS EC2 instance with Linux2 as OS.
I faced the same issue. My configuration is T4Large - ARM64 - Ubuntu Server 20.04 LTS and I resolved using the following command
$ sudo apt install docker-compose
This will install the 1.25.0 version of docker compose which is a bit older. At the time of writing this answer the current version is 1.29.1
Another alternative is to use Docker Compose as Docker Container.
Use the following commands to run Docker Compose Container
$ sudo curl -L --fail https://raw.githubusercontent.com/linuxserver/docker-docker-compose/master/run.sh -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
This should work on any Linux distro and on architecture like x86_64, armv7l/armhf, and aarch64/armv8/arm64 if you already have docker installed.
The important point to note is, this runs compose inside a container and does not require installing python on the host system.
Docker hub reference: https://hub.docker.com/r/linuxserver/docker-compose
Installing Docker-Compose in EC2 instance follow below steps
1st command: # sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
2nd command: # sudo chmod +x /usr/local/bin/docker-compose
3rd command: # ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
4th Command to verify docker compose: # docker-compose version
It will install latest version of docker-compose. 3rd command is necessary.
This will do everything you need.
Cut. paste. done.
cat ec2-al2-docker-compose.sh
#the following script will install docker and docker compose on amazon linux 2
# run this script then
# sudo docker ps; docker-compose up
# note docker-compose up doesnt use sudo
sudo yum update -y
sudo yum install docker git -y
sudo usermod -a -G docker ec2-user
id ec2-user
newgrp docker
sudo yum install python3-pip -y
sudo pip3 install docker-compose
sudo systemctl enable docker.service
sudo systemctl start docker.service
I have Dockerfile that looks like this
FROM alpine:3.7
WORKDIR /home/tmp
RUN apk add autoconf && apk add py-pip && apk add python3 &&\
pip install --upgrade pip && pip install wheel
Originally I wanted to execute a .sh script on startup (via ENTRYPOINT) and immediately destroy the container. However as it failed to find the file I decided to do that manually.
I run container like this
docker run -it --rm -v c:/projects/mega-nz-sdk:/home/tmp mega_sdk_python
And it connects me to bash in the container.
In the list of files I can see the script I want to execute
/home/tmp # ls
Dockerfile compile.sh sdk-develop
/home/tmp #
However when I try to run it it cannot find the script
/home/tmp # ./compile.sh
/bin/sh: ./compile.sh: not found
/home/tmp #
What is the problem?
Script compile.sh looks like this
#!/bin/bash
cd sdk-develop
sh autogen.sh
./configure --disable-silent-rules --enable-python --disable-examples &&\
make
cd /bindings/python
python setup.py bdist_wheel
Ideally I would like to execute during instantiation of the container in order to have already configured container on startup (without need to run script each I run the container).
It seems in order to execute my .sh file I need to run it like this
sh compile.sh
So I added
CMD ["sh", "compile.sh"]
And I started to work (though failed with other errors like missing make etc. but that's due to missing packages in Alpine Linux itself so a separate matter).
Guess It is something to do with Alpine Linux itself. But I am not sure.
Alpine Linux is a very minimal distribution; it includes a minimal version of most Unix tools that conform to the POSIX specification, but no more. In particular it does not include GNU Bash.
Your script doesn't actually use any special Bash features, so it would be enough to change the first line of the script to run the default system Bourne shell
#!/bin/sh
Using the Alpine apk package manager to install bash would work too, but it's not necessary for what you're showing here.
Usually you'd run the sorts of "compile" commands you show during the course of building an image, not when the image starts up. I'd expect a much more typical Dockerfile to COPY the application source code and in then RUN the commands you show. That would happen just once, when you docker build the image, and not every time you want to run the packaged application.
you mounted the directory from your win host to the docker machine. I BELIEVE this is a permission problem - the file is not executable flagged.
Show detailed listing
# ls -lh
copy the folder to internal dir and add executable bit
cp /home/tmp /home/tmp2 -r
chmod +x /home/tmp2/*.sh
/home/tmp2/compile.sh
hello i have a problem with docker, recently i make dockerfile for create a image of "mosquitto-mqtt" to make my own broken mqtt with ssl protection. i build dockerfile all is good, i don't have a problem but if i run a new container with " docker run -itd --name broken ce69ee4b2f4e" a container run and exit automaticly, and if a check log all is good "[ ok .] Starting network daemon:: mosquitto.". i don't have why ? check my dockerfile. i need help to solve it, thanks you
#Download base image debian
FROM debian:latest
#Update system
RUN apt-get update -y
#Install Wget and gnup2
RUN apt-get install wget -y && apt-get install gnupg2 -y
#Download and add key
RUN wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
RUN apt-key add mosquitto-repo.gpg.key
RUN rm mosquitto-repo.gpg.key
## append apt mirror for debian
RUN echo "# mirror" >> /etc/apt/source.list
RUN echo "deb http://repo.mosquitto.org/debian stretch main" >> /etc/apt/source.list
#Update and upgrade system
RUN apt-get update -y && apt-get upgrade -y
#install mosquitto
RUN apt-get install mosquitto -y
#Copy file configuration
COPY mosquitto.conf /etc/mosquitto
#Copy certificate folder
COPY certs/mosquitto-ca.crt /etc/mosquitto/certs
COPY certs/mosquitto-server.crt /etc/mosquitto/certs
COPY certs/mosquitto-server.key /etc/mosquitto/certs
#Run command
ENTRYPOINT ["/etc/init.d/mosquitto", "start"]
log print
[ ok .] Starting network daemon:: mosquitto.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d00bd23ae2d6 ce69ee4b2f4e "/etc/init.d/mosquit…" 9 minutes ago Exited (0) 9 minutes ago broken
Containers are a wrapper around a process, and when that process exits, the container exits. In this case:
ENTRYPOINT ["/etc/init.d/mosquitto", "start"]
That process is /etc/init.d/mosquitto which almost certainly runs, spawns a daemon in the background, and exits (standard for anything in init.d). You should instead run mosquito directly with foreground options if available.
If that's some possible, something like supervisord would be a less than optimal fallback, with the ability to watch a background daemon.
And if neither of those work, you can run your command from a script that ends with a tail -f /dev/null, but that would be the worst option since you ignore any errors.
it works ! i found the solution, it just need to add "-C" on command and specify directory
this is a good method
ENTRYPOINT ["mosquitto", "-c", "/etc/mosquitto/mosquitto.conf"]
thanks all to help Me!
I am trying to run docker-compose using sudo.
I have both docker and docker-compose installed on Ubuntu 16.01.
Due to an error while trying to download compose using curl, I ended up installing it using pip.
Docker version 1.12.0, build 8eab29e
docker-compose version 1.8.0, build 94f7016
Yet, when I try to run docker-compose with sudo I get the following (using sudo with docker is fine)
sudo: docker-compose: command not found
I suppose there are differing definitions of what 'installed' means. I have been using docker-compose on the same computer that claims it is not installed.
$ dpkg -s docker-compose
dpkg-query: package 'docker-compose' is not installed and no information is available
Use dpkg --info (= dpkg-deb --info) to examine archive files,
and dpkg --contents (= dpkg-deb --contents) to list their contents.
$ whereis docker-compose
docker-compose: /home/user/.local/bin/docker-compose
$ pip show --files docker-compose
---
Metadata-Version: 2.0
Name: docker-compose
Version: 1.8.0
Summary: Multi-container orchestration for Docker
Home-page: https://www.docker.com/
Author: Docker, Inc.
Author-email: UNKNOWN
Installer: pip
License: Apache License 2.0
Location: /home/anton/.local/lib/python2.7/site-packages
Requires: six, jsonschema, enum34, cached-property, websocket-client, docker-py, requests, docopt, dockerpty, PyYAML, texttable
Classifiers:
Development Status :: 5 - Production/Stable
Environment :: Console
Intended Audience :: Developers
License :: OSI Approved :: Apache Software License
Programming Language :: Python :: 2
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
Programming Language :: Python :: 3.4
Files:
../../../bin/docker-compose
compose/GITSHA
compose/__init__.py
compose/__init__.pyc
compose/__main__.py
compose/__main__.pyc
compose/bundle.py
compose/bundle.pyc
compose/cli/__init__.py
compose/cli/__init__.pyc
compose/cli/colors.py
compose/cli/colors.pyc
compose/cli/command.py
compose/cli/command.pyc
compose/cli/docker_client.py
compose/cli/docker_client.pyc
compose/cli/docopt_command.py
compose/cli/docopt_command.pyc
compose/cli/errors.py
compose/cli/errors.pyc
compose/cli/formatter.py
compose/cli/formatter.pyc
compose/cli/log_printer.py
compose/cli/log_printer.pyc
compose/cli/main.py
compose/cli/main.pyc
compose/cli/signals.py
compose/cli/signals.pyc
compose/cli/utils.py
compose/cli/utils.pyc
compose/cli/verbose_proxy.py
compose/cli/verbose_proxy.pyc
compose/config/__init__.py
compose/config/__init__.pyc
compose/config/config.py
compose/config/config.pyc
compose/config/config_schema_v1.json
compose/config/config_schema_v2.0.json
compose/config/environment.py
compose/config/environment.pyc
compose/config/errors.py
compose/config/errors.pyc
compose/config/interpolation.py
compose/config/interpolation.pyc
compose/config/serialize.py
compose/config/serialize.pyc
compose/config/sort_services.py
compose/config/sort_services.pyc
compose/config/types.py
compose/config/types.pyc
compose/config/validation.py
compose/config/validation.pyc
compose/const.py
compose/const.pyc
compose/container.py
compose/container.pyc
compose/errors.py
compose/errors.pyc
compose/network.py
compose/network.pyc
compose/parallel.py
compose/parallel.pyc
compose/progress_stream.py
compose/progress_stream.pyc
compose/project.py
compose/project.pyc
compose/service.py
compose/service.pyc
compose/state.py
compose/state.pyc
compose/utils.py
compose/utils.pyc
compose/volume.py
compose/volume.pyc
docker_compose-1.8.0.dist-info/DESCRIPTION.rst
docker_compose-1.8.0.dist-info/INSTALLER
docker_compose-1.8.0.dist-info/METADATA
docker_compose-1.8.0.dist-info/RECORD
docker_compose-1.8.0.dist-info/WHEEL
docker_compose-1.8.0.dist-info/entry_points.txt
docker_compose-1.8.0.dist-info/metadata.json
docker_compose-1.8.0.dist-info/pbr.json
docker_compose-1.8.0.dist-info/top_level.txt
Entry-points:
[console_scripts]
docker-compose=compose.cli.main:main
I have tried the following - but still get the same error:
$ chmod +x /home/username/.local/bin/docker-compose
$ chmod +x /home/username/.local/lib/python2.7/site-packages
On Ubuntu 16.04
Here's how I fixed this issue: Refer Docker Compose documentation
sudo curl -L https://github.com/docker/compose/releases/download/1.21.0/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
After you do the curl command , it'll put docker-compose into the
/usr/local/bin
which is not on the PATH.
To fix it, create a symbolic link:
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
And now if you do:
docker-compose --version
You'll see that docker-compose is now on the PATH
The output of dpkg -s ... demonstrates that docker-compose is not installed from a package. Without more information from you there are at least two possibilities:
docker-compose simply isn't installed at all, and you need to install it.
The solution here is simple: install docker-compose.
docker-compose is installed in your $HOME directory (or other location not on root's $PATH).
There are several solution in this case. The easiest is probably to replace:
sudo docker-compose ...
With:
sudo `which docker-compose` ...
This will call sudo with the full path to docker-compose.
You could alternatively install docker-compose into a system-wide directory, such as /usr/local/bin.
If you have tried installing via the official docker-compose v1 page, where you need to download the binary using curl:
sudo curl -L https://github.com/docker/compose/releases/download/1.29.2/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
Then do not forget to add executable flag to the binary:
sudo chmod +x /usr/local/bin/docker-compose
If docker-compose is installed using python-pip
sudo apt-get -y install python-pip
sudo pip install docker-compose
try using pip show --files docker-compose to see where it is installed.
If docker-compose is installed in user path, then try:
sudo "PATH=$PATH" docker-compose
As I see from your updated post, docker-compose is installed in user path /home/user/.local/bin and if this path is not in your local path $PATH, then try:
sudo "PATH=$PATH:/home/user/.local/bin" docker-compose
I have same issue , i solved issue :
Following installs Docker Compose v1:
step-1 : download docker-compose using following command.
1. sudo su
2. sudo curl -L https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
Step-2 : Run command
chmod +x /usr/local/bin/docker-compose
Step-3 : Check docker-compose version
docker-compose --version
I will leave this here as a possible fix, worked for me at least and might help others. Pretty sure this would be a linux only fix.
I decided to not go with the pip install and go with the github version (option one on the installation guide).
Instead of placing the copied docker-compose directory into /usr/local/bin/docker-compose from the curl/github command, I went with /usr/bin/docker-compose which is the location of Docker itself and will force the program to run in root. So it works in root and sudo but now won't work without sudo so the opposite effect which is what you want to run it as a user anyways.
If docker-compose is installed for your user but not installed for root user and if you need to run it only once and forget about it afterwords perform the next actions:
Find out path to docker-compose:
which docker-compose
Run the command specifying full path to docker-compose from the previous command, eg:
sudo /home/your-user/your-path-to-compose/docker-compose up
On Ubuntu just run
sudo apt install docker-compose
This worked on Ubuntu 22*
You might be using a deprecated code. Try writing docker compose instead of docker-compose.
Simple, just reinstall the docker-compose. It will fix the issue.
sudo apt install docker-compose
There are too many pending edits on Stack Overflow, so I'm submitting a new answer that supplements earlier answers.
I propose the answer is syntax: docker compose vs docker-compose depending on how you have it installed, which is the same proposed answer by ibrahim shazly on Sep 27, 2022 and Gaurav Kumar Verma on Sep 29, 2022.
Here is an explanation. I followed https://docs.docker.com/engine/install/ubuntu/#install-docker-engine to install Docker on Ubuntu 22 LTS. Then, I attempted to run docker compose like so:
$ docker-compose up -d myserver
Command 'docker-compose' not found, but can be installed with:
...
As Gaurav Kumar Verma points out, https://docs.docker.com/compose/install/other/ has a disclaimer:
Note that Compose standalone uses the -compose syntax instead of the current standard syntax compose.
For example type docker-compose up when using Compose standalone, instead of docker compose up.
Thus, I ran:
$ docker compose up -d myserver
And it ran successfully.
Or, just add your binary path into the PATH. At the end of the bashrc:
...
export PATH=$PATH:/home/user/.local/bin/
save the file and run:
source .bashrc
and the command will work.
docker-compose is not installed. Please try to install it first from their official documentation
Whenever this type of error occurs please try to verify the installation of the apps by typing which command_name it will search for the executables and show you the location.
Had to delete a large log file then restart docker-compose for the space to clear up.
I used a script shell that cron runs once a day:
rm /var/log/cron/log_prodTEST.log
cd /opt/test/metrics
/usr/local/bin/docker-compose up --build --force-recreate -d -T
Apperntly by using the full path to docker-compose, I could use it inside the corn shell
you might try run the code by using docker compose instead of docker-compose
if it work you might want to check this link her
https://docs.docker.com/compose/install/other/
sudo curl -L "https://github.com/docker/compose/releases/download/v2.12.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo mv /usr/local/bin/docker-compose /usr/bin/docker-compose
sudo chmod +x /usr/bin/docker-compose
When i run docker build -t example . on the below im getting an error
FROM ruby:2.1
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
ENV NVM_DIR /usr/local/nvm
ENV NODE_VERSION 4.4.2
RUN curl https://raw.githubusercontent.com/creationix/nvm/v0.31.0/install.sh | bash \
&& source $NVM_DIR/nvm.sh \
&& nvm install $NODE_VERSION \
&& nvm alias default $NODE_VERSION \
&& nvm use default
ENV NODE_PATH $NVM_DIR/v$NODE_VERSION/lib/node_modules
ENV PATH $NVM_DIR/v$NODE_VERSION/bin:$PATH
RUN node -v
I get the following error:
Step 9 : RUN node -v ---> Running in 6e3fac36d2fc /bin/sh: node:
command not found The command '/bin/sh -c node -v' returned a non-zero
code: 127
Can't understand why node is not found in the path. i tried executing the nvm.sh file as well but it didnt have an effect.
Node version manager is an excellent application for switching versions of Node.js on your development machine, but Docker begs a specific kind of image/container design that is meant to be both ephemeral and stripped down to the bare essentials in order to support the "best practice" of microservices. Docker is just a fancy way to run a process, not a full VM. That last sentence has helped me a lot in how to think about Docker. And so here, you can make things easier on yourself by creating different versions of your image, instead of making one container with many versions of Node.js inside of it. This way, you can reference the Node version you want to run inside of your docker run command instead of trying to feed in environment variables trying to get NVM to select the right version. For example:
docker build -t=jamescharlesworth-node:4.x-latest .
And of course your Dockerfile will have in it the install command in your RUN directive that you mention in the comments:
RUN curl -sL https://deb.nodesource.com/setup_4.x | bash -
RUN apt-get install -y nodejs