I have been trying to start an already launched EC2 instance via python. I have configured AWS CLI from command prompt using the command below
aws configure
aws_access_key_id = MY_ACCESS_KEY
aws_secret_access_key = MY_SECRET_KEY
region=us-west-2b
output=Table
Now I used the following code from Spyder IDE of Anaconda
import boto3
instanceID = 'i-XXXXXXXXXXad'
ec2 = boto3.client('ec2', region_name='us-west-2b')
ec2.start_instances(InstanceIds=['i-XXXXXXXXXad'])
This gives the following error
EndpointConnectionError: Could not connect to the endpoint URL: "https://ec2.us-west-2b.amazonaws.com/"
I have been trying to debug the error for hours now, any kind of help will be useful. Also, I have a .pem as well as .ppk file created to start the instance via Putty, the .ppk file also has a paraphrase, do I need to do any kind of additional steps for this?
region=us-west-2b
is not a region, it is an availability zone. Try:
region=us-west-2
You can test by:
$ host ec2.us-west-2b.amazonaws.com
Host ec2.us-west-2b.amazonaws.com not found: 3(NXDOMAIN)
$ host ec2.us-west-2.amazonaws.com
ec2.us-west-2.amazonaws.com has address 54.240.251.131
Related
Wanted to learn AWS and found the tutorial Build a Serverless Web Application. In my research the closest Q&A I could find for my issue was Unable to locate credentials aws cli.
My process has been:
Created a repo in Github
Navigated to IAM and created a user trainer. Tutorial didn't specify policies so chose AdministratorAccess. Per instructions went the Security credentials and Create access key. Downloaded the file locally.
Went to Configuration basics and did Importing a key pair via .CSV file with the command of:
aws configure import --csv file:///Users/path/to/file/aws-training.csv
params:
User name: trainer
Access key ID: ****57
Secret access key: *****1b
but then found that the file didn't contain region or format so did:
aws configure --profile trainer
and re-did all values based on the CSV (Quick Setup):
AWS Access Key ID: ****57
AWS Secret Access Key: *****1b
Default region name: us-east-1
Default output format: json
Made sure to reboot my terminal and locally in a directory I run the command:
aws s3 cp s3://wildrydes-us-east-1/WebApplication/1_StaticWebHosting/website ./ --recursive
The terminal has a delay then throws:
fatal error: Unable to locate credentials
Research
Q&As I've read through to try and see if I could diagnose the problem:
aws cli with shell script: upload failed: Unable to locate credentials
Bash with AWS CLI - unable to locate credentials
Unable to locate credentials aws cli
Unable to locate credentials in boto3 AWS
Get "fatal error: Unable to locate credentials" when I'm copying file from S3 to EC2 using aws cli
Unable to locate credentials when trying to copy files from s3-bucket to my ec2-instance
How can I resolve my error of Unable to locate credentials and what am I doing wrong or misunderstanding?
Per the comment:
Check the content of ~/.aws/credentials and ~/.aws/config
credentials
command:
nano ~/.aws/credentials
renders:
[training]
aws_access_key_id = *****57
aws_secret_access_key = ***1b
[trainer]
aws_access_key_id = *****57
aws_secret_access_key = ***1b
config
command:
nano ~/.aws/config
renders:
[profile training]
region = us-east-1
output = json
[profile trainer]
region = us-east-1
output = json
You've configured the profile with the name trainer. You didn't create a default profile, you created a named profile. You're getting the current error because the CLI tool is looking for a default profile, and you don't have one configured.
In order to use the trainer profile you either have to add --profile trainer to every aws command you run in the command line, or you need to set the AWS_PROFILE environment variable inside your command line environment:
export AWS_PROFILE=trainer
It looks like you also tagged this with nodejs, so I recommend going the environment variable route, which will also work with the nodeJS AWS SDK.
hi i am using celery to schedule the tasks and run to read the data from the s3 files , but i am getting the error like Unable to locate credentials , but the same code is working fine with the local environement , when we deployed the code in the production(EC2) i am getting above error , and without using the celery i am able to connect the s3 and able to read the files.
the packages i have used
boto3 1.13.13
botocore 1.16.13
celery 4.4.4
both credentials are placed in same location like ~/.aws
can any one help
in the dev environment we have multiple users like our code is running on user1 and i have give user name in configured celery.conf files of supervisor is user2 so that's why it is not not finding
I'm new on cloud computing and I'm trying to use SSH to control my VM instance but when I use command (with debug)
gcloud compute ssh my-instance-name --verbosity=debug
it's show error
DEBUG: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code
[255]. Traceback (most recent call last): File
"/google/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line
983, in Execute
resources = calliope_command.Run(cli=self, args=args) File "/google/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py",
line 784, in Run
resources = command_instance.Run(args) File "/google/google-cloud-sdk/lib/surface/compute/ssh.py", line 262, in
Run
return_code = cmd.Run(ssh_helper.env, force_connect=True) File "/google/google-cloud-sdk/lib/googlecloudsdk/command_lib/util/ssh/ssh.py",
line 1256, in Run
raise CommandError(args[0], return_code=status) CommandError: [/usr/bin/ssh] exited with return code [255]. ERROR:
(gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
I try to solve the problem in this link but it's not work
https://groups.google.com/forum/#!topic/gce-discussion/O-c10TM4ZLM
SSH error code 255 is a general error returned by GCP. You can try one of the following options.
1. Wait a few minutes and try again. It is possible that:
The instance has not finished starting up.
Metadata for SSH keys has not finished being propagated to the project or instance.
The Guest Environment has not yet read the SSH keys metadata.
2. Verify that SSH access to the instance is not blocked by a firewall.
gcloud compute firewall-rules list | grep "tcp:22"
If necessary, create a firewall rule to allow TCP 22 for a given VPC network, subnet, or instance tag.
gcloud compute firewall-rules create ssh-allow-incoming --priority=0 --allow=tcp:22 --network=[VPC-Network]
3. Make sure that the root volume is not out of disk space. Messages like the following will be visible in the console log when it is out of disk space:
...No space left on device...
...google-accounts: ERROR Exception calling the response handler.
[Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp',
'/usr/tmp', '/']...
4. Make sure that the instance has not run out of memory
5. Verify that temporary SSH Keys metadata is set for either the project or instance.
Finally you could follow any of their supported or third-party methods
Assuming you have the correct IAM permissions, it is much easier and preferred by GCP to use OSlogin to ssh into an instance, rather than manage ssh keys
in cloud shell, enter this
gcloud compute --project PROJECTID project-info add-metadata --metadata enable-oslogin=TRUE
This enables OSLogin on all instances in a project, instead of using ssh keys gcp will check your IAM permissions and authenticate based on those.
If you are not project owner, make sure you have the compute.osloginviewer or admin permissions in Cloud IAM
Once enables, try SSHing into the instance again using the command you posted.
This is not a concrete answer but I think at first you should set your project by :
gcloud config set project PROJECT_ID
Then
gcloud compute ssh my-instance-name --verbosity=debug
This link would be useful:
https://cloud.google.com/sdk/gcloud/reference/compute/ssh
I was in the need to move files with a aws-lambda from a SFTP server to my AWS account,
then I've found this article:
https://aws.amazon.com/blogs/compute/scheduling-ssh-jobs-using-aws-lambda/
Talking about paramiko as a SSHclient candidate to move files over ssh.
Then I've written this calss wrapper in python to be used from my serverless handler file:
import paramiko
import sys
class FTPClient(object):
def __init__(self, hostname, username, password):
"""
creates ftp connection
Args:
hostname (string): endpoint of the ftp server
username (string): username for logging in on the ftp server
password (string): password for logging in on the ftp server
"""
try:
self._host = hostname
self._port = 22
#lets you save results of the download into a log file.
#paramiko.util.log_to_file("path/to/log/file.txt")
self._sftpTransport = paramiko.Transport((self._host, self._port))
self._sftpTransport.connect(username=username, password=password)
self._sftp = paramiko.SFTPClient.from_transport(self._sftpTransport)
except:
print ("Unexpected error" , sys.exc_info())
raise
def get(self, sftpPath):
"""
creates ftp connection
Args:
sftpPath = "path/to/file/on/sftp/to/be/downloaded"
"""
localPath="/tmp/temp-download.txt"
self._sftp.get(sftpPath, localPath)
self._sftp.close()
tmpfile = open(localPath, 'r')
return tmpfile.read()
def close(self):
self._sftpTransport.close()
On my local machine it works as expected (test.py):
import ftp_client
sftp = ftp_client.FTPClient(
"host",
"myuser",
"password")
file = sftp.get('/testFile.txt')
print(file)
But when I deploy it with serverless and run the handler.py function (same as the test.py above) I get back the error:
Unable to import module 'handler': No module named 'paramiko'
Looks like the deploy is unable to import paramiko (by the article above it seems like it should be available for lambda python 3 on AWS) isn't it?
If not what's the best practice for this case? Should I include the library into my local project and package/deploy it to aws?
A comprehensive guide tutorial exists at :
https://serverless.com/blog/serverless-python-packaging/
Using the serverless-python-requirements package
as serverless node plugin.
Creating a virtual env and Docker Deamon will be required to packup your serverless project before deploying on AWS lambda
In the case you use
custom:
pythonRequirements:
zip: true
in your serverless.yml, you have to use this code snippet at the start of your handler
try:
import unzip_requirements
except ImportError:
pass
all details possible to find in Serverless Python Requirements documentation
You have to create a virtualenv, install your dependencies and then zip all files under sites-packages/
sudo pip install virtualenv
virtualenv -p python3 myvirtualenv
source myvirtualenv/bin/activate
pip install paramiko
cp handler.py myvirtualenv/lib/python
zip -r myvirtualenv/lib/python3.6/site-packages/ -O package.zip
then upload package.zip to lambda
You have to provide all dependencies that are not installed in AWS' Python runtime.
Take a look at Step 7 in the tutorial. Looks like he is adding the dependencies from the virtual environment to the zip file. So I'd assume your ZIP file to contain the following:
your worker_function.py on top level
a folder paramico with the files installed in virtual env
Please let me know if this helps.
I tried various blogs and guides like:
web scraping with lambda
AWS Layers for Pandas
spending hours of trying out things. Facing SIZE issues like that or being unable to import modules etc.
.. and I nearly reached the end (that is to invoke LOCALLY my handler function), but then my function even though it was fully deployed correctly and even invoked LOCALLY with no problems, then it was impossible to invoke it on AWS.
The most comprehensive and best by far guide or example that is ACTUALLY working is the above mentioned by #koalaok ! Thanks buddy!
actual link
Version of Apache Spark: spark-1.2.1-bin-hadoop2.4
Platform: Ubuntu
I have been using the spark-1.2.1-bin-hadoop2.4/ec2/spark-ec2 script to create temporary clusters on ec2 for testing. All was working well.
Then I started to get the following error when trying to launch the cluster:
[Errno 185090050] _ssl.c:344: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib
I have traced this back to the following line in the spark_ec2.py script:
conn = ec2.connect_to_region(opts.region)
Thus, the first time the script interacts with ec2, it is throwing this error. Spark is using the Python boto library (included with the Spark download) to make this call.
I assume the error I am getting is because of a bad cacert.pem file somewhere.
My question: which cacert.pem file gets used when I try to invoke the spark-ec2 script, and why is it not working?
I also had this error with spark-1.2.0-bin-hadoop2.4
SOLVED: the embedded boto library that comes with Spark found a ~/.boto config file I had for another non-Spark project (actually it was for the Google Cloud Services...GCS installed it, I had forgotten about it). That was screwing everything up.
As soon as I deleted the ~/.boto config file GCS installed, everything started working again for Spark!