Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
I am trying to add system time as timestamp to my s3 bucket folder names so that every time i run the code, it would create a separate folder with a different time stamp on s3.
How do i achieve this ?
import json
import boto3
s3 = boto3.resource('s3')
s3object = s3.Object('your-bucket-name', 'your_file.json')
s3object.put(
Body=(bytes(json.dumps(json_data).encode('UTF-8')))
)
You would use standard Python date functions to construct the folder name you want, then set that string as part of the S3 object's key. Something like this:
import json
import boto3
from datetime import datetime
s3 = boto3.resource('s3')
prefix = 'folder_' + datetime.now().strftime("%I%p") + "/"
s3object = s3.Object('your-bucket-name', prefix + 'your_file.json')
s3object.put(
Body=(bytes(json.dumps(json_data).encode('UTF-8')))
)
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
How can I get all the Request URL in chrome network
Right click -> Copy -> Copy all as HAR
Then you can import it like this:
import json
obj = json.loads(
'''
<paste here>
'''
)
You can then get urls = [ entry['request']['url'] for entry in obj['log']['entries'] ]
You may need to replace \" with \\" in your text editor for it to compile.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I need to make a command line application which has two parameters:
the location of the input file, and
the location of the output file.
The input file is a GTFS (.txt) file.
The output file is a .shp file.
How should I do this?
To get command line parameters:
% python3 your_script.py parameter1 parameter2
Where parameter1 and parameter2 are you input file and output file names:
import sys
parameters = sys.argv
#parameters now contains 3 strings:
#"your_script.py" in parameters[0]
#"parameter1" in parameters [1]
#"parameter2" in parameters [2]
So you can use the command line arguments as variables. To open the files:
in_file = open(parameters[1] + ".txt") #from your problem statement, it sounds
#like your filenames don't include extensions. If they do, remove
#the (+ ".txt") part
out_file = open(parameters[2] + ".shp", 'w')
For more information about I/O operations in Python, see this link
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 6 years ago.
Improve this question
I'm using python and am trying to create a new csv file using the csv module (i.e. one that doesn't currently exist). Does anyone know how to do this?
Thanks in advance,
Max
if you want simply create a csv file you can use built in method open, for more about open check this
with open("filename.csv","a+") as f:
f.write(...)
or if you want to read an exist csv file you can use this
import csv
with open('filename.csv', 'rb') as csvfile:
spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')
for row in
print ', '.join(row)
#if you want to save the file into given path
import os
os.rename("filename.csv","path/to/new/desination/for/filename.csv")
for more check docs
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
The download link I want to manipulate is below:
http://hfrnet.ucsd.edu/thredds/ncss/grid/HFR/USWC/6km/hourly/RTV/HFRADAR,_US_West_Coast,_6km_Resolution,_Hourly_RTV_best.ncd?var=u&var=v&north=47.20&west=-126.3600&east=-123.8055&south=37.2500&horizStride=1&time_start=2015-11-01T00%3A00%3A00Z&time_end=2015-11-03T14%3A00%3A00Z&timeStride=1&addLatLon=true&accept=netcdf
I want to make anything that's in bold a variable, so I can ask the user what coordinates and data set they want. This way I can download different data sets by using this script. I would also like to use the same variables to name the new file that was downloaded ex:USWC6km20151101-20151103.
I did some research and learned that I can use the urllib.parse and urllib2, but when I try experimenting with them, it says "no module named urllib.parse."
I can use the webbrowser.open() to download the file, but manipulating the url is giving me problems
THANK YOU!!
Instead of urllib you can use requests module that makes downloading content much easier. The part that makes actual work is just 4 lines long.
# first install this module
import requests
# parameters to change
location = {
'part': 'USWC',
'part2': '_US_West_Coast',
'km': '6km',
'north': '45.0000',
'west': '-120.0000',
'east': '-119.5000',
'south': '44.5000',
'start': '2016-10-01',
'end': '2016-10-02'
}
# this is template for .format() method to generate links (very naive method)
link_template = "http://hfrnet.ucsd.edu/thredds/ncss/grid/HFR/{part}/{km}/hourly/RTV/\
HFRADAR,{part2},_{km}_Resolution,_Hourly_RTV_best.ncd?var=u&var=v&\
north={north}&west={west}&east={east}&south={south}&horizStride=1&\
time_start={start}T00:00:00Z&time_end={end}T16:00:00Z&timeStride=1&addLatLon=true&accept=netcdf"
# some debug info
link = link_template.format(**location)
file_name = location['part'] + location['km'] + location['start'].replace('-', '') + '-' + location['end'].replace('-', '')
print("Link: ", link)
print("Filename: ", file_name)
# try to open webpage
response = requests.get(link)
if response.ok:
# open file for writing in binary mode
with open(file_name, mode='wb') as file_out:
# write response to file
file_out.write(response.content)
Probably the next step would be running this in loop on list that contains location dicts. Or maybe reading locations from csv file.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 11 years ago.
Improve this question
I have written a script which takes mysql dumps and uploads it to s3 and I have added the script to the cronjob and script runs at 2 o clock in the mid night and uploads the mysql dump to S3. I am using the date and time stamp as the file name before uploading it to S3.
My problem is I need to manage back ups of 7 days on S3 and automatically I have to delete the 8th day backup file from S3 since I am using the date and Time stamp as file name to make each file unique, I am not able to figure out how to do it.
And also I have to restore the latest backup in another EC2 instance.
You can grab the XML response from your S3 bucket host, such as
http://YOUR_BUCKET.s3.amazonaws.com/
It should return a XML like :-
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>...</Name>
<Prefix/>
<Marker/>
<MaxKeys>1000</MaxKeys>
<IsTruncated>true</IsTruncated>
<Contents>
<Key>xxxx.gz (if you gzip the dump)</Key>
<LastModified>2011-11-10T02:38:49.000Z</LastModified>
<ETag>"xxxxx"</ETag>
<Size>xxx</Size>
<StorageClass>STANDARD</StorageClass>
</Contents>
And with the value from LastModified node, you can determine when the file is created.
S3 has SDK api available is different languages,
you can download a copy then programmatically do the purging
As for replicating, with the SDK api, you can grab the content from original S3 bucket then post it to another S3 bucket.
SDK Api:-
http://aws.amazon.com/sdkforphp/ (PHP)
http://aws.amazon.com/sdkfornet/ (.Net)
http://aws.amazon.com/sdkforjava/ (Java)
You can create backup file use date string as filename , and use date -7 days to figure out
the files need delete.