I'm trying to download a zip file from FEMA using python3 and requests. The url works when I paste it in the browser and also when I wget. There's a redirect and I see it being successfully redirected using requests. However, the returned content has significantly fewer bytes than what I expected, like a malformed zip.
Here's the URL and snippet of code:
>>> import requests
>>> url = 'https://hazards.fema.gov/femaportal/NFHL/Download/ProductsDownLoadServlet?DFIRMID=10001C&state=DELAWARE&county=KENT COUNTY&fileName=10001C_20190130.zip'
>>> resp = requests.get(url)
>>> len(resp.content)
2583
>>> resp.headers['content-length']
'66892906'
>>> resp.url
'https://hazards.fema.gov/nfhlv2/output/County/10001C_20190130.zip'
UPDATE: I was running this in a docker container. Outside of the container, it's working as expected.
Python 3.7.3 (default, May 10 2019, 15:15:13)
[Clang 10.0.1 (clang-1001.0.46.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.__version__
'2.21.0'
>>> url = 'https://hazards.fema.gov/nfhlv2/output/County/10001C_20190130.zip'
>>> r = requests.get(url)
>>> len(r.content)
66892906
>>>
Related
I'm trying to load my tar image in podman.
Python 3.7.8 (default, Jun 29 2020, 05:46:05)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import podman
>>> c = podman.Client(uri='unix://var/run/docker.sock') #docker.sock is my mount point for podman socket
>>> c.
c.containers c.images c.pods c.system
>>> c.images.
c.images.build( c.images.get( c.images.list( c.images.search(
c.images.delete_unused( c.images.import_image( c.images.pull(
>>> c.images.
I don't see any load image method.
Is there any method like c.load_image("/tmp/se.tgz")? similar method exists for docker and easily doable.
There is an existing API that can be used like this.
c = podman.Client(uri='unix://var/run/docker.sock')
with c._client() as podman:
results = podman.LoadImage("", "/tmp/se.tgz", False, False)
If my file name is 5f0756fa-75bc-4c70-9ba8-fbd1b6a9f843_20200616T10_50_UTC.wav .. Output to be 5f0756fa-75bc-4c70-9ba8-fbd1b6a9f843.wav . How can we trim the file name using python?
There are several ways to do, for instance
import os
l=os.path.splitext("5f0756fa-75bc-4c70-9ba8-fbd1b6a9f843_20200616T10_50_UTC.wav")
l[0].split("_")[0] + l[1]
I use os.path.splitext to separate the possible extension
Execution with and without '_' and with and without extension :
pi#raspberrypi:~ $ python3
Python 3.7.3 (default, Dec 20 2019, 18:57:59)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>>
>>> def f(s):
... l = os.path.splitext(s)
... return l[0].split("_")[0] + l[1]
...
>>> f("5f0756fa-75bc-4c70-9ba8-fbd1b6a9f843_20200616T10_50_UTC.wav")
'5f0756fa-75bc-4c70-9ba8-fbd1b6a9f843.wav'
>>>
>>> f("5f0756fa-75bc-4c70-9ba8-fbd1b6a9f843.wav")
'5f0756fa-75bc-4c70-9ba8-fbd1b6a9f843.wav'
>>>
>>> f("5f0756fa-75bc-4c70-9ba8-fbd1b6a9f843_20200616T10_50_UTC")
'5f0756fa-75bc-4c70-9ba8-fbd1b6a9f843'
>>>
>>> f("5f0756fa-75bc-4c70-9ba8-fbd1b6a9f843")
'5f0756fa-75bc-4c70-9ba8-fbd1b6a9f843'
>>>
I am creating a tar archive with tarfile module in Python. All ok, but I get a gzip file named archive.gz which containa an archive named archive.gz too. How can I make the archive without repetitive name?
here the code:
import tarfile
import os
src_archive = 'C:\\source_archive'
dest_archive = 'C:\\archive.tar.gz'
def make_tarfile(output_filename, source_dir):
with tarfile.open(output_filename, "w:gz") as tar:
tar.add(source_dir, arcname=os.path.basename(source_dir))
if __name__ == "__main__":
print('creating archive')
make_tarfile(dest_archive,src_archive)
I had a similar issue, what I did was use UUID to randomly generate a string and append that to the end of the file (which requires a unique name)
Another thing you could do is use a database, and append name by index number.
Python 3.7.3 (default, Mar 27 2019, 09:23:15)
[Clang 10.0.1 (clang-1001.0.46.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import uuid
>>> src_archive = f'C:\\source_archive-{str(uuid.uuid4()).split("-")[0]}'
>>> print(src_archive)
C:\source_archive-674fce6b
I just finished installing Tensorflow 1.3 on RPi 3. When validating the installation (according to this https://www.tensorflow.org/install/install_sources) somehow a lowercase "b" shown up. See these codes:
root#raspberrypi:/home/pi# python
Python 3.4.2 (default, Oct 19 2014, 13:31:11)
[GCC 4.9.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> hello = tf.constant('Hello, TensorFlow!')
>>> sess = tf.Session()
>>> print(sess.run(hello))
b'Hello, TensorFlow!'
>>>
No, it's not a bug. Your installation is perfectly fine, this is the normal behavior.
The b before the string is due to the Tensorflow internal representation of strings.
Tensorflow represents strings as byte array, thus when you "extract" them (from the graph, thus tensorflow's internal representation, to the python enviroment) using sess.run(hello) you get a bytes type and not a str type.
You can verify this using the type function:
import tensorflow as tf
hello = tf.constant('Hello, TensorFlow!')
sess = tf.Session()
print(type(sess.run(hello)))
results in <class 'bytes'> whilst if you do:
print(type('Hello, TensorFlow!'))
results in <class 'str'>
In interpreter 'urllib.request.urlretrieve' work great like this:
Python 3.4.3+ (default, Oct 14 2015, 16:03:50)
[GCC 5.2.1 20151010] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request
>>>urllib.request.urlretrieve('https://download.owncloud.org/download/repositories/stable/Ubuntu_15.10/Release.key', 'Release.key')
('Release.key', <http.client.HTTPMessage object at 0x7fa008f76518>)
In this function will not work:
def owncloudI():
print(ok('Installing OwnCloud...\n'))
urllib.request.urlretrieve('https://download.owncloud.org/download/repositories/stable/Ubuntu_15.10/Release.key', 'Release.key') #<---- stuck here
call(['apt-key', 'add', 'Release.key'])
rep = open('/etc/apt/sources.list.d/owncloud.list', 'a')
print('deb http://download.owncloud.org/download/repositories/stable/Ubuntu_15.10/', file=rep)
rep.close()
call(['apt-get', 'update'])
call(['apt-get', 'install', 'owncloud'])
Code launch with sudo and execute in /tmp cat, therefore i think with rights all ok. Maybe someone has solved a similar problem.
Fix it this with
with urllib.request.urlopen('https://download.owncloud.org/download/repositories/stable/Ubuntu_15.10/Release.key') as response, open('resp.key', 'wb') as out_file:
data = response.read()
out_file.write(data)