I am trying to use EECodes in python to get various weather information, such as temperature, humidity, etc out of grib2 files. I am using the GFS files. I would like to be able to extract the data as (lat,lon,alt,$data_point), and as a 2d array for each altitude.
I have tried the example programs located here: https://confluence.ecmwf.int/display/ECC/grib_iterator_bitmap
I can't figure out what I am looking in the output of that program. When I load the messages using their keys, it is not obvious how to make a grid. When I load the grid, the data doesn't have labels I understand.
#craeft have a look to https://github.com/ecmwf/cfgrib. cfgrib is the new standard for python and grib file handling. It is easy to install and easy to access files. Please install the latest version because it supports GFS files.
Related
Is there any existing python package that can be used to handle METEOR RADAR data sets which are in .hwd file format?
I want to work on atmoshpereic science project on tide analysis in the MLT region using python.So, the source of the data is METEOR RADAR which stores data in .hwd file format(height width depth).
I tried searching the internet for specific packages that could help me file handle .hwd files but ended up finding no packages or libraries that are currently active.
Could you please help me?
Thank you.
I figured this out!
There is no need for external packages to work on hwd files in python.
hwd files stand for Horizontal Wind Data files. So, METEOR radar stores data in hwd file format, which can be treated as a normal text(.txt) file for file handling in python.
I'm having trouble trying to plot METAR data from the THREDDS data server, which comes in .nc format, onto a map. I'm using Siphon to grab the data and Xarray to open the dataset with remote_access, but the problem comes with attempting to convert the data to .csv or .txt. When using NetCDF Dataset and attempting to read the NetCDF4 file, I get a file does not exist error. I'm not really sure where to start as there doesn't seem to be much on the internet about this. Any help would be appreciated as I'm also really new to this.
I am trying to create (what I thought was) a simple image classification pipeline between s3 and SageMaker.
Images are stored in an s3 bucket with their class labels in their file names currently, e.g.
My-s3-bucket-dir
cat-1.jpg
dog-1.jpg
cat-2.jpg
..
I've been trying to leverage several related example .py scripts, but most seem to be download data sets already in .rec format or containing special manifest or annotation files I don't have.
All I want is to pass the images from s3 to the SageMaker image classification algorithm that's located in the same region, IAM account, etc. I suppose this means I need a .lst file
When I try to manually create the .lst it doesn't seem to like it and it also takes too long doing manual work to be a good practice.
How can I automatically generate the .lst file (or otherwise send the images/classes for training)?
Things I read made it sound like im2rec.py was a solution, but I don't see how. The example I'm working with now is
Image-classification-fulltraining-highlevel.ipynb
but it seems to download the data as .rec,
download('http://data.mxnet.io/data/caltech-256/caltech-256-60-train.rec')
download('http://data.mxnet.io/data/caltech-256/caltech-256-60-val.rec')
which just skips working with the .jpeg files. I found another that converts them to .rec but again it has essentially the .lst already as .json and just converts it.
I have mostly been working in a Python Jupyter notebook within the AWS console (in my browser) but I have also tried using their GUI.
How can I simply and automatically generate the .lst or otherwise get the data/class info into SageMaker without manually creating a .lst file?
Update
It looks like im2py can't be run against s3. You'd have to completely download everything from all s3 buckets into the notebook's storage...
Please note that [...] im2rec.py is running locally,
therefore cannot take input from the S3 bucket. To generate the list
file, you need to download the data and then use the im2rec tool. - AWS SageMaker Team
There are 3 options to provide annotated data to the Image Classification algo: (1) packing labels in recordIO files, (2) storing labels in a JSON manifest file ("augmented manifest" option), (3) storing labels in a list file. All options are documented here: https://docs.aws.amazon.com/sagemaker/latest/dg/image-classification.html.
Augmented Manifest and .lst files option are quick to do since they just require you to create an annotation file with a usually quick for loop for example. RecordIO requires you to use im2rec.py tool, which is a little more work.
Using .lst files is another option that is reasonably easy: you just need to create annotation them with a quick for loop, like this:
# assuming train_index, train_class, train_pics store the pic index, class and path
with open('train.lst', 'a') as file:
for index, cl, pic in zip(train_index, train_class, train_pics):
file.write(str(index) + '\t' + str(cl) + '\t' + pic + '\n')
I'm trying to make simple Python 3 program to read weather information from XML web source, convert it into Python-readable object (maybe dictionary) and process it (for example visualize multiple observations into graph).
Source of data is national weather service's (direct translation) xml file at link provided in code.
What's different from typical XML parsing related question in Stack Overflow is that there are repetitive tags without in-tag identificator (<station> tags in my example) and some with (1st line, <observations timestamp="14568.....">). Also I would like to try parse it straight from website, not local file. Of course, I could create local temporary file too.
What I have so far, is simply loading script, that gives string containing xml code for both forecast and latest weather observations.
from urllib.request import urlopen
#Read 4-day forecast
forecast= urlopen("http://www.ilmateenistus.ee/ilma_andmed/xml/forecast.php").read().decode("iso-8859-1")
#Get current weather
observ=urlopen("http://www.ilmateenistus.ee/ilma_andmed/xml/observations.php").read().decode("iso-8859-1")
Shortly, I'm looking for as universal as possible way to parse XML to Python-readable object (such as dictionary/JSON or list) while preserving all of the information in XML-file.
P.S I prefer standard Python 3 module such as xml, which I didn't understand.
Try xmltodict package for simple conversion of XML structure to Python dict: https://github.com/martinblech/xmltodict
I've managed to compile GDCM with VTK and I have a particular application I would like to use, which is the "gdcm2vtk.exe".
Now, how's the syntax for converting a stack of imags into a ".vti" file? so far I have this:
gdcm2vtk Input_Directory file.vti
Now, when I run somthing like this:
gdcm2vtk "C:/dicom/dicom directory" output.vti I get an error:
could not find no reader to handle file: "C:/dicom/dicom directory"
Is there anything I'm missing there?
gdcm2vtk does not handle a directory as input as specified in the documentation.
You may want to convert your DICOM Series into a single DICOM Instance using gdcmimg
As of GDCM 2.6 gdcm2vtk is now able to take as input a directory. Pay attention to sort the file according to the well known Image Orientation (Patient) & Image Position (Patient) strategy instead of relying on the filenames ordering to reconstruct your VTK (*.vti) file:
$ gdcm2vtk --ipp-sort input_dir output.vti