I would like to store many PRISM rasters in hdf5 format using h5py in python but I'm having difficulty figuring out how to get the coordinate reference systems stored so that a GIS software like ARCGIS or QGIS or other python modules (RasterIO) can read the file and know where it exists in space.
Basically I'm trying to follow the structure of how MODIS data is stored which is HDF4 but in hdf5 and with h5py.
precip_path = os.path.join(wrk_dir,"prism_data","Yearly_PRISM_PRCP_Clipped_1961_2021","*" )
# Get list of precip rasters
precip_list = sorted(glob(precip_path))
# Open on raster and get all needed information
raster_ds = rxr.open_rasterio(precip_list[0]).squeeze()
ds_size = raster_ds.shape
ds_name = os.path.basename(precip_list[0])[:-4]
x_dims = raster_ds.coords['x'].values
y_dims = raster_ds.coords['y'].values
# # Get georeference information
crs = raster_ds.rio.crs
crs
# # # Create Groups
hf.close()
hf = h5py.File("precip_hdf.h5",'w')
grp = hf.create_group("PRISM_PRCP")
# Create datasets that contain the x and y coordinates of the prism dataset
x_coords = grp.create_dataset("x_coords",data=x_dims)
y_coords = grp.create_dataset("y_coords", data=y_dims)
dset = grp.create_dataset(ds_name, data=raster_ds)
grp[ds_name].dims[0].label = 'x'
grp[ds_name].dims[1].label = 'y'
x_coords.make_scale('x')
y_coords.make_scale('y')
# Attach a scale to the dimensions of the prism dataset
grp[ds_name].dims[0].attach_scale(grp['x_coords'])
grp[ds_name].dims[1].attach_scale(grp['y_coords'])
grp.values
hf.close()
When I ran the above code all the information is stored and I can view the raster in HDFview and see all the groups with spatial information but when I try to open this file in GIS software it doesn't recognize the spatial information. I'm guessing there is a bit of code that does this but I can't figure out what it is.
Related
So I have been working on an object detection problem and I loaded my databunch object from the image list in fastai but I am not able to fully understand what kind of patches or images are genuinely present in my input data so I need a visualization of the data like how in what manner is it present how many positive classes, how much the data have hard negatives and their distributions.
This is how I created my data bunch in fastai:
#here train_images,valid_images is a list of <object_detection_fastai.helper.wsi_loader.SlideContainer object>
batch_size = 64
do_flip = True
flip_vert = True
max_rotate = 90
max_zoom = 1.1
max_lighting = 0.2
max_warp = 0.2
p_affine = 0.75
p_lighting = 0.75
tfms = get_transforms(do_flip=do_flip,
flip_vert=flip_vert,
max_rotate=max_rotate,
max_zoom=max_zoom,
max_lighting=max_lighting,
max_warp=max_warp,
p_affine=p_affine,
p_lighting=p_lighting)
train, valid ,test = ObjectItemListSlide(train_images), ObjectItemListSlide(valid_images), ObjectItemListSlide(test_images)
#print("type",(test_set[12]))
item_list = ItemLists(".", train, test)
lls = item_list.label_from_func(lambda x: x.y, label_cls=SlideObjectCategoryList)
lls = lls.transform(tfms, tfm_y=True, size=patch_size)
data = lls.databunch(bs=batch_size, collate_fn=bb_pad_collate,num_workers=0).normalize()
Now I need to find out what patches or images are present. They are of which distribution like how many positives and how many negatives and background class, as after the transform function they are changed and randomized into the data bunch. Hence, the data bunch becomes a black box of my input data, I need to have insights on what is the actual distribution of data inside databunch.
I am following the Rasterio documentation to access the geospatial raster data downloaded from here -- a large tiff image. Unfortunately, I do not have enough memory so numpy throws an ArrayMemoryError.
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 77.8 GiB for an array with shape (1, 226112, 369478) and data type uint8
My code is as follow:
import rasterio
import rasterio.features
import rasterio.warp
file_path = r'Path\to\file\ESACCI-LC-L4-LC10-Map-10m-MEX.tif'
with rasterio.open(file_path) as dataset:
# Read the dataset's valid data mask as a ndarray.
mask = dataset.dataset_mask()
# Extract feature shapes and values from the array.
for geom, val in rasterio.features.shapes(
mask, transform=dataset.transform):
# Transform shapes from the dataset's own coordinate
# reference system to CRS84 (EPSG:4326).
geom = rasterio.warp.transform_geom(
dataset.crs, 'EPSG:4326', geom, precision=6)
# Print GeoJSON shapes to stdout.
print(geom)
I need a way to store the numpy array to disk, so I tried looking into numpy nemmap, but I do not understand how to implement it for this. Additionally, I do not need to the full geospacial data, I am only interested in the lat, long, and the type of land cover as I planed to merge this with another dataset.
Using python 3.9.
Edit:
I updated my code to try using a window.
with rasterio.open(file_path) as dataset:
mask = dataset.read(1, window=Window(0, 0, 226112, 369478))
...
I can obviously adjust the window and upload the file in sections now. However, I do not understand how this has almost halved the memory required from 77.8 to 47.6.
I have a bunch of medical images in dicom that I want to correct for bias field inhomogeneity using SimpleITK in Python. The workflow is straightforward: I want to (1) open the dicom image, (2) create a binary mask of the object in the image, (3) apply N4 bias field correction to the masked image, (4) write back the corrected image in dicom format. Note that no spatial transformation is applied to the image, but only intensity transformation, so that I could copy all spatial information and all meta data (except for date/hour of creation and instance number) from the original to the corrected image.
I have written this function to achieve my goal:
def n4_dcm_correction(dcm_in_file):
metadata_to_set = ["0008|0012", "0008|0013", "0020|0013"]
filepath = PurePath(dcm_in_file)
root_dir = str(filepath.parent)
file_name = filepath.stem
dcm_reader = sitk.ImageFileReader()
dcm_reader.SetFileName(dcm_in_file)
dcm_reader.LoadPrivateTagsOn()
inputImage = dcm_reader.Execute()
metadata_to_copy = [k for k in inputImage.GetMetaDataKeys() if k not in metadata_to_set]
maskImage = sitk.OtsuThreshold(inputImage,0,1,200)
filledImage = sitk.BinaryFillhole(maskImage)
floatImage = sitk.Cast(inputImage,sitk.sitkFloat32)
corrector = sitk.N4BiasFieldCorrectionImageFilter();
output = corrector.Execute(floatImage, filledImage)
output.CopyInformation(inputImage)
for k in metadata_to_copy:
print("key is: {}; value is {}".format(k, inputImage.GetMetaData(k)))
output.SetMetaData(k, inputImage.GetMetaData(k))
output.SetMetaData("0008|0012", time.strftime("%Y%m%d"))
output.SetMetaData("0008|0013", time.strftime("%H%M%S"))
output.SetMetaData("0008|0013", str(float(inputImage.GetMetaData("0008|0013")) + randint(1, 999)))
out_file = "{}/{}_biascorrected.dcm".format(root_dir, file_name)
writer = sitk.ImageFileWriter()
writer.KeepOriginalImageUIDOn()
writer.SetFileName(out_file)
writer.Execute(sitk.Cast(output, sitk.sitkUInt16))
return
n4_dcm_correction("/path/to/my/dcm/image.dcm")
As much as the bias correction part works (the bias is removed), the writing part is a mess. I would expect my output dicom to have the exact same metadata of the original one, however they are all missing, notably the patient name, the protocol name and the manufacturer. Similalry, something is very wrong with the spatial information, since if I try to convert the dicom to the nifti format with dcm2niix, the directions are reversed: superior is down and inferior is up, forward is back and backward is front. What step am I missing ?
I suspect you are working with a MRI series, not a single file. Likely this example does what you want, read-modify-write a volume stored in a set of files.
If the example did not resolve your issue, please post to the ITK discourse which is the primary location for ITK/SimpleITK related discussions.
I am sorry if the title is unclear, I am new to python and my vocabulary is limited.
What I am trying to do is apply a standard deviation stretch to each band in a .tif raster and then create a new raster (.tif) by stacking those bands using GDAL (Python).
I able to create new false color rasters with differing band combinations and save them, and I am able to create my desired IMAGE in python using dstack (first block of code), but I am unable to save that image as a georectified .tif file.
So to create the stretched image using dstack my code looks like:
import os
import numpy as np
import matplotlib.pyplot as plt
import math
from osgeo import gdal
# code from my prof
def std_stretch_data(data, n=2):
"""Applies an n-standard deviation stretch to data."""
# Get the mean and n standard deviations.
mean, d = data.mean(), data.std() * n
# Calculate new min and max as integers. Make sure the min isn't
# smaller than the real min value, and the max isn't larger than
# the real max value.
new_min = math.floor(max(mean - d, data.min()))
new_max = math.ceil(min(mean + d, data.max()))
# Convert any values smaller than new_min to new_min, and any
# values larger than new_max to new_max.
data = np.clip(data, new_min, new_max)
# Scale the data.
data = (data - data.min()) / (new_max - new_min)
return data
# open the raster
img = gdal.Open(r'/Users/Rebekah/ThesisData/TestImages/OG/OG_1234.tif')
#open the bands
red = img.GetRasterBand(1).ReadAsArray()
green = img.GetRasterBand(2).ReadAsArray()
blue = img.GetRasterBand(3).ReadAsArray()
# create alpha band where a 0 indicates a transparent pixel and 1 is a opaque pixel
# (this is from class and i dont FULLY understand it)
alpha = np.where(red + green + blue ==0, 0, 1).astype(np.byte)
red_stretched = std_stretch_data(red, 1)
green_stretched = std_stretch_data(green, 1)
blue_stretched = std_stretch_data(blue, 1)
data_stretched = np.dstack((red_stretched, green_stretched, blue_stretched, alpha))
plt.imshow(data_stretched)
plt.show()
And that gives me a beautiful image of exactly what I want in a separate window. But no where in that code is an option to assign projections, or save it as a multiband tif.
So I took that and applied it the best I could to the code I use to create false color images and it fails (code below). If I create a 4 band tif with the alpha band the output is an empty tif, and if I create a 3 band tif and omit the alpha band the output is an entirely black tif.
import os
import numpy as np
import matplotlib.pyplot as plt
import math
from osgeo import gdal
#code from my professor
def std_stretch_data(data, n=2):
"""Applies an n-standard deviation stretch to data."""
# Get the mean and n standard deviations.
mean, d = data.mean(), data.std() * n
# Calculate new min and max as integers. Make sure the min isn't
# smaller than the real min value, and the max isn't larger than
# the real max value.
new_min = math.floor(max(mean - d, data.min()))
new_max = math.ceil(min(mean + d, data.max()))
# Convert any values smaller than new_min to new_min, and any
# values larger than new_max to new_max.
data = np.clip(data, new_min, new_max)
# Scale the data.
data = (data - data.min()) / (new_max - new_min)
return data
#open image
img = gdal.Open(r'/Users/Rebekah/ThesisData/TestImages/OG/OG_1234.tif')
# get geotill driver
gtiff_driver = gdal.GetDriverByName('GTiff')
# read in bands
red = img.GetRasterBand(1).ReadAsArray()
green = img.GetRasterBand(2).ReadAsArray()
blue = img.GetRasterBand(3).ReadAsArray()
# create alpha band where a 0 indicates a transparent pixel and 1 is a opaque pixel
# (this is from class and i dont FULLY understand it)
alpha = np.where(red + green + blue ==0, 0, 1).astype(np.byte)
# apply the 1 standard deviation stretch
red_stretched = std_stretch_data(red, 1)
green_stretched = std_stretch_data(green, 1)
blue_stretched = std_stretch_data(blue, 1)
# create empty tif file
NewImg = gtiff_driver.Create('/Users/riemann/ThesisData/TestImages/FCI_tests/1234_devst1.tif', img.RasterXSize, img.RasterYSize, 4, gdal.GDT_Byte)
if NewImg is None:
raise IOerror('could not create new raster')
# set the projection and geo transform of the new raster to be the same as the original
NewImg.SetProjection(img.GetProjection())
NewImg.SetGeoTransform(img.GetGeoTransform())
# write new bands to the new raster
band1 = NewImg.GetRasterBand(1)
band1.WriteArray(red_stretched)
band2 = NewImg.GetRasterBand(2)
band2.WriteArray(green_stretched)
band3= NewImg.GetRasterBand(3)
band3.WriteArray(blue_stretched)
alpha_band = NewImg.GetRasterBand(4)
alpha_band.WriteArray(alpha)
del band1, band2, band3, img, alpha_band
I am not entirely sure how to go from here and create a new file displaying the stretch on the different bands.
The image is just a 4 band raster (NAIP) downloaded from earthexplorer, I can upload the specific image I am using for my test if needed but there is nothing inherently special about this file compared to other NAIP images.
You should close the new Dataset (NewImg) as well by either adding it to the del list you already have, or setting it to None.
That properly closes the file and makes sure all data is written to disk.
There is however another issue, you are scaling your data between 0 and 1, but storing it as a Byte. So either change the output datatype from gdal.GDT_Byte to something like gdal.GDT_Float32. Or multiply your scaled data to fit the output datatype, in the case of Byte multiple with 255 (don't forget the alpha), you should properly round it for accuracy, GDAL will otherwise truncate to the nearest integer.
You can use np.iinfo() to check what the range of a datatype is, in case you are unsure what multiplication to use for other datatypes.
Depending on your use case, it might be easiest to use gdal.Translate for the scaling. If you would modify your scaling function a little to return the scaling parameteters instead of the data, you could use something like:
ds = gdal.Translate(output_file, input_file, outputType=gdal.GDT_Byte, scaleParams=[
[old_min_r, old_max_r, new_min_r, new_max_r], # red
[old_min_g, old_max_g, new_min_g, new_max_g], # green
[old_min_b, old_max_b, new_min_b, new_max_b], # blue
[old_min_a, old_max_a, new_min_a, new_max_a], # alpha
])
ds = None
You could also add the exponents keyword for non-linear stretching.
Using gdal.Translate would save you from all the standard file creation boilerplate, you still would need to think about the datatype, since that might change compared to the input file.
I went through Pyimagesearch face Recognition tutorial,
but my application need to compare two faces only,
I have embedding of two faces, how to compare them using opencv ?
about the trained model which is use to extract embedding from face is mentioned in link,
I want to know that what methods I should try to compare two face embedding.
(Note: I am new to this field)
First of all your case is similar to given tutorial, instead of multiple images you have single image that you need to compare with test image,
So you don't really need training step here.
You can do
# read 1st image and store encodings
image = cv2.imread(args["image"])
rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
boxes = face_recognition.face_locations(rgb, model=args["detection_method"])
encodings1 = face_recognition.face_encodings(rgb, boxes)
# read 2nd image and store encodings
image = cv2.imread(args["image"])
rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
boxes = face_recognition.face_locations(rgb, model=args["detection_method"])
encodings2 = face_recognition.face_encodings(rgb, boxes)
# now you can compare two encodings
# optionally you can pass threshold, by default it is 0.6
matches = face_recognition.compare_faces(encoding1, encoding2)
matches will give you True or False based on your images
Based on the article you mentioned, you can actually compare if two faces are the same using only the face_recognition library.
You can use the compare faces to determine if two pictures have the same face
import face_recognition
known_image = face_recognition.load_image_file("biden.jpg")
unknown_image = face_recognition.load_image_file("unknown.jpg")
biden_encoding = face_recognition.face_encodings(known_image)[0]
unknown_encoding = face_recognition.face_encodings(unknown_image)[0]
results = face_recognition.compare_faces([biden_encoding], unknown_encoding)