I have trained the following Sagemaker model: https://github.com/awslabs/amazon-sagemaker-examples/tree/master/introduction_to_amazon_algorithms/object_detection_pascalvoc_coco
I've tried both the JSON and RecordIO version. In both, the algorithm is tested on ONE sample image. However, I have a dataset of 2000 pictures, which I would like to test. I have saved the 2000 jpg pictures in a folder within an S3 bucket and I also have two .mat files (pics + ground truth). How can I apply this model to all 2000 pictures at once and then save the results, rather than doing it one picture at a time?
I am using the code below to load a single picture from my S3 bucket:
object = bucket.Object('pictures/pic1.jpg')
object.download_file('pic1.jpg')
img=mpimg.imread('pic1.jpg')
img_name = 'pic1.jpg'
imgplot = plt.imshow(img)
plt.show(imgplot)
with open(img_name, 'rb') as image:
f = image.read()
b = bytearray(f)
ne = open('n.txt','wb')
ne.write(b)
import json
object_detector.content_type = 'image/jpeg'
results = object_detector.predict(b)
detections = json.loads(results)
print (detections['prediction'])
I'm not sure if I understood your question correctly. However, if you want to feed multiple images to the model at once, you can create a multi-dimensional array of images (byte arrays) to feed the model.
The code would look something like this.
import numpy as np
...
# predict_images_list is a Python list of byte arrays
predict_images = np.stack(predict_images_list)
with graph.as_default():
# results is an list of typical results you'd get.
results = object_detector.predict(predict_images)
But, I'm not sure if it's a good idea to feed 2000 images at once. Better to batch them in 20-30 images at a time and predict.
Related
I tried to import pictures from the camera and I could get 3-dim data from the image.
img = WebcamModule.getImg(True, size=[240,120])
img = image.img_to_array(img)
then, from the code below I tried to expand one more dimention which is batch size. But it doesn't work.
How do I import batch size into the data.
There is a picture of the result below.
img = np.expand_dims(img,axis=0)
Finally, the code below for predicting doesn't work.
val = float(model.predict(img))
I usually do this:
input = input_raw[np.newaxis, ...]
prediction = model.predict(input)
pred = np.squeeze(prediction)
draw(pred)
Check the shape of the image and the image the network wants, check if the model is channel first or channel last, but this always worked for me.
Running the fairseq-preprocess script produces binary files with integer indices corresponding to token ids in a dictionary.
When I no longer have the original tokenized texts, what is the simplest way to explore the binarized dataset? The documentation does not say much about how a dataset can be loaded for debugging purposes.
I worked around this by loading the trained model and using it to decode the binarized sentences back to strings:
from fairseq.models.transformer import TransformerModel
model_dir = ???
data_dir = ???
model = TransformerModel.from_pretrained(
model_dir,
checkpoint_file='checkpoint_best.pt',
data_name_or_path=data_dir,
bpe='sentencepiece',
sentencepiece_model=model_dir + '/sentencepiece.joint.bpe.model'
)
model.task.load_dataset('train')
data_bin = model.task.datasets['train']
train_pairs = [
(model.decode(item['source']), model.decode(item['target']))
for item in data_bin
]
I am following the Rasterio documentation to access the geospatial raster data downloaded from here -- a large tiff image. Unfortunately, I do not have enough memory so numpy throws an ArrayMemoryError.
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 77.8 GiB for an array with shape (1, 226112, 369478) and data type uint8
My code is as follow:
import rasterio
import rasterio.features
import rasterio.warp
file_path = r'Path\to\file\ESACCI-LC-L4-LC10-Map-10m-MEX.tif'
with rasterio.open(file_path) as dataset:
# Read the dataset's valid data mask as a ndarray.
mask = dataset.dataset_mask()
# Extract feature shapes and values from the array.
for geom, val in rasterio.features.shapes(
mask, transform=dataset.transform):
# Transform shapes from the dataset's own coordinate
# reference system to CRS84 (EPSG:4326).
geom = rasterio.warp.transform_geom(
dataset.crs, 'EPSG:4326', geom, precision=6)
# Print GeoJSON shapes to stdout.
print(geom)
I need a way to store the numpy array to disk, so I tried looking into numpy nemmap, but I do not understand how to implement it for this. Additionally, I do not need to the full geospacial data, I am only interested in the lat, long, and the type of land cover as I planed to merge this with another dataset.
Using python 3.9.
Edit:
I updated my code to try using a window.
with rasterio.open(file_path) as dataset:
mask = dataset.read(1, window=Window(0, 0, 226112, 369478))
...
I can obviously adjust the window and upload the file in sections now. However, I do not understand how this has almost halved the memory required from 77.8 to 47.6.
I have a bunch of medical images in dicom that I want to correct for bias field inhomogeneity using SimpleITK in Python. The workflow is straightforward: I want to (1) open the dicom image, (2) create a binary mask of the object in the image, (3) apply N4 bias field correction to the masked image, (4) write back the corrected image in dicom format. Note that no spatial transformation is applied to the image, but only intensity transformation, so that I could copy all spatial information and all meta data (except for date/hour of creation and instance number) from the original to the corrected image.
I have written this function to achieve my goal:
def n4_dcm_correction(dcm_in_file):
metadata_to_set = ["0008|0012", "0008|0013", "0020|0013"]
filepath = PurePath(dcm_in_file)
root_dir = str(filepath.parent)
file_name = filepath.stem
dcm_reader = sitk.ImageFileReader()
dcm_reader.SetFileName(dcm_in_file)
dcm_reader.LoadPrivateTagsOn()
inputImage = dcm_reader.Execute()
metadata_to_copy = [k for k in inputImage.GetMetaDataKeys() if k not in metadata_to_set]
maskImage = sitk.OtsuThreshold(inputImage,0,1,200)
filledImage = sitk.BinaryFillhole(maskImage)
floatImage = sitk.Cast(inputImage,sitk.sitkFloat32)
corrector = sitk.N4BiasFieldCorrectionImageFilter();
output = corrector.Execute(floatImage, filledImage)
output.CopyInformation(inputImage)
for k in metadata_to_copy:
print("key is: {}; value is {}".format(k, inputImage.GetMetaData(k)))
output.SetMetaData(k, inputImage.GetMetaData(k))
output.SetMetaData("0008|0012", time.strftime("%Y%m%d"))
output.SetMetaData("0008|0013", time.strftime("%H%M%S"))
output.SetMetaData("0008|0013", str(float(inputImage.GetMetaData("0008|0013")) + randint(1, 999)))
out_file = "{}/{}_biascorrected.dcm".format(root_dir, file_name)
writer = sitk.ImageFileWriter()
writer.KeepOriginalImageUIDOn()
writer.SetFileName(out_file)
writer.Execute(sitk.Cast(output, sitk.sitkUInt16))
return
n4_dcm_correction("/path/to/my/dcm/image.dcm")
As much as the bias correction part works (the bias is removed), the writing part is a mess. I would expect my output dicom to have the exact same metadata of the original one, however they are all missing, notably the patient name, the protocol name and the manufacturer. Similalry, something is very wrong with the spatial information, since if I try to convert the dicom to the nifti format with dcm2niix, the directions are reversed: superior is down and inferior is up, forward is back and backward is front. What step am I missing ?
I suspect you are working with a MRI series, not a single file. Likely this example does what you want, read-modify-write a volume stored in a set of files.
If the example did not resolve your issue, please post to the ITK discourse which is the primary location for ITK/SimpleITK related discussions.
When I try to save hyperopt.trials object, which contains information about auto params tuning in neural network,
best = fmin(fn = objective,
space = space,
algo = tpe.suggest, # or rand.suggest for random params selection
max_evals = max_trials,
trials = trials) #, rstate = np.random.RandomState(50)
pickle.dump(trials, open("neuro.hyperopt", "wb"))
it gives the error:
can't pickle _thread.RLock objects
Moreover, it loads on my local drive a file of 10GB size. That is, it saves not only the trials object, but the whole model.
Would you help me to save trials object with less size (e.g. the XGBoost trials file's size is 1Mb) and avoid the error.
Thank you.
In my case it was because the models stored in the trials were not pickle-able.
I tried to save tf.keras.optimizers.Adam(learning_rate = 0.001) object.
When I added the string 'Adam' instead, the error disapeared.
Of course, it creates another problem: how to setup learining rate for the optimizor. But it seems to be easier. One way is to replace the keras object with string in the trials.trials object before saving:
for trial in trials.trials:
if 'result' in trials.keys():
trials['result'].pop('model', None) # https://stackoverflow.com/questions/15411107/delete-a-dictionary-item-if-the-key-exists
# proceed with pickling
pickle.dump(trials, open("trials.pkl","wb"))
(I took it from here)