Input-Sample
I am trying to pre-process my images in order to improve the ocr quality. However, I am stuck with a problem.
The Images I am dealing with contain different text orientations within the same image (2 pages, 1st is vertical, the 2nd one is horizontally oriented and they are scanned to the same image.
The text direction is automatically detected for the first part. nevertheless, the rest of the text from the other page is completely missed up.
I was thinking of creating a zonal template to detect the regions of interest but I don't know how.
Or automatically detect the border and split the image adaptively then flip the splitted part to achieve the required result.
I could set splitting based on a fixed pixel height but it is not constant as well.
from tesserocr import PyTessBaseAPI, RIL
import cv2
from PIL import Image
with PyTessBaseAPI() as api:
filePath = r'sample.jpg'
img = Image.open(filePath)
api.SetImage(img)
boxes = api.GetComponentImages(RIL.TEXTLINE, True)
print('Found {} textline image components.'.format(len(boxes)))
for i, (im, box, _, _) in enumerate(boxes):
# im is a PIL image object
# box is a dict with x, y, w and h keys
api.SetRectangle(box['x'], box['y'], box['w'], box['h'])
ocrResult = api.GetUTF8Text()
conf = api.MeanTextConf()
for box in boxes:
box = boxes[0][1]
x = box.get('x')
y = box.get('y')
h = box.get('h')
w = box.get('w')
cimg = cv2.imread(filePath)
crop_img = cimg[y:y+h, x:x+w]
cv2.imshow("cropped", crop_img)
cv2.waitKey(0)
output image
as you can see i can apply an orientation detection but I wount get any meaningful text out of such an image.
Try Tesseract API method GetComponentImages and then DetectOrientationScript on each component image.
Related
I am using pyautogui.locateOnScreen() function to locate elements in chrome and get their x,y coordinates and click them. But at some point I need to take a screenshot of a part of the screen and search for the object I want in this screenshot. Then I get coordinates of it. Is it possible to do it with pyautogui?
My example code:
coord_one = pyautogui.locateOnScreen("first_image.png",confidence=0.95)
scshoot = pyautogui.screenshot(region=coord_one)
coord_two = # search second image in scshoot and if it can be detected get coordinates of it.
If it is not possible with pyautogui, can you advice the easiest-smartest way?
Thanks in advance.
I don't believe there is a built-in direct way to do what you need but the python-opencv library does the job.
The following code sample assumes you have an screen capture you just took "capture.png" and you want to find "logo.png" in that capture, which you know is an subsection of "capture.png".
Minimal example
"""Get bounding box of cropped image from original image."""
import cv2 as cv
import numpy as np
img_rgb = cv.imread(r'res/original.png')
# the cropped image, expected to be smaller
target_img = cv.imread(r'res/crop.png')
_, w, h = target_img.shape[::-1]
res = cv.matchTemplate(img_rgb,target_img,cv.TM_CCOEFF_NORMED)
# with the method used, the date in res are top left pixel coords
min_val, max_val, min_loc, max_loc = cv.minMaxLoc(res)
top_left = max_loc
# if we add to it the width and height of the target, then we get the bbox.
bottom_right = (top_left[0] + w, top_left[1] + h)
cv.rectangle(img_rgb,top_left, bottom_right, 255, 2)
cv.imshow('', img_rgb)
MatchTemplate
From the docs, MatchTemplate "simply slides the template image over the input image (as in 2D convolution) and compares the template and patch of input image under the template image." Under the hood, this offers methods such as square difference to compare the images represented as arrays.
See more
For a more in-depth explanation, check the opencv docs as the code is entirely based off their example.
I'm making a Python file that will make a filter to have color on the Canny filter in OpenCV. I do this change from grayscale to color using the code provided below. My problem is when I apply the concatenate method (to add the color back as Canny filter is converted to grayscale), it cuts the width of the screen in 3 as I show in the 2 screenshots of before the color is added and after. The code snippet shown is only the transformation from grayscale to colored images.
What I've tried:
Tried using NumPy.tile: this wasn't the wisest attempt as it just repeated the same 1/3 of the screen twice more and didn't expand it to take up the whole screen as I had hoped.
Tried changing the image to only be from the index of 1/3 of the screen to cover the entire screen.
Tried setting the column index that is blank to equal None.
Image without the color added
Image with the color added
My code:
def convert_pixels(image, color):
rows, cols = image.shape
concat = np.zeros(image.shape)
image = np.concatenate((image, concat), axis=1)
image = np.concatenate((image, concat), axis=1)
image = image.reshape(rows, cols, 3)
index = image.nonzero()
#TODO: turn color into constantly changing color wheel or shifting colors
for i in zip(index[0], index[1], index[2]):
color.next_color()
image[i[0]][i[1]] = color.color
#TODO: fix this issue below:
#image[:, int(cols/3):cols] = None # turns right side (gliched) into None type
return image, color
In short, you're using concatenate on the wrong axis. axis=1 is the "columns" axis, so you're just putting two copies of zeros next to each other in the x direction. Since you want a three-channel image I would just initialize color_image with three channels and leave the original grayscale image alone:
def convert_pixels(image,color):
rows, cols = image.shape
color_image = np.zeros((rows,cols,3),dtype=np.uint8)
idx = image.nonzero()
for i in zip(*idx):
color_image[i] = color.color
return color_image,color
I've changed the indexing to match. I can't check this exactly since I don't know what your color object is, but I can confirm this works in terms of correctly shaping and indexing the new image.
I have about a hundred photos that aren't very sharp and I'd like to make them sharper.
So I created a script with python that already tries with one. I have tried with PIL, OpenCV and OCR readers to read texts from Images.
# External libraries used for
# Image IO
from PIL import Image
# Morphological filtering
from skimage.morphology import opening
from skimage.morphology import disk
# Data handling
import numpy as np
# Connected component filtering
import cv2
black = 0
white = 255
threshold = 160
# Open input image in grayscale mode and get its pixels.
img = Image.open("image3.png").convert("LA")
pixels = np.array(img)[:,:,0]
# Remove pixels above threshold
pixels[pixels > threshold] = white
pixels[pixels < threshold] = black
# Morphological opening
blobSize = 1 # Select the maximum radius of the blobs you would like to remove
structureElement = disk(blobSize) # you can define different shapes, here we take a disk shape
# We need to invert the image such that black is background and white foreground to perform the opening
pixels = np.invert(opening(np.invert(pixels), structureElement))
# Create and save new image.
newImg = Image.fromarray(pixels).convert('RGB')
newImg.save("newImage1.PNG")
# Find the connected components (black objects in your image)
# Because the function searches for white connected components on a black background, we need to invert the image
nb_components, output, stats, centroids = cv2.connectedComponentsWithStats(np.invert(pixels), connectivity=8)
# For every connected component in your image, you can obtain the number of pixels from the stats variable in the last
# column. We remove the first entry from sizes, because this is the entry of the background connected component
sizes = stats[1:,-1]
nb_components -= 1
# Define the minimum size (number of pixels) a component should consist of
minimum_size = 100
# Create a new image
newPixels = np.ones(pixels.shape)*255
# Iterate over all components in the image, only keep the components larger than minimum size
for i in range(1, nb_components):
if sizes[i] > minimum_size:
newPixels[output == i+1] = 0
# Create and save new image.
newImg = Image.fromarray(newPixels).convert('RGB')
newImg.save("newImage2.PNG")
But it returns:
I would prefer it not to be black and white, the best output would be one which upscale both text and image
As mentioned in the comments, the quality is very bad. This is not an easy problem. However, there may be a couple of tricks you can try.
This looks like it is due to some anti-aliasing that has been applied to the image/scan. I would try reversing anti-aliasing if possible. As descriped in the post the steps would be similar to this:
Apply low pass filter
difference = original_image - low_pass_image
sharpened_image = original_image + alpha*difference
Code may look something like this:
from skimage.filters import gaussian
alpha = 1 # Sharpening factor
low_pass_image = gaussian(image, sigma=1)
difference = original_image - low_pass_image
sharpened_image = original_image + alpha*difference
Also, scikit image has an implementation of an unsharp mask as well as the wiener filter.
I'm Marius, a maths student in the first year.
We have recieved a team-assignment where we have to implement a fourier transformation and we chose to try to encode the transformation of an image to a JPEG image.
to simplify the problem for ourselves, we chose to do it only for pictures that are greyscaled.
This is my code so far:
from PIL import Image
import numpy as np
import sympy as sp
#
#ALLEMAAL INFORMATIE GEEN BEREKENINGEN
img = Image.open('mario.png')
img = img.convert('L') # convert to monochrome picture
img.show() #opens the picture
pixels = list(img.getdata())
print(pixels) #to see if we got the pixel numeric values correct
grootte = list(img.size)
print(len(pixels)) #to check if the amount of pixels is correct.
kolommen, rijen = img.size
print("het aantal kolommen is",kolommen,"het aantal rijen is",rijen)
#tot hier allemaal informatie
pixelMatrix = []
while pixels != []:
pixelMatrix.append(pixels[:kolommen])
pixels = pixels[kolommen:]
print(pixelMatrix)
pixelMatrix = np.array(pixelMatrix)
print(pixelMatrix.shape)
Now the problem forms itself in the last 3 lines. I want to try to convert the matrix of values back into an Image with the matrix 'pixelMatrix' as it's value.
I've tried many things, but this seems to be the most obvious way:
im2 = Image.new('L',(kolommen,rijen))
im2.putdata(pixels)
im2.show()
When I use this, it just gives me a black image of the correct dimensions.
Any ideas on how to get back the original picture, starting from the values in my matrix pixelMatrix?
Post Scriptum: We still have to implement the transformation itself, but that would be useless unless we are sure we can convert a matrix back into a greyscaled image.
import pytesseract
from PIL import Image
def textFromTesseractOCR(croppedImage):
for i in range(14):
text = pytesseract.image_to_string(croppedImage, lang = 'eng', boxes = False ,config = '--psm '+ str(i) +' --oem 3')
print("PSM Mode", i)
print("Text detected: ",text)
imgPath = "ImagePath" #you can use image I have uploaded
img = Image.open(imgPath)
textFromTesseractOCR(img)
I am working on extracting Table data from PDF. For this I am converting pdf to png. Detecting Lines, ascertaining table by line intersection and then cropping individual cells to get their text.
This all works fine, but tesseract is not working on cells image which has text of length 2 or less.
Works for this image:
Result from tesseract:
Does not work for this image:
Result from tesseract: return empty string.
It also returns empty for numbers of text length 2 or less.
I have tried resizing the image(which I knew wouldn't work), also tried appending dummy text to the image but the result was bad(was working only for few and I didn't the exact location to append the dummy text in the image)
It would be great if someone could help me with this.
So I finally came with a workaround for this situation. The situation being tesseract-OCR giving empty string when the image contains only 1 or 2 length string(eg "1" or "25").
To get output in this situation I appended the same image multiple time at the original image so as to make its length greater than 2. For example, if the original image contained only "3", I appended "3" image(the same image) 4 more times and thereby making it an image which contains the text "33333". We then give this image to tesseract which gives output "33333"(most of the times).Then we just have to replace space with blank in the text output from the Tesseract and divide the resulting string length by 5 to get the index up to which we would want to text out from the whole text.
Please see code for reference, hope this helps:
import pytesseract ## pip3 install pytesseract
Method which calls tesseract for OCR or calls our workaround code if we get an empty string from tesseract output.
def textFromTesseractOCR(croppedImage):
text = pytesseract.image_to_string(croppedImage)
if text.strip() == '': ### program that handles our problem
if 0 not in croppedImage:
return ""
yDir = 3
xDir = 3
iterations = 4
img = generate_blocks_dilation(croppedImage, yDir, xDir, iterations)
## we dilation to get only the text portion of the image and not the whole image
kernelH = np.ones((1,5),np.uint8)
kernelV = np.ones((5,1),np.uint8)
img = cv2.dilate(img,kernelH,iterations = 1)
img = cv2.dilate(img,kernelV,iterations = 1)
image = cropOutMyImg(img, croppedImage)
concateImg = np.concatenate((image, image), axis = 1)
concateImg = np.concatenate((concateImg, image), axis = 1)
concateImg = np.concatenate((concateImg, image), axis = 1)
concateImg = np.concatenate((concateImg, image), axis = 1)
textA = pytesseract.image_to_string(concateImg)
textA = textA.strip()
textA = textA.replace(" ","")
textA = textA[0:int(len(textA)/5)]
return textA
return text
Method for dilation.This method is used to dilate only the text region of the image
def generate_blocks_dilation(img, yDir, xDir, iterations):
kernel = np.ones((yDir,xDir),np.uint8)
ret,img = cv2.threshold(img, 0, 1, cv2.THRESH_BINARY_INV)
return cv2.dilate(img,kernel,iterations = iterations)
Method to crop the dilated part of the image
def cropOutMyImg(gray, OrigImg):
mask = np.zeros(gray.shape,np.uint8) # mask image the final image without small pieces
_ , contours, hierarchy = cv2.findContours(gray,cv2.RETR_LIST,cv2.CHAIN_APPROX_SIMPLE)
for cnt in contours:
if cv2.contourArea(cnt)!=0:
cv2.drawContours(mask,[cnt],0,255,-1) # the [] around cnt and 3rd argument 0 mean only the particular contour is drawn
# Build a ROI to crop the QR
x,y,w,h = cv2.boundingRect(cnt)
roi=mask[y:y+h,x:x+w]
# crop the original QR based on the ROI
QR_crop = OrigImg[y:y+h,x:x+w]
# use cropped mask image (roi) to get rid of all small pieces
QR_final = QR_crop * (roi/255)
return QR_final
I tried running tesseract on given 2 image but it does not returns text in shorter text image.
Another thing you can try is "Train a machine learning model (probably neural net) to on alphabets, numbers and special character, then when you want to get text from image, feed that image to model and it will predict text/characters."
Training dataset would look like :
Pair of (Image of character, 'character').
First element of pair is independent variable for model.
Second element of pair is corresponding character present in that image. It will be dependent variable for model.