I am learning python and I am trying to do some text preprocessing and I have been reading and borrowing ideas from Stackoverflow. I was able to come up with the following formulations below, but they don't appear to do what I was expecting, and they don't throw any errors either, so I'm stumped.
First, in a Pandas dataframe column, I am trying to remove the third consecutive character in a word; it's kind of like running a spell check on words that are supposed to have two consecutive characters instead of three
buttter = butter
bettter = better
ladder = ladder
The code I used is below:
import re
docs['Comments'] = [c for c in docs['Comments'] if re.sub(r'(\w)\1{2,}', r'\1', c)]
In the second instance, I just want to to replace multiple punctuations with the last one.
????? = ?
..... = .
!!!!! = !
---- = -
***** = *
And the code I have for that is:
docs['Comments'] = [i for i in docs['Comments'] if re.sub(r'[\?\.\!\*]+(?=[\?\.\!\*])', '', i)]
It looks like you want to use
docs['Comments'] = docs['Comments'].str.replace(r'(\w)\1{2,}', r'\1\1', regex=True)
.str.replace(r'([^\w\s]|_)(\1)+', r'\2', regex=True)
The r'(\w)\1{2,}' regex finds three or more repeated word chars and \1\1 replaces with two their occurrences. See this regex demo.
The r'([^\w\s]|_)(\1)+' regex matches repeated punctuation chars and captures the last into Group 2, so \2 replaces the match with the last punctuation char. See this regex demo.
Related
Assume there's a string
"An example striiiiiing with other words"
I need to replace the 'i's with '*'s like 'str******ng'. The number of '*' must be same as 'i'. This replacement should happen only if there are consecutive 'i' greater than or equal to 3. If the number of 'i' is less than 3 then there is a different rule for that. I can hard code it:
import re
text = "An example striiiiing with other words"
out_put = re.sub(re.compile(r'i{3}', re.I), r'*'*3, text)
print(out_put)
# An example str***iing with other words
But number of i could be any number greater than 3. How can we do that using regex?
The i{3} pattern only matches iii anywhere in the string. You need i{3,} to match three or more is. However, to make it all work, you need to pass your match into a callable used as a replacement argument to re.sub, where you can get the match text length and multiply correctly.
Also, it is advisable to declare the regex outside of re.sub, or just use a string pattern since patterns are cached.
Here is the code that fixes the issue:
import re
text = "An example striiiiing with other words"
rx = re.compile(r'i{3,}', re.I)
out_put = rx.sub(lambda x: r'*'*len(x.group()), text)
print(out_put)
# => An example str*****ng with other words
I have some strings, some of which are gibberish, a mixture of digits and letters. The gibberish, I would like to remove, but those with a pattern, I would like to keep.
I am providing an example for illustrative purposes.
strings = ["1Z83E0590391137855",
"55t5555t5t5tttt5t5555tttttttgggggggggggggggsss",
"1st", "2nd", "3rd", "4th", "5th"
]
import pandas as pd
df = pd.DataFrame(strings, columns=['strs'])
df
I would like to remove strings that look like
1Z83E0590391137855
55t5555t5t5tttt5t5555tttttttgggggggsss
and keep strings that look like ones below
1st
2nd
3rd
4th
5th
Given my limited regex and python experience, I am having some difficulty coming up with the right formulation. What I have tried, has removed everything, except the first row:
df['strs'] = df['strs'].str.replace(r'(?=.*[a-z])(?=.*[\d])[a-z\d]+', '', regex=True)
I suggest only matching alphanumeric strings containing both letters and digits that contain a certain amount of chars.
In the example below, I set the threshold to 18, i.e. the strings shorter than 18 chars won't be matched and thus will remain in the column. All the strings equal or longer will get removed:
df['strs'] = df['strs'].str.replace(r'^(?=.{18})(?:[a-zA-Z]+\d|\d+[a-zA-Z])[a-zA-Z\d]*$', '', regex=True)
Details:
^ - start of string
(?=.{18}) - the string must start with 18 chars other than line break chars
(?:[a-zA-Z]+\d|\d+[a-zA-Z]) - one or more letters and then a digit or one or more digits and then a letter
[a-zA-Z\d]* - zero or more alphanumeric chars
$ - end of string.
See the regex demo.
You could check that the line does not start with 1st 2nd.. to remove only those lines.
^(?!\d+(?:st|nd|rd|th)$).*$
Regex demo
I want to write a function that converts the given string T and group them into three blocks.
However, I want to split the last block into two if it can't be broken down to three numbers.
For example, this is my code
import re
def num_format(T):
clean_number = re.sub('[^0-9]+', '', T)
formatted_number = re.sub(r"(\d{3})(?=(\d{3})+(?!\d{3}))", r"\1-", clean_number)
return formatted_number
num_format("05553--70002654")
this returns : '055-537-000-2654' as a result.
However, I want it to be '055-537-000-26-54'.
I used the regular expression, but have no idea how to split the last remaining numbers into two blocks!
I would really appreciate helping me to figure this problem out!!
Thanks in advance.
You can use
def num_format(T):
clean_number = ''.join(c for c in T if c.isdigit())
return re.sub(r'(\d{3})(?=\d{2})|(?<=\d{2})(?=\d{2}$)', r'\1-', clean_number)
See the regex demo.
Note you can get rid of all non-numeric chars using plain Python comprehension, the solution is borrowed from Removing all non-numeric characters from string in Python.
The regex matches
(\d{3}) - Group 1 (\1): three digits...
(?=\d{2}) - followed with two digits
| - or
(?<=\d{2})(?=\d{2}$) - a location between any two digit sequence and two digits that are at the end of string.
See the Python demo:
import re
def num_format(T):
clean_number = ''.join(c for c in T if c.isdigit())
return re.sub(r'(\d{3})(?=\d{2})|(?<=\d{2})(?=\d{2}$)', r'\1-', clean_number)
print(num_format("05553--70002654"))
# => 055-537-000-26-54
I have a list of strings something like this:
a=['bukt/id=gdhf/year=989/month=98/day=12/hgjhg.csv','bukt/id=76fhfh/year=989/month=08/day=128/hkngjhg.csv']
ids are unique.I want to have a output list which will be something like this
output_list = ['bukt/id=gdhf/','bukt/id=76fhfh/']
So basically need a regex expression to match any id and remove the rest of the part from the string
How can I do that in most efficient way considering the length of the input list is more than 100K
import re
rgx = r'(bukt/id=[a-zA-Z0-9]+/).+'
re.search(rgx, string).group(1)
The result will be in group 1. This captures "bukt/id=", followed by any alphanumeric characters and then a slash, and throws away the rest.
There's no need for regex, you can just split your string on /, discard everything after the second / and then join again with /:
a=['bukt/id=gdhf/year=989/month=98/day=12/hgjhg.csv','bukt/id=76fhfh/year=989/month=08/day=128/hkngjhg.csv']
out = ['/'.join(u.split('/')[:2]) for u in a]
print(out)
Output:
['bukt/id=gdhf', 'bukt/id=76fhfh']
If you want the trailing /, just add an empty string to the end of the split array:
out = ['/'.join(u.split('/')[:2] + ['']) for u in a]
Output:
['bukt/id=gdhf/', 'bukt/id=76fhfh/']
If I had the sentence sentence = 'There is light!' and I was to split this sentence with mysentence = sentence.split(), how would I have the output as 'There, is, light, !' of print(mysentence)? What I specifically wanted to do was split the sentence including all punctuation, or just a list of selected punctuation. I got some code but the program is recognizing the characters in the word, not the word.
out = "".join(c for c in punct1 if c not in ('!','.',':'))
out2 = "".join(c for c in punct2 if c not in ('!','.',':'))
out3 = "".join(c for c in punct3 if c not in ('!','.',':'))
How would I use this without recognizing each character in a word, but the word itself. Therefore, the output of "Hello how are you?" should become "Hello, how, are, you, ?" Any way of doing this
You may use a \w+|[^\w\s]+ regex with re.findall to get those chunks:
\w+|[^\w\s]
See the regex demo
Pattern details:
\w+ - 1 or more word chars (letters, digits or underscores)
| - or
[^\w\s] - 1 char other than word / whitespace
Python demo:
import re
p = re.compile(r'\w+|[^\w\s]')
s = "There is light!"
print(p.findall(s))
NOTE: If you want to treat an underscore as punctuation, you need to use something like [a-zA-Z0-9]+|[^A-Za-z0-9\s] pattern.
UPDATE (after comments)
To make sure you match an apostrophe as part of the words, add (?:'\w+)* or (?:'\w+)? to the \w+ in the pattern above:
import re
p = re.compile(r"\w+(?:'\w+)*|[^\w\s]")
s = "There is light!? I'm a human"
print(p.findall(s))
See the updated demo
The (?:'\w+)* matches zero or more (*, if you use ?, it will match 1 or 0) occurrences of an apostrophe followed with 1+ word characters.