I am working on a dataset which has data (text) entries captured in different styles like we see in the table below in 1000's of rows:
**School Name **
Abirem school
Abirem sec School
Abirem Secondary school
Abirem second. School
Metropolitan elementary
Metropolitan Element.
Metropolitan ele
I need help to extract the unique data values within a group of similar entries regardless of the style it was entered. The output I want should look like we see below:
**School Name **
Abirem school
Metropolitan elementary
I have tried using the functions; EXACT, UNIQUE, MATCH and even XLOOKUP (with the wildcard option) but none of them gives me the output I want.
Is there a logical function that can be used?
This will prove to be tricky. Excel would not know wheather or not two different names that look similar are actually meant to be similar. Even for us humans it will become trivial. I mean; would School1 ABC be similar to School1 DEF or not? Without actually knowing geographical locations about these two schools these could well be two different schools with a similar first word in their names.
Either way, if you happen to be willing to accept this ambiguity you could make a match on the 1st word of each line here and return only those where they match first:
Formula in C1:
=LET(a,A1:A7,UNIQUE(XLOOKUP(TEXTSPLIT(a," ")&" *",a&" ",a,,2)))
I have 100 people and I want them to judge words as either positive or negative (e.g. 'insurance' and 'car accident'). I have a total of 100 of such words. I also want each person to do three words as I am interested in some statistical properties (i.e. seeing how well people agree).
I want assign words to people by creating three columns with the same words in each column. However, I want words to randomized in a way so that there is no repetition in any row. Randomization is obviously important as I want to avoid any bias, but it would be silly to ask the same person the same two (or worse, three) words.
So, here is the data structure that I try to achieve:
person1, word1, word65, word33;
person2, word55, word56, word44;
person3, word23, word23, word3; <--- This should not happen
Is there a simple formula or other way to do this form of column-spanning randomization without repetition in LibreOffice Calc or Excel?
Thanks in advance!
What you need is a random permutation of the words that you type in difference cells. You can do this task using the Libreoffice extension Permutate! (download here: https://sourceforge.net/projects/permutate/). Since I am the developer of this simple extension, please do not hesitate to ask for any clarifications.
Ok so, twice a month I receive a large file of about 100 rows, which contains 4 columns:
Building name - value - county - state
I´ve to complete 2 other columns based on a master list that have thousands of entries.
I want to produce something very similar to this fabulous add-in (http://www.microsoft.com/en-us/download/details.aspx?id=15011), but a bit simpler and that I could use at work without problems.
What I need to do is the following:
In order to match my input with the master file, I know the county and state must match, but then, the building names can change a bit in each file for the same building (ie "John Miller #34" can be "Miller, John 34 A"), and that the values may vary but not too much.
Based on that, I want to bring from the master to my file, all the entries that may match each of my rows, filtering by County and State first, and then by similarity in name and value.
Could you please share your thoughts on how you´d approach this?
I know this is not a simple thing, but anything may help!
You could also use wildcards to try and match on the primary identifier within the name. from your example, that might be "Miller", for example.
Unfortunately for you, the vlookup "fuzzy logic" is nowhere near reliable for your purpose (see the comment on my answer below for details), and you won't have any indicator as to whether the returned result is accurate or not.
It's possible to get 100% of what you want through some heavy coding in a user-defined function, but this is probably well beyond your comfort zone.
A clunky solution, although somewhat easy to explain and adopt, is to create an "identity column" for every unique scenario that can occur. So, for example:
Then you can import your master sheet and add the same identity column to the left, and perform your vlookup. When a new configuration is added you can just add that to the master list and it will populate in your imported file in future instances.
That said, if you are interested in learning, there have been many people who have walked in your shows and felt your pain. You may want to indulge in this:
http://www.mrexcel.com/forum/excel-questions/195635-fuzzy-matching-new-version-plus-explanation.html
Because what you are truly requesting is an algorithm. It's not a simple thing, but it's very possible. And if you take the time to learn you not only solve your immediate problem, but make yourself marketable as an Excel wiz. Good luck!
I'm working in SAS 9.2, in an existing dataset. I need a simple way to match a single word within string values of a single variable, and then replace entire string value with a blank. I don't have experience with SQL, macros, etc. and I'm hoping for a way to do this (even if the code is less efficient" that will be clear to a novice.
Specifically, I need to remove the entire string containing the word "growth" in a variable "pathogen." Sample values include "No growth during two days", "no growth," "growth did not occur," etc. I cannot enter all possible strings since I don't yet know how they will vary (we have only entered a few observations so far).
TRANSWD and TRANSLATE will not work as they will not allow me to replace an entire phrase when the target word is only a part of the string.
Other methods I've looked at (for example, a SESUG paper using PRX at http://analytics.ncsu.edu/sesug/2007/CC06.pdf) appear to remove all instances of the target string in every variable in the dataset, instead of just in the variable of interest.
Obviously I could subset the dataset to a single variable before I perform one of these actions and then merge back, but I'm hoping for something less complicated. Although I will certainly give something more complicated a shot if someone can provide me with sample code to adapt (and it would be greatly appreciated).
Thanks in advance--Kim
Could you be a little more clear on who the data set is constructed? I think mjsqu's solution will work if your variable pathogen is stored sentence by sentence. If not then I would say your best bet is to parse the blocks into sentences and then apply mjsqu's solution.
DATA dataset1;
format Ref best1.
pathogen $40.;
input Ref pathogen $40. ;
datalines;
1 No growth during two days
2 no growth,
3 growth did not occur,
4 does not have the word
;
RUN;
DATA dataout;
SET dataset1;
IF index(lowcase(pathogen),"growth") THEN pathogen="";
RUN;
I need to write an algorithm that returns the closest match for a contact based on the name and address entered by the user. Both of these are troubling, since there are so many ways to enter a company name and address, for instance:
Company A, 123 Any Street Suite 200, Anytown, AK 99012
Comp. A, 123 Any St., Suite 200, Anytown, AK 99012
CA, 123 Any Street Ste 200, Anytown, AK 99012
I have looked at doing a Levenshtein distance on the Name, but that doesn't seem a great tool, since they could abbreviate the name. I am looking for something that matches on the most information possible.
My initial attempt was to limit the results first by the first 5 digits of the postal code and then try to filter down to one based on other information, but there must be a more standard approach to getting this done. I am working in .NET but will look at any code you can provide to get an idea on how to accomplish this.
I don't exactly now how this is accomplished, but all major delivery companies (FedEx, USPS, UPS) seem to have a way of matching an address you input against their database and transforming it to a normalized form. As I've seen this happen on multiple websites (Amazon comes to mind), I am assuming that there is an API to this functionality, but I don't know where to look for it and whether it is suitable for your purposes.
Just a thought though.
EDIT: I found the USPS API
I have solved this problem with a combination of address normalization, Metaphone, and Levenshtein distance. You will need to separate the name from the address since they have different characteristics. Here are the steps you need to do:
1) Narrow down you list of matches by using the (first six characters of the) zip code. Basically you will need to calculate the Levenshtein distance of the two strings and select the ones that have a distance of 1 or 2 at the most. You can potentially precompute a table of zip codes and their "Levenshtein neighbors" if you really need to speed up the search.
http://en.wikipedia.org/wiki/Levenshtein_distance
2) Convert all the address abbreviations to a standard format using the list of official prefix and suffix abbreviations from the USPS. This will help make sure your results for the next step are more uniform:
https://www.usps.com/send/official-abbreviations.htm
3) Convert the address to a short code using the Methaphone algorithm. This will get rid of most common spelling mistakes. Just make sure that your implementation can eliminate all non word characters, pass numbers intact and handle multiple words (make sure each word is separated by a single space):
http://en.wikipedia.org/wiki/Metaphone
4) Once you have the Methaphone result of the compare the address strings using the Levenshtein distance. Calculate a percentage of change score by dividing the result by the number of characters in the longer string.
5) Repeat steps 3 and 4 but now use the names instead of the addresses.
6) Compute the score for each entry using this formula: (Weight for address * Address score) + (Weight for name * Name score). Pick your weights based on what is more important. I would start with .9 for the address (since the address is more specific) and .1 for the name but the weights may depend on your application. Pick the entry with the lowest score. If the score is too high (say over .15 you may declare that there are no matches).
I think filtering based on zip code first would be the easiest, as finding it is fairly unambiguous. From there you can probably extract the city and street. I'm not sure how you would go about finding the name, but it seems matching it against the address if you already have a database of (name, address) pairs is feasible.
Dun & Bradstreet do this. They charge money because it's really hard. There's no "standard" solution. It's mostly a painful choice between a service like D&B or roll your own.
As a start, I'd probably do a word-indexed search. That would mean two stages:
Offline stage: Generate an index of all the addresses by their keywords. For example, "Company", "A" and "123" would all become an keywords for the address you provided above. You could do some stemming, which would mean for words like "street" you'd also add a word "st" into its index.
Online stage: The user gives you a search query. Break down the search query into all its keywords, and find all possible matches of each keyword in the database. Tally the number of matched keywords on each address. Then sort the results by the number of matched keywords. This should be able to be done quite quickly if there aren't too many matches, as its just a few sorted list merges and increments, followed finally by a sort.
Given that you know the domain of your problem, you could specialise the algorithm to use knowledge about the domain - for example the zip code filtering mentioned before.
Also just to enable me to provide you with a better answer, are you using an SQL database at all? I ask because the way I would do it is I'd store the keyword index in the SQL database, and then the SQL query to search by keyword becomes quite easy, since the database does all the work.
Maybe instead of using Levenshtein for the name only, it could be useful when used with the entire string representation of a contact. For instance, the distance of your first example to the second is 7 and to the third 9. Considering the strings have lengths 54, 50 and 45, this seems to be a relatively useful and quite simple similarity measure.
This is what I would do. I am not aware of algorithms, so I just use what makes sense.
I am assuming that the person would provide name, street address, city name, state name, and zipcode.
If the zipcode is provided in 9 numbers, or has a hyphen, I would strip it down to 5 numbers. I would search the database for all of the addresses that has that zipcode.[query 1]
Then I would compare the state letter with the one from the database. If it's not a match, then I would tell that to the user. Same goes for the city name.
From what I understand, a street name is not in numbers, only the house on a street had numbers in it. Further more, the house number is usually at the beginning unless it is house or suite number.
So I would do regex to search for the numbers and the next space or comma next to it. Then find position of the first word that does not has a period(.) or ends in comma. I have part of the street name, so I could do a comparison against the rows fetched earlier, or I would change the query to have the street name LIKE %streetName%.
I am guessing the database has a beginning number and ending number of the house on a block. I would check against that street row to see if the provided street number is on that street.
By now you would know the correct data to show, and could look up in a different table as to which name is associated with that house number. I am not sure why you want to compare it. Only use for name comparing would be if you want to find people whose address was not provided. You can look here for comparing string ways Similar String algorithm
If you can reliably figure out general structure of each address (perhaps by the suggestions in the other answers), your best bet would be to run the data through a USPS-certified (meaning: the results are reliable, accurate, and conform to federal standards) address verification service.
#RyanDelucchi, it is a fun problem, but only once you've solved it. So, #SteveBering, I would recommend submitting your list of contacts to a list processing service which will flag duplicates based on the address -- according to USPS guidelines.
Since I work in the address verification field, I would suggest SmartyStreets (which I work for) since it will deliver the most value to your specific need -- however, there are a few CASS-Certified vendors who will do basically similar things.