If you’re just doing small batches that are relatively well-formed, then the compare.linkage() or compare.dedup() functions in the RecordLinkage package should be a great starting point. But if you have big batches, then you might have to do some more tinkering. I use the functions jarowinkler(), levenshteinSim(), and soundex() in RecordLinkage to write my own … Read more
Aggregate the column by COUNT, then use a HAVING clause to find values that appear greater than one time. SELECT column_name, COUNT(column_name) FROM table_name GROUP BY column_name HAVING COUNT(column_name) > 1;
Do dtEmp on your current working DataTable: DataTable distinctTable = dtEmp.DefaultView.ToTable( /*distinct*/ true); It’s nice.
Use array_unique(). Example: $array = array(1, 2, 2, 3); $array = array_unique($array); // Array is now (1, 2, 3)
Here is another way. No intermediate variables are saved. We used this to de-duplicate results from a variety of overlapping queries. $input = array_map(“unserialize”, array_unique(array_map(“serialize”, $input)));