Since no one has ever bothered to post the code to do this, here it is: // \p{Mn} or \p{Non_Spacing_Mark}: // a character intended to be combined with another // character without taking up extra space // (e.g. accents, umlauts, etc.). private readonly static Regex nonSpacingMarkRegex = new Regex(@”\p{Mn}”, RegexOptions.Compiled); public static string RemoveDiacritics(string text) … Read more
This should be useful which handles almost all the cases. function Unaccent($string) { return preg_replace(‘~&([a-z]{1,2})(?:acute|cedil|circ|grave|lig|orn|ring|slash|th|tilde|uml|caron);~i’, ‘$1’, htmlentities($string, ENT_COMPAT, ‘UTF-8’)); }
There’s no ambiguity here: RFC3986 says no, that is, URIs cannot contain unicode characters, only ASCII. An entirely different matter is how browsers represent encoded characters when displaying a URI, for example some browsers will display a space in a URL instead of ‘%20’. This is how IDN works too: punycoded strings are encoded and … Read more
As of Mongo 3.2, you can use $text and set $diacriticSensitive to false: { $text: { $search: <string>, $language: <string>, $caseSensitive: <boolean>, $diacriticSensitive: <boolean> } } See more in the Mongo docs: https://docs.mongodb.com/manual/reference/operator/query/text/
I recommend using Unidecode module: >>> from unidecode import unidecode >>> unidecode(u’ıöüç’) ‘iouc’ Note how you feed it a unicode string and it outputs a byte string. The output is guaranteed to be ASCII.
Finally, I’ve solved it by using the Normalizer class. import java.text.Normalizer; public static String stripAccents(String s) { s = Normalizer.normalize(s, Normalizer.Form.NFD); s = s.replaceAll(“[\\p{InCombiningDiacriticalMarks}]”, “”); return s; }
The standard ecmascript regex isn’t ready for unicode (see http://blog.stevenlevithan.com/archives/javascript-regex-and-unicode). So you have to use an external regex library. I used this one (with the unicode plugin) in the past : http://xregexp.com/ In your case, you may have to escape the char é as \u00E9 and defining a range englobing e, é, ê, etc. EDIT … Read more
if you have http://php.net/manual/en/book.intl.php available, you can use this: $string = “Fóø Bår”; $transliterator = Transliterator::createFromRules(‘:: Any-Latin; :: Latin-ASCII; :: NFD; :: [:Nonspacing Mark:] Remove; :: Lower(); :: NFC;’, Transliterator::FORWARD); echo $normalized = $transliterator->transliterate($string);
JavaScript’s regex implementation is not Unicode-aware. It only knows the ‘word characters’ in standard low-byte ASCII, which does not include é or any other accented or non-English letters. Because é is not a word character to JS, é followed by a space can never be considered a word boundary. (It would match \b if used … Read more
SQL Fiddle: http://sqlfiddle.com/#!6/9eecb7d/1607 declare @a nvarchar(32) = ‘àéêöhello!’ declare @b nvarchar(32) = ‘aeeohello!’ select case when (cast(@a as varchar(32)) collate SQL_Latin1_General_Cp1251_CS_AS) = @a then 0 else 1 end HasSpecialChars select case when (cast(@b as varchar(32)) collate SQL_Latin1_General_Cp1251_CS_AS) = @b then 0 else 1 end HasSpecialChars (based on solution here: How can I remove accents on … Read more