How do I specify a range of unicode characters

The syntax of your unicode range will not do what you expect.

  1. The raw r'' string prevents \u escapes from being parsed, and the regex engine will not do this. The only range in this set is [0-\]:

    >>> re.compile(r'[\u0020-\u00d7ff]', re.DEBUG)
    in
      literal 117
      literal 48
      literal 48
      literal 50
      range (48, 117)
      literal 48
      literal 48
      literal 100
      literal 55
      literal 102
      literal 102
    
  2. Making it a Unicode literal causes \u parsing while leaving other backslashes alone (although that’s not a concern here), but the leading zeroes are messing it up. The syntax is \uxxxx or \Uxxxxxxxx, so it’s parsed as “\u00d7, f, f“.

    >>> re.compile(ur'[\u0020-\u00d7ff]', re.DEBUG)
    in
      range (32, 215)
      literal 102
      literal 102
    
  3. Removing the leading zeroes or switching to \U0000d7ff will fix it:

    >>> re.compile(ur'[\u0020-\ud7ff]', re.DEBUG)
    in
      range (32, 55295)
    

Leave a Comment