How do I tokenize a string sentence in NLTK?

This is actually on the main page of nltk.org:

>>> import nltk
>>> sentence = """At eight o'clock on Thursday morning
... Arthur didn't feel very good."""
>>> tokens = nltk.word_tokenize(sentence)
>>> tokens
['At', 'eight', "o'clock", 'on', 'Thursday', 'morning',
'Arthur', 'did', "n't", 'feel', 'very', 'good', '.']

Related Contents:

  1. How to get rid of punctuation using NLTK tokenizer?
  2. How to use Stanford Parser in NLTK using Python
  3. Python NLTK pos_tag not returning the correct part-of-speech tag
  4. Creating a new corpus with NLTK
  5. Stopword removal with NLTK
  6. Ordinal numbers replacement
  7. How to config nltk data directory from code?
  8. Classification using movie review corpus in NLTK/Python
  9. Convert words between verb/noun/adjective forms
  10. nltk NaiveBayesClassifier training for sentiment analysis
  11. NLTK Named Entity recognition to a Python list
  12. Using NLTK and WordNet; how do I convert simple tense verb into its present, past or past participle form?
  13. Python – RegEx for splitting text into sentences (sentence-tokenizing) [duplicate]
  14. Computing N Grams using Python
  15. English grammar for parsing in NLTK
  16. NLTK and language detection
  17. Fast/Optimize N-gram implementations in python
  18. Extract Word from Synset using Wordnet in NLTK 3.0
  19. How to apply NLTK word_tokenize library on a Pandas dataframe for Twitter data?
  20. How do I do dependency parsing in NLTK?
  21. training data format for NLTK punkt
  22. How to tweak the NLTK sentence tokenizer
  23. Refering to a directory in a Flask app doesn’t work unless the path is absolute
  24. Calculate cosine similarity given 2 sentence strings
  25. How do I download NLTK data?
  26. How to compute the similarity between two text documents?
  27. How to remove stop words using nltk or python
  28. How to determine the language of a piece of text?
  29. Why is my NLTK function slow when processing the DataFrame?
  30. n-grams in python, four, five, six grams?
  31. wordnet lemmatization and pos tagging in python
  32. Failed loading english.pickle with nltk.data.load
  33. How do I compute the similarity between two text documents?
  34. Python: tf-idf-cosine: to find document similarity
  35. All synonyms for word in python? [duplicate]
  36. Opening A large JSON file
  37. NLTK-based text processing with pandas
  38. Fuzzy String Comparison
  39. NLTK Tagging spanish words using a corpus
  40. Counting the Frequency of words in a pandas data frame
  41. Efficiently count word frequencies in python
  42. nltk doesn’t add $NLTK_DATA to search path?
  43. LDA model generates different topics everytime i train on the same corpus
  44. error installing nltk supporting packages : nltk.download()
  45. NLTK download SSL: Certificate verify failed
  46. Extract list of Persons and Organizations using Stanford NER Tagger in NLTK
  47. How can I split a string of a mathematical expressions in python?
  48. How to get all the hyponyms of a word/synset in python nltk and wordnet?
  49. Spell Checker for Python
  50. Tokenize a paragraph into sentence and then into words in NLTK

Leave a Comment