| 1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192939495969798 |
- Pretrained Punkt Models -- Jan Strunk (New version trained after issues 313 and 514 had been corrected)
- Most models were prepared using the test corpora from Kiss and Strunk (2006). Additional models have
- been contributed by various people using NLTK for sentence boundary detection.
- For information about how to use these models, please confer the tokenization HOWTO:
- http://nltk.googlecode.com/svn/trunk/doc/howto/tokenize.html
- and chapter 3.8 of the NLTK book:
- http://nltk.googlecode.com/svn/trunk/doc/book/ch03.html#sec-segmentation
- There are pretrained tokenizers for the following languages:
- File Language Source Contents Size of training corpus(in tokens) Model contributed by
- =======================================================================================================================================================================
- czech.pickle Czech Multilingual Corpus 1 (ECI) Lidove Noviny ~345,000 Jan Strunk / Tibor Kiss
- Literarni Noviny
- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- danish.pickle Danish Avisdata CD-Rom Ver. 1.1. 1995 Berlingske Tidende ~550,000 Jan Strunk / Tibor Kiss
- (Berlingske Avisdata, Copenhagen) Weekend Avisen
- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- dutch.pickle Dutch Multilingual Corpus 1 (ECI) De Limburger ~340,000 Jan Strunk / Tibor Kiss
- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- english.pickle English Penn Treebank (LDC) Wall Street Journal ~469,000 Jan Strunk / Tibor Kiss
- (American)
- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- estonian.pickle Estonian University of Tartu, Estonia Eesti Ekspress ~359,000 Jan Strunk / Tibor Kiss
- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- finnish.pickle Finnish Finnish Parole Corpus, Finnish Books and major national ~364,000 Jan Strunk / Tibor Kiss
- Text Bank (Suomen Kielen newspapers
- Tekstipankki)
- Finnish Center for IT Science
- (CSC)
- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- french.pickle French Multilingual Corpus 1 (ECI) Le Monde ~370,000 Jan Strunk / Tibor Kiss
- (European)
- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- german.pickle German Neue Zürcher Zeitung AG Neue Zürcher Zeitung ~847,000 Jan Strunk / Tibor Kiss
- (Switzerland) CD-ROM
- (Uses "ss"
- instead of "ß")
- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- greek.pickle Greek Efstathios Stamatatos To Vima (TO BHMA) ~227,000 Jan Strunk / Tibor Kiss
- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- italian.pickle Italian Multilingual Corpus 1 (ECI) La Stampa, Il Mattino ~312,000 Jan Strunk / Tibor Kiss
- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- norwegian.pickle Norwegian Centre for Humanities Bergens Tidende ~479,000 Jan Strunk / Tibor Kiss
- (Bokmål and Information Technologies,
- Nynorsk) Bergen
- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- polish.pickle Polish Polish National Corpus Literature, newspapers, etc. ~1,000,000 Krzysztof Langner
- (http://www.nkjp.pl/)
- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- portuguese.pickle Portuguese CETENFolha Corpus Folha de São Paulo ~321,000 Jan Strunk / Tibor Kiss
- (Brazilian) (Linguateca)
- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- slovene.pickle Slovene TRACTOR Delo ~354,000 Jan Strunk / Tibor Kiss
- Slovene Academy for Arts
- and Sciences
- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- spanish.pickle Spanish Multilingual Corpus 1 (ECI) Sur ~353,000 Jan Strunk / Tibor Kiss
- (European)
- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- swedish.pickle Swedish Multilingual Corpus 1 (ECI) Dagens Nyheter ~339,000 Jan Strunk / Tibor Kiss
- (and some other texts)
- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- turkish.pickle Turkish METU Turkish Corpus Milliyet ~333,000 Jan Strunk / Tibor Kiss
- (Türkçe Derlem Projesi)
- University of Ankara
- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- The corpora contained about 400,000 tokens on average and mostly consisted of newspaper text converted to
- Unicode using the codecs module.
- Kiss, Tibor and Strunk, Jan (2006): Unsupervised Multilingual Sentence Boundary Detection.
- Computational Linguistics 32: 485-525.
- ---- Training Code ----
- # import punkt
- import nltk.tokenize.punkt
- # Make a new Tokenizer
- tokenizer = nltk.tokenize.punkt.PunktSentenceTokenizer()
- # Read in training corpus (one example: Slovene)
- import codecs
- text = codecs.open("slovene.plain","Ur","iso-8859-2").read()
- # Train tokenizer
- tokenizer.train(text)
- # Dump pickled tokenizer
- import pickle
- out = open("slovene.pickle","wb")
- pickle.dump(tokenizer, out)
- out.close()
- ---------
|
