POS tagger and lemmatizer evaluation: A TextFlows open science approach

author: Matej Martinc, Department of Knowledge Technologies, Jožef Stefan Institute
published: May 23, 2017,   recorded: April 2017,   views: 1216


Related Open Educational Resources

Related content

Report a problem or upload files

If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.
Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.
Lecture popularity: You need to login to cast your vote.


Part of speech (POS) tagging (assigning the word class to a word) and lemmatization (converting the word to its base form) are core techniques used in natural language processing (NLP) and text mining. Therefore the performance of POS taggers and lemmatizers is important and can have a significant impact on the success of NLP and text mining tasks where these operations are used. Over the years, many POS tagging and lemmatization algorithms have been developed, which are publicly available in open source natural language processing libraries. Open source libraries provide the researchers with accessible NLP tools, but further work on evaluating these tools is still needed, since there is insufficient documentation on the comparison of different POS tagging and lemmatization tools in terms of their effectiveness.

For our research we conducted an experimental evaluation of several publicly available open source POS taggers and lemmatizers on a number of annotated evaluation corpora. Several POS taggers were evaluated and the influence of various factors, such as training corpus length, training corpus genre and the use of pretrained models has been tested. Results have shown non-linear dependence between the size of the training set and the accuracy of the POS taggers, domain-specificity of POS taggers and that the use of pretrained models is not always appropriate. In lemmatizer evaluation, we have shown that preprocessing steps, such as conversion of text to lowercase, have a significant influence on the performance of some lemmatizers. In extrinsic evaluation, where the performance of the tool is measured indirectly (in the context of a larger text mining task), we tested the POS taggers' influence on the lemmatization and gender classification tasks, while lemmatizers and stemmers were evaluated through text categorization.

The experiments were conducted in the TextFlows platform, which is a cloud-based web application for composition, execution and sharing of interactive text mining workflows. The POS tagger and lemmatizer implementations in this online platform enable transparent and reproducible results, reusability of the workflow components for new tasks and provide precomposed workflows for algorithm comparison and evaluation, enabling future algorithms to be tested in the same environment.

See Also:

Download slides icon Download slides: ipssc2017_martinc_textflows_01.pdf (763.5 KB)

Help icon Streaming Video Help

Link this page

Would you like to put a link to this lecture on your homepage?
Go ahead! Copy the HTML snippet !

Write your own review or comment:

make sure you have javascript enabled or clear this field: