A common solution for tokenization and part-of-speech tagging: one-pass Viterbi algorithm vs. Iterative approaches
Ver/Abrir
Use este enlace para citar
http://hdl.handle.net/2183/147Colecciones
- GI-COLE - Artigos [10]
Metadatos
Mostrar el registro completo del ítemTítulo
A common solution for tokenization and part-of-speech tagging: one-pass Viterbi algorithm vs. Iterative approachesFecha
2002Cita bibliográfica
Proceedings of the Fifth International Conference on Text, Speech and Dialogue (TSD-2002), Brno (Czech Republic).Sojka, P.; Kopecek, I.; Pala, K. (eds.). Lecture Notes in Artificial Intelligence, vol. 2448, pp. 3-10.
Resumen
Current taggers assume that input texts are already tokenized, i.e. correctly segmented in \emph{tokens} or high level information units that identify each individual component of the texts. This working hypothesis is unrealistic, due to the heterogeneous nature of the application texts and their sources. The greatest troubles arise when this segmentation is ambiguous. The choice of the correct segmentation alternative depends on the context, which is precisely what taggers study. In this work, we develop a tagger able not only to decide the tag to be assigned to every token, but also to decide whether some of them form or not the same term, according to different segmentation alternatives. For this task, we design an extension of the Viterbi algorithm able to evaluate streams of tokens of different lengths over the same structure. We also compare its time and space complexities with those of the classic and iterative versions of the algorithm.
ISSN
0302-9743
ISBN
3-540-44129-8