Introduction

Tokenizing

Obsolete: The tokenizer file tok.txt

Just as for Northern Sámi, the Lule Sámi preprocessing was earlier done with the Xerox tokenize tool and the language-specific file tok.txt. The code itself is written as a set of regular expressions, and the source file (tok.txt) was compiled by xfst. As explained for the sme preprocessing, this approach was replaced by a preprocessor script, written in perl, gt/script/preprocess.

The current preprocessor

Preprocessing is done by the perl script gt/script/preprocess, which is language-independent. The script is documented here. The language dependent part of the script shall be done via a

Handling abbreviations

On the other hand side, as long as we have neither lexicon nor texts for Lule Sámi, we have no need for a Lule Sámi preprocessor either.

Spell relaxation of æ/ä, ø/ö

This is a feature common to Lule and Southern Sami, not to be found in Northern Sami. The letter æ/ä and ø/ö are used interchangeably in Norway and Sweden. The parser should thus accept any version of them.

The xfst file to handle this is short, it consists of one line:

ń (->) ñ, ŋ (->) ñ, æ (->) ä, ø (->) ö ;

The line says that æ may optionally be replaced by ä and that ø may optionally be replaced with ö, and the same for the different ways of writing ŋ. A problem with this script is that it does not work, although it worked earlier in 2005.

Initial capitalization

This works like the Northern Sami file, cf. the documentation for Northern Sami initial capitalization. The content of the file is slightly different, since the letter repertoire is different as well.

Capitalization of whole words

This has not yet been implemented.

Last modified $Date: 2005/09/06 11:13:32 $, by $Author: boerre $