Preprocesor for Sámi language tools

Usage:

Usage: preprocess [OPTIONS] --abbr=<file_name> FILE
Split text in FILE into sentences and words.
Options:
--hyph          Show the hyphenation points, i.e. change the <hyph> tags
                to hyphens. The default is to just remove the <hyph> tags.
--use-hyph-tags Leave the <hyph> tags untouched
--help          Print the help text and exit.
--v             Print information of the execution of the script
--corr=<file>   Use the list of common typos and their corrections (e.g. typos.txt)

Example use:

cat file.txt | preprocess --abbr=sme/bin/abbr.txt | lookup ...

Overview and intro

This document describes how the preprocessing of the text into separate words is implemented in the project. The document contains overall description of the preprocessing task and some implementation details.

Preprocessor is a Perl script preprocess. It takes text as input and produces a list of words (tokens) separated by newline. The preprocessor has two main functions: it cuts the the text into sentences and sentences into tokens (words and other units such as numerical expressions and punctuation). Sentence delimiters and most of the punctuation are treated as separate tokens.

The output of the preprocessor is a list of tokens separated by newline. The list of tokens is input to the morphological parser. (See the flowchart of the parsing process.) The parser gives each token a morphological analysis, a set of tags. For the analysis to be succesful, the preprocessor must be compatible with the parser; the preprocessor should produce tokens that are recognized by the parser. This means that for example multiword expressions like "earret eará" have to be recognized as a single token (not as two separate words) both by the preprocessor and the parser. To achieve this, the lexicon entries of "special" tokens, such as punctuation, abbreviations and multiword expressions are planned parallel with the preprocessor (see the section Abbreviations and the lexical files).

Dividing text to sentences

The text is divided to sentences according to sentence-delimiters (?,! and .); the question mark and exlamation mark are always sentence-delimiters, but each full stop is examined separately. Full stop may belong to an abbreviation that may or may not end the sentence. Similarly, a full stop that ends a numeral expression is sentence-delimiter only if the next word starts with a capital letter.

When a full stop that belongs to an abbreviation can be considered to be a sentence delimiter as well, an extra full stop is added to mark the sentence boundary. The sentence boundaries are relevant information later in CG. For example:

Vuođđologut (lohkosánit) leat: okta, guokte, golbma jna. Sulliilogu sánit leat: moadde, máŋga jna.
Vuođđologut
(
lohkosánit
)
leat
:
okta
,
guokte
,
golbma
jna.
.
Sulliilogu
sánit
leat
:
moadde
,
máŋga
jna.
.

The abbreviations are divided to three types according to their behaviour with other words. Some abbreviations which are followed by a capitalized word cannot be considered to end a sentence, see abbreviation section.

The corpus database used in the project contains additional paragraph marking and the end of paragraph is marked with ¶ in the input to the preprocessor. The paragraph end is at the same time a sentence end and makes it possible to consider e.g. titles as "sentences". The paragraph end mark sometimes adds an extra dot to the preprocessor output, sometimes it is recognized only later in the CG, these functions should be unified.

Tokens

The string that is given to the preprocessor is divided to tokens, which are the basic processing units used in the lexical analysis. There are four types of tokens: words, multiword expressions, numerical expressions and punctuation. Numerical expressions consist of numbers, dates, time, prices, etc. Words include abbreviations. The punctuation that does not belong to any of the mentioned expressions, forms its own class, which includes for example {}[]()?!;"

The starting point is that a string surrounded by space is one token. A string that consist only of alphabetical characters surrounded by space is always an instance of one word token, if it does not belong to a multiword expressions. So if two words are accidentally typed without separating space, it is an error that's not accounted for. Word tokens may contain some punctuation, such as hyphen: Davvi-Norga.

The punctuation that occurs in the text is divided to two classes: 1. those that are inseparable part of a word or numeric expression and 2. those that are independent. Independent punctuation may split the input string into two or more parts. For example the string (gielddat/guovllut). will be divided into six parts: both parentheses are their own tokens as well as the dot. Both words and the slash are also separate tokens. The preprocessor output is thus:

(
gielddat
/
guovllut
)
.

The splitting depends on the surrounding characters. For example, a dot rarely splits a numerical expression into two but generally splits a string that consist only of alphabetical characters (1.4.2004, Minä lähdin.En tule takaisin.). The treatment of punctuation is explained in detail later in the document.

The preprocess script

The preprocessor is a perl-script called preprocess; it reads STDIN for input. The preprocessor is given one command line parameter, the file from where to read the formerly generated list of abbreviations and multiword expressions. The file is created by make and the name is usually lang/bin/abbr.txt. Usage:

preprocess --abbr=sme/bin/abbr.txt

The other available flags are help, verbose and hyphenation points:

        --hyph          show the hyphenation points, i.e. change the "hyph" tags
                        to hyphens. The default is to just remove the "hyph" tags.
        --use-hyph-tags leave the "hyph" tags untouched
        --help          prints the help text and exits.
        --v             prints information of the execution of the script
        

The output of the script is a list of analyzable tokens, separated by newline.

The text is handled one paragraph at the time (paragraph ends when two subsequent newlines are encountered). If there are no paragraphs in the text, the whole input is slurped in in one go. The newlines are replaced with space and the paragraph is treated as a single line. The line is divided using space (tab, newline) as a separator and the elements stored into an array. Example:

First read the text. (Then split by space.)

First
read
the
text.
(Then
split
by
space.)

The elements in the array are processed one at the time. The element may consist of two or more tokens, the tokens are stored in an array. First, the punctuation preceding and not belonging to the token is removed and stored to the tokens array. This class of punctuation includes {}[]()?!;,'\". If the remaining part starts with an alphabetical character optionally preceded by one non-word character, it is considered to be a word token. Otherwise it is a numerical expression.

The word tokens are tested against the list of multiword expressions. The punctuation at the end of the word that does not belong to the token is stored to the tokens array. The word may be divided to several tokens if it contains delimiters such as /.

If the word contains a dot, it is tested against the list of abbreviations to see if the dot is a sentence delimiter or belongs to the token. The treatment of the numerical expressions differ in that most of the punctuation, such as /, is considered to be part of the token. The dot that follows a number is considered to be a sentence delimiter if the following word starts with a capital letter. Otherwise the dot belongs to the token which may be an ordinal number, date etc.

There are some constant variables that affect to the preprocessing, here are the current values:

$MULTIWORD_SIZE = 3;
$SINGLE_PUNCT contains:  «»‹›“”„‘’‚ ´`'" |{}[]()?!,;/\ … and • ¶
$CONTAIN_PUNCT = 'ja\/dahje|http|:\/\/|km\/h|www|@|\.jpg|\.doc|\.pdf|\.html|\.txt';

$MULTIWORD_SIZE defines the size of a multiword expression, i.e. how many words should be included in testing if the expression is a multiword. The constant $SINGLE_PUNCT contains all the punctuation that is considered to be individual tokens in the beginning or end of a word or numeral expression. In non-numeral expression, the punctuation in variable $SINGLE_PUNCT is searched also in the middle of the word, thus splitting it ("gielddat/guovllut"). The word tokens that contain punctuation which belongs to the token are defined in the constant $CONTAIN_PUNCT. This constant is currently the only language dependent part of the script.

Punctuation

As most of the punctuation in a numerical expression is inseparable part of it, the numerical expressions are treated as a separate class. The punctuation in other strings generally causes the string to be splitted, but not always, depending on the punctuation mark. In the following table, the punctuation marks are listed:

Punctuation	Numerical expressions (digits and non-word characters)	Words (word characters, no digits)
(){}[]– «»‹›“”„‘’‚ ´`'"?!,;\… • ¶	Always their own tokens.	Always their own tokens.
:	Belongs to the expression if not followed by space (14:30, 10:s)	Belongs to the expression if not followed by space (Namdal:as)
/	Belongs to the expression (5/2004)	Always its own token (gielddat/guovllut). Exceptions: ja/dahje which is treated as one token; html-addresses, file names etc.
-, --	Belongs to the expression, also when surrounded by spaces (1-3, 1 - 3)	Belongs to the expression (Davvi-Norga), but not in the beginning of the word.
.	Belongs to the expression unless it is a sentence delimiter.	Its own token. Exception: belongs to an abbreviation and does not end the sentence.
_*+=	Belongs to the expression	Always its own token
%	Belongs to the expression, also when separated by space (50%, 50 %)	Always its own token

Abbreviations

The abbreviations are divided in three classes according to whether they are able to end the sentence or not. The first class contains transitive abbreviations (TRAB) that never end the sentence. The intransitive abbreviations (ITRAB) are considered to end the sentence whenever they are followed by a capital letter or number. The third class (TRNUMAB) contains abbreviations that end the sentence when followed by a capital letter but not when followed by number. In sum:

TRAB

(abbreviations that take an object: 'Mr. Peters')

ITRAB

(abbreviations that do not take an object: 'Lloyds Ltd.')

TRNUMAB

(abbreviations that take NUM objects: 'Downing Str. 10') There is sentence boundary after TRNUMAB only for the capital+small combinations and all small-initial strings that consist of more than one letter. For all other strings (one small letter, one or several capital letters) there is no sentence boundary after TRNUMAB.

Abbreviations and the lexicon files

All the abbreviations are extracted from the lexicon files. They are extracted from the file as a part of the compilation process make. The relevant commands are stored to the Makefile in the gt/lang/src directory. The script that handles the extraction is called abbr-extract and located in the gt/script directory; it is intended to be a language-independent script.

The main source is abbr-lang-lex.txt, where the real abbreviations are listed. There also exists a number of multi-word expressions which are dispersed to different lexicon files, for example the file adv-sme-lex.txt contains multi-word adverbials like earret eará. Not to mention the propernoun lexicon, which contains (at the moment) over 500 multi-word propernouns.

In order to be able to treat the multi-word expressions as single tokens already at the preprocessing phase, the multi-word expressions are extracted from the lexicon files. The relevant lexicon file names are given to the script abbr-extract in lang/src/Makefile. There are no restrictions to how to list the multiword expressions, they can be used in lexicon files as other tokens. Only the structure of the file abbr-lang-tex.txt is restricted.

Usage of the script abbr-extract:

abbr-extract --output= --abbr_lex=--lex=

The structure of abbr-lang-lex.txt

The structure of the abbr-lang-lex.txt is free in all the other respects but the extracted part must have the following syntax:

LEXICON ITRAB
abbr  TAG ;
ab.br  TAG ;
ab  TAG ;
abbr% abbr TAG ;

And the same to the other abbreviation classes, LEXICON TRAB and LEXICON TRNUMAB. The ordering of the abbreviation classes is free.

The abbreviations may contain one or more dots but due to xfst, the dot that ends the abbreviation must be removed (operator). The pattern matching in the preprocessor is always done without the dot. It is possible to add the dot to the abbreviation again in the lex-file.

The space in the multi-word abbreviations is marked in the lexicons by the %-sign (literal operator) which is recognized by the abbr-extract-script and removed.

The abbreviations in abbr-lang-lex.txt are lower-cased unless there exists only upper cased version of the abbreviation. The abbr-extract-script generates upper case versions of all the abbreviations (they may begin the sentence).

Possible flaws

Some of the transitive abbreviations may end the sentence. Some of the abbreviations are homonyms:

min. min

Abbreviations may be followed by capital letter even when not ending the sentence

Siellä oli Pekka ym. Mäkisiä.

Siellä
oli
Pekka
ym
.
Mäkisiä
.

Siellä oli Pekka ym. Mäkinen ei tullut.

ym
.
Mäkinen

Last modified: $Date: 2006/12/18 13:45:09 $, by $Author: saara $