Corpus collector's manual

This file provides an overview of the corpus conversion process. Basically, when a new document is recieved, it is classified according the language and genre and stored as such to the directory structure for the original files. All the original files are left untouched. The text and structural information contained in the document is extracted by a conversion script and transformed to an xml-file. Xml-files are stored in separate directory hierarchy where they can be easily accessed and used as a test material for grammar checkers and spellers. The metainformation associated with the document, such as the name of the author, is stored in an xsl-file and appended to the xml-file during the conversion. The metainformation is used elsewhere, in different corpus applications.

First section explains the conversion process step by step. It is followed by more detailled descriptions of the directory structure and the handling of the file-specific xsl-files. There is also a small section of the log-files and the different uses of the convert2xml.pl -script.

When you have a file that you want to add to our corpus directory, you have two alternatives. You can either use the web upload script in http://www.divvun.no/upload/upload_corpus_file.html or insert the file manually.

Web upload script

Open the page http://www.divvun.no/upload/upload_corpus_file.html.

Choose the file you want to upload, or specify the url.
Choose the license type for the text: "free", which means that the text is free to use by everyone, "standard" which denotes the license that is specified in the contracts with text provides or "other", when some other than the standard license will be used (the information has to be added manually later).
Fill in the metainformation. The uploading takes a while especially if the file is large or with lot of structure. After submitting the form, a page will open with fields for filling the metainformation. The fields "language" and "genre" are obligatory. If you nevertheless feel unsecure about the genre of the document, you may leave it open, but please inform the maintainers about that.
If any problems persist, please contact the maintainers. You will be informed on how the uploading succeeded.

Manual conversion

If you have a file that you want to add to the corpus repository by hand

Start with determining the main language: eng, fin, nno, nob, sma, sme, smj, swe. And genre: admin, bible, facta, ficti, laws, news.See section directory structure for details.
Copy the file to the directory /usr/local/share/corp/orig/$lang/$genre/. (Check that there isn't already a file with the same name.) (there should perhaps be a md5sum test for the files here as well, add it to convert2xml). For example, a North Sámi document in genre "facta", copy the file to directory /usr/local/share/corp/orig/sme/facta/. For news genre, there are subdirectories for different sources of news, and for bible texts, there are subdirectories for ot and nt.
Run the convert2xml.pl script for the file. For example, a MS Word document, with language and genre as above, is converted with command (relative path is fine as well, but do not use tilde.) Give the language as command line option.
```
    convert2xml.pl --lang=sme /usr/local/share/corp/orig/sme/facta/file_name.doc
```
The conversion takes a while especially if the file is large or with lot of structure. If you want to see all the error messages generated during conversion, use option --nolog.
Check out the resulting xml-file in the corresponding bound-directory. If there are fatal errors during the conversion, you will see an error message stating "ERROR <something>", consult the maintainers of the corpus files or see the log file (in /usr/local/share/corp/tmp) for details. The script informs you of the character conversion and the original (or intermediate) encoding.
Update the metainformation by editing the file /usr/local/share/corp/orig/$lang/$genre/file_name.xsl. The file is under version control, so check it out for writing with command
```
     co -l file_name.xsl
```
Specify the values for fields that are incorrect or missing by giving a value to the corresponding variable. Check in your changes with command:
```
     ci -m"your log message" xsl-file.xsl 
```
Run the convert2xml.pl script once more to get the changes to the xml-file.

The corpus is situated on the machine victorio.uit.no. The two main directories in the corpus are orig and bound. orig contains the original files with the original names (bar spaces replaced with _) and in the original format. The file-specific xsl-files are in under the orig-hierarchy as well, in the same subdirectory as the document itself.bound has the same structure as orig, but contains the files converted from the original format to our internal xml-format. The bound-directory contains copies of all the files in the corpus database, with restricted access only for the project team members. There is a directory free for the xml-versions of the documents that are freely available. If the access to the document is restricted, the conversion updates only the version in the bound-hierarchy. If the document is marked as publicly available, both bound- and free-directories are updated.

Inside these directories are directories for various languages. They contain parallell translations to the sami documents. Below is an outline of the directory structure.

                
/usr/local/share/corp/
                    orig/
                        eng/
                        fin/
                        nob/
                        sma/
                        sme/
                        smj/
                        swe/
                    bound/
                        eng/
                        fin/
                        nob/
                        sma/
                        sme/
                        smj/
                        swe/

Inside each of the language directories the structure outlined below is used.

                
admin/
    depts/
    guovda/
    karas/
    others/
    sd/
bible/
    ot/
    nt/
facta/
ficti/
laws/
news/
    Assu
    MinAigi
    NRK
    YLE
    other

The subdirectories under orig contain the original documentes stored to the corpus database in their original format (doc, pdf, txt, html and paratext at the moment). During the conversion, the original documents are transformed to text and stored in an xml-structure. The xml-files are stored under bound-hierarchy, whereas the original documents stay unthouched. The conversion from the original format to the xml-file can be done any time, and should be done if e.g the tools improve or the xml-structure changes.

The xml-document contains a header for the metainformation such as the title of the document, author(s), publication info and so on. Since we cannot edit the original document and do not want to edit the xml-file, the changes to the metasection are done in an xsl-script, which adds the metainformation to the xml-file during the conversion. In addition to metainformation, the xsl-file may contain processing instructions, such as the languages to which the text is categorized or some information of the text layout.

The xsl-file is created during the conversion by the convert2xml-script. When the corpus file is uploaded to the directory hierarchy via web interface, the script xsl-process.cgi creates the xsl-file. The xsl-file is stored to the same directory as the original document, with the same name as the original document, and suffix .xsl.

File specific xsl-file contains variables for main language and metainformation of the corpus document. By changing or adding values on these variables, we are able modify the the xml-file without touching the original.

The variables in the xsl-file

The variables are defined by expressions as the following:

<xsl:variable name="title" select="''"/>
<xsl:variable name="author" select="''"/>
<xsl:variable name="author-gender" select="'f'"/>
<xsl:variable name="publisher" select="''"/>
<xsl:variable name="translated-from" select="''"/>

<xsl:variable name="mainlang" select="'sme'"/>

The information is updated by adding the new value to the "select"-field, inside single quotes. Most of the fields, such as the title and the name of the author can contain free text, but the values of some fields are restricted. The variables are explained in detail in following.

General variables

filename: The original name of the file, that comes from the submitter of the document. Sometimes the filename has to be changed e.g. for the system requirements: no white spaces or special characters are allowed in the filename.
author1_fn, author1_ln, author1_born, author1_nat: Information of the first author: first name, last name, date or year of birth, nationality. The variables for author2 etc. respectively.
author1_gender: The gender is selected from the list: f, m or unknown.
translated_from: The original language of the document: select from the list: sme, smj, sma, nob, nno, fin, ger, eng, .. and so on.
publChannel: The values are either: published or unpublished
publisher, place, collection, ISBN, ISSN: The name of the publisher, place, the name of the collection etc. as free text.
yeer: Publishing year, or if the document is unpublished, the year the document was written.
translator_fn, translator_ln, translator_gender, translator_nat, translator_born: The values are similar as with the author1.
license_type: The availability of the document. Some documents are freely available, some are under license. The type of the license is either free, standard or other.
sub_name, sub_email: The name and email of the submitter.
wordcount: The wordcount for size estimates.
metadata: If the xsl-file is finished. The values are complete and uncomplete
template_version, current_version: Version information of the original XSL-template and the version number of the current xsl-file. This information is provided automatically.
note: Free text field for notes.

The languages of the document

mainlang: The main language of the document: sme, smj, sma, nob, fin, etc. The main language should be the same as the language directory where the file is in.

monolingual, multilingual

Select "1" for either of these. If both are undefined, the document is assumed to be multilingual.

mlang_sme, mlang_nob, mlang_fin, mlang_swe,..: The languages that appear in the document in addition to the main language.

Parallel texts

parallel_texts: Select "1" if the document has parallel texts.

para_sme, para_nob, para_fin, para_swe: Give the names of the parallel documents. The parallel file is assumed to be located in the corresponding subdirectory under the selected language. E.g. the Bokmål paralle file of the filecorp/orig/sme/facta/file.doc is assumed to be located in corp/orig/nob/facta/. If this is not the case, give the full path to the file. If more than one document in some language correspond the document, give the filenames separated by comma (,).

Other processing instructions in the xsl-file.

Changing the language of a paragraph

It is possible to explicitely state that some element in the xml-document is categorized to some language. The part of xsl-file that specifies the block is the following:

<!-- Add all paragraphs that should have xml:lang=X-->
<!-- Uncomment the following and add the paths, for example: -->
<!-- <xsl:template match="/root/section[2]/paragraph[5] |
                      /root/section[3]/paragraph[2] "> -->

<xsl:template match="">
    <xsl:element name="p">
    <xsl:attribute name="xml:lang">
        <xsl:value-of select="$smelang"/>
    </xsl:attribute>
    <xsl:apply-templates/>
</xsl:element>
</xsl:template>

Change the value of "template match" to the XPath-expression of the xml-element. Change the value of "select" to the desired language.

Translating characters to other characters in the text

The problematic characters in the document that cannot be reached by the character conversion can be replaced by xsl-script. The line of code is:

<xsl:value-of select="translate($text,'ðq','đá') "/>

In the example, the ð is translated to đ and q to á.

Excluding a file from conversion

The xsl-file contains line for including the processing instructions:

<xsl:include href="/usr/local/share/corp/bin/common.xsl"/>

Sometimes the converted text does not fulfill the quality requirements for some reason, e.g. the Sámi characters are missing or encoded strangely. The file can be excluded from the conversion by replacing "common.xsl" with "empty.xsl".

XSL-files and version control

xsl-files are under RCS version control, which is indicated by the suffix ,v. A revision must be checked out for reading and processing as well as for editing. The relevant commands for handling the version control are described in the following:

   co xsl-file.xsl     # Check out the file for reading or processing (the file is write protected).
   co -l xsl-file.xsl  # Check out the file for editing, creates a lock.
   ci -m"your log message" xsl-file.xsl     # Check in the file, remove the lock.

Sometimes people check out files and forget to check them in again, thus keeping the file locked. If so, ask the owner of the lock to remove it or it that is not possible, steal the lock with rcs -usee the rcs man page for details. The RCS files can be accessed by anyone belonging to the cvs group. The files can also be deleted by persons belonging to the cvs group, so beware! Check the permissions.. todo.

If you create an xsl-file manually, use the following command to add the initial revision under version control:

   ci -t-"log message" -q -i xsl-file.xsl

The error messages, warnings and other information of the conversion are stored in special logfile. The log file is named after current date and time and stored to a temporary directory, by default the directory tmp/ under corpdir. The directory can be given as a command line option as well. The log file contains the information that was printed to STDERR during the conversion. The errors that prevent some document to be converted are still printed to the screen as well. If you want all the errors to be printed to screen, use option --nolog. The log file is for example: tmp/Feb-10-8-57.log

The old log files are stored to a compact format under /usr/local/share/corp/tmp/log/. There you can look for the log file from the day the current xml-document was created. To examine the tar-file from 10th of February, first copy the tar-file to your home directory and then open it:

    cp tmp/log/2006-Feb/Feb-10.tar.gz ~/tmp/
    gunzip tmp/log/2006-Feb/Feb-10.tar.gz
    tar -xvf tmp/log/2006-Feb/Feb-10.tar.gz

There are several options available which control the conversion process, it is e.g possible to skip the xsl-processing, the character decoding, or the hyphenation. The available options:

Usage: convert2xml.pl [OPTIONS] [FILES]
The available options:
    --xsl=<file>    The xsl-file which is used in the conversion.
                    If not specified, the default values are used.
    --dir=<dir>     The directory where to search for converted files.
                    If not given, only FILE is processed.
    --tmpdir=<dir>  The directory where the log and other temporary files are stored.
    --nolog         Print error messages to screen, not to log files.
    --corpdir=<dir> The corpus directory, default is /usr/local/share/corp.
    --no-decode     Do not decode the characters.
    --multi-coding  Resolve the character coding separately for each paragraph.
    --no-hyph       Do not add <hyph/> tags.
    --no-xsl        Skip the file-specific xsl-processing.
    --all-hyph      Tag all hyphens (default is at the end of the line).
    --no-hyph       Do not tag hyphens.
    --upload        Do conversion in the upload-directory.
    --lang          The main language of the document.
    --help          Print this message and exit.

The file to be converted is given to convert2xml.pl as command line argument. The behaviour can be changed by giving the --dir, then the directory specified is searched for files to be converted. The file types that are supported at the moment are: doc, pdf, html, text and paratext. The corresponding file suffixes are are .doc, .pdf, .html, txt and .ptx. At the users' point of view, there is no difference between the file types, the technical documentation is provided at corpus_conversion_tech.html.

The script convert2xml.pl is mainly used for the basic corpus directory hierarchy, but it can be configured for other environments as well. How this is done is explained in the following:

Some documents get wrongly utf-8-encoded by the conversion tools and they are fixed using a Perl module samiChar::Decode.pm. It is installed in victorio, and you should not notice it's presence. If you are converting files in some other machine, you should install the module.. Another Perl module that is needed is XML::Twig. Instructions for installing both modules can be found in corpus_conversion_tech.html.

Let the original file be original.pdf. Use for example the command

  $ perl convert2xml.pl --corpdir=/home/mydir/samipdf --tmpdir=mytmp original.pdf

Generally each file has only one character coding which may be wrongly utf-8-encoded. The best way to decode the file is determined statistically. By using the option --multi-coding, the file is assumed to contain more than one character codings, which are decoded paragraph-wise.

The resulting file is fully converted xml-file original.pdf.xml When working with the full directory hierarchy, the original file is expected to be found in some subdirectory under /usr/local/share/corp/orig. Then the resulting xml-file is generated to the corresponding subdirectory under /usr/local/share/corp/bound. If you have the same hierarchy in the corpdir given in commandline, the process will be similar. You may use relative path names to the files and directories but do not use tilde (~).

The hyphenation points are tagged as <hyph/> . The script tags hyphens that are found at line breaks or end of the paragraphs followed by a suitable word. If all the hyphenation points should be tagged, convert2xml.pl can be called with option --all-hyph.

Corpus collector's manual

Introduction

Basic conversion process