Current status

Parts implemented:

• Basic low-level Lucene index implementation, supporting our current Java bean index document model, URI based index document handling, resource extraction, system specific fields and synchronization. See source code in package org.vortikal.index.lucene.
• Components for updating indexes and keep them in sync with Vortex resources (resource change API). These have been placed in the org.vortikal.index.observation package. No major changes in functionality, except for refactoring and structural improvements. URI- based resource filtering functionality has also been placed in this package. The API suggests using an observer/notifier pattern, where an observer implementation subscribes to a notifier to receive messages of altered resources, and then updates indexes accordingly. Filtering can be plugged in between the notifier and observer. Updating Lucene indexes is now implemented as such an observer, in the class org.vortikal.index.lucene.ResourceChangeObserverImpl. The notification of changes is still based on the Vortex resource replication database changelog table, which is polled periodically.
• Index supporting the "Mappevurdering" project. This means support classes for extracting index documents, index document definition (Java bean type) and querying. In conjunction with this, a basic security filter has been implemented above the index query layer. More about this in security section. The index might resemble what a final DMS index will look like, but also contains fields specific to the Mappevurdering project. Also, code has been created to allow index queries to be performed from within XSLT transformations and via HTTP, providing the result data as XML.
• Some work in improving the index query result API to allow filtering of results.
• The Tavle project is now running on several hosts (for testing), using our current indexing functionality.

Important parts missing:

• Formal query/search API (work has previously been done on this). The index document model (currently Java beans) needs to be discussed further. Right now, index searching is performed using Lucene-specific classes. Multi-index queries are dependent upon what document index model we eventually end up with (to resolve result set compatibility issues, etc.).
• Index reliability during resource change updates. Policy on handling of observers/indexes that for some reason fail to process resource change events (what to do to avoid inconsistency, etc.). Synchronous (sequential) or asynchronous (all-at-once) event distribution model to observers. Redo-logging at the observer level ?
• Backup solutions. Handling corruption and crash situations as gracefully as possible. In general we need better support for synchronizing and recovery of indexes, etc.
• More flexible formal index document model. Currently, a new bean type must be programmed in Java code, for each index document type. Needs discussing. Many things depend on this, so it is important to get it right.
• Completely generic/default and flexible DMS index. Depends heavily on index document model and query API, for widespread use. It will be important to determine what goes into this index, how it should be used, and how indexes should be configured, in general. Also depends on index security model, when exposed to "clients" (Vortex extension code, HTTP clients, web services, etc).
• Index security model implemented in the query layer. Depends on index document model, and how to reflect the Vortex ACL security model efficiently and properly in indexes, and applying this on queries. An inefficient repository based post-query-lookup result filter is currently being used for the Mappevurdering project. This will require some thinking.
• Cross-host index querying.
• Formal index runtime maintenance functionality. Typical operations are listing of active indexes found in a running Vortex instance, hot/cold-backup of indexes, enable/disable, lock/unlock, re index/synchronize, consistency-check, contents-listing, corruption-test, query-test, status report, size/stats and so forth. API will most likely consist of a set of interfaces in the org.vortikal.index.maintenance package, which gives access to all of this functionality. Some functionality might need to use lucene directly for efficiency, while others may work on top of the to-be-done index access API, using Vortex index document model.

---

Redaksjon: Vortex
Dokument opprettet: 30.01.2006, endret: 31.01.2006Administrer dette dokumentet