Assessment Interoperability

First draft - see new version

Draft Paper for the LIFE Project

Produced by the European IMS Network (EIN)

1. Assessment Overview

Assessment interoperability has made measured progress over the last 5 years, since initial international specifications first became available. As with most reform activities, the lowest hanging fruit approach has been taken with attention focused on providing basic interoperability at a granular level of test items based on electronic question types that have been in vogue for over ¼ of a century. At the same time, the movement towards open specification-based content interoperability has required addressing the communication of summative assessment.

With the first broad adoption of content interoperability, practitioners and policy makers have begun seeking greater detail and reporting of assessment data in order to facilitate accountability and to underpin the provision of customized learning programs for individual learners. Innovation prompted by research or the desire for product differentiation has led to advanced assessment models and techniques that are not currently supported by open specifications.

Furthermore, the evolving assessment marketplace is establishing new models for the delivery of assessments through integrated services provided by a variety of organizations rather than vertically integrated offerings from a single vendor. Without intervention, the drivers for interoperability in the present and future marketplaces are limited. Additionally, new online learning environments, such as online gaming, simulation, and collaborative spaces, require different approaches to assessment so that we assess in context. Assessment methods and assessment interoperability in these areas are underdeveloped.

A key gap in the open specifications-based elearning community is a consensus model that integrates content and assessment and supports a variety of delivery models.

NEED TO ADD HERE A FULLER OVERVIEW OF THE PRESENT STATUS OF ASSESSMENT INTEROPERABILITY. IT SHOULD BE POSSIBLE TO EXTRACT THIS FROM SUBSEQUENT SECTIONS

2. Stakeholder Perspectives

Teaching and Training Organizations
These groups are seeking greater reusability of items and tests and access to item banks from both internal and distributed collections. However, there are varying degrees of confidence in the relevance and applicability of such collections, with use most developed in areas with unified curricula and centralized control over summative procedures

There is a demand for more advanced types of assessments and a need to develop greater detail in and a shared understanding of the reporting on the competencies that will be tracked within the ePortfolios or learning portfolios required for mobile workforce development and customized learning and training.

Teaching and training organizations seek to have more options with respect to the vendors and tools that they choose from to provide a learning experience for their learners.
Content and Test Developers
Content developers are seeking a common framework for the structuring and delivery of content. They see assessment interoperability as a part of that common approach. Recently, adoption of item level interoperability has been motivated by the promise of internal operational efficiencies, e.g. by sharing assessments across book and online course production processes.

The large test publishing firms have begun to respond to internal needs for greater efficiency and to acknowledge customer demands for open specification support. In a recent Association of Test Publishers Annual Conference, a major conference program element "Interviews with the Titans of Assessment", a senior Vice President of one of the world's largest publishers listed 3 major trends in 2005, one of which was the use of an open specification (IMS QTI) for assessment interoperability.

Test publishers have historically competed by providing vertically integrated services and technology and have offered advanced assessment types, e.g. spreadsheet assessment tasks, advanced drag and drop, variable replacement. Based on the move of major publishers towards more open specifications, it can be assumed that customers have happily or grudgingly accepted lesser functional assessments that are more broadly interoperable versus higher function, lower interoperable assessments. In part, this may reflect a trend to a broader and larger market having more limited assessment strategy aspirations.
Test Authoring and Delivery Tool Vendors
Test authoring and delivery tool vendors, like most tool vendors, are caught between the common need to differentiate their product and an increasing call for interoperability from consumers. The open specifications and standards do not support; the full range of advanced assessment types, or advance assessment authoring, or advanced delivery, e.g. local, hosted, and hybrid assessment delivery models where an outsourced company might provide essay grading services. The advantage from a requirements perspective is that there is substantial practice instead of wishful thinking that can inform requirements collection.

3. The Expert's View

A number of assessment experts were brought together in a workshop to consider a range of issues and drivers/obstacles surrounding assessment interoperability. The following lists summarise these views but also reflect the discussion and, where appropriate, the consensus reached at the workshop.

MORE WORK NEEDED TO SYNTHESISE THESE LISTS. SOME ISSUES SHOULD GO IN THE PRESENT STATUS SECTION.

Issues

• It is important to have stable versions of standards and specifications such as IMS QTI 2.1, Content Packaging v1.2, etc.
• There is a need for harmonisation of assessment standards and specifications with related specifications; for example, standard results formats for sharing with eportfolios, semantic interoperability with Learning Design, etc.
• The development of international standards is on-going with varying interpretations of existing standards. Current specifications are loose which permits compliance without true interoperability.
• The development of specifications is slower than that of commercial products; for example, the range of question types defined by QTI lags behind the range of question types supported in most commercial systems.
• Although, at the outset, vendors were a key participant in the drive for standards development, this has declined of late. This could result in a divergence between technical specifications and actual products.
• The absence of a formal accreditation procedure permits various, and frequently incompatible, products to claim standards compliance.
• Data exchanges within a system which are using two different specifications (for example QTI and learner information profiling) may re-interpret data in different manners, with differing assumptions.
• There is an ambiguous level of acceptance by the developer community, particularly developers of major tools (Questionmark, WebCT, Moodle, etc). To date, interoperability has not been as high a priority as public statements suggest.
• Continuing acceptance and promotion by organisations that support and fund development work (i.e. JISC, SURF, HEFCE) is essential. As yet, ownership is insufficiently embedded for interoperability, even at the present level, to be self sustaining. Support is needed, from either the standards bodies themselves or related organisations such as CETIS. Necessary activities for users include developer fora, training, and conferences. Also, where appropriate, the development, dissemination and maintenance of application profiles and mappings/crosswalks will be necessary.
• The available standards will need further development to meet a growing range of needs, for example in dealing with advanced question types or the reporting of competencies for which the evidence is portfolio based.
• There si a continuing need for the sharing of eassessment knowledge and expertise and (innovative) practice. This might be achieved through a European eAssessment network, as well as informal mentoring and support processes.
• Interoperability would be promoted by the development and support of free open source tools and content.
• The development of high quality items is resource intensive. Ensuring that items are standards compliant protects this investment, Any threat to the uptake of these standards threatens that investment.
• Much of the present discussion of interoperability is framed in terms of technical specifications relating to a limited part of the existing repertoire of assessment practice, i.e. automated computer-mediated testing. However, assessment that is authentic will continue to be based around a much broader range of diverse tasks. These include; the production of specified documents (e.g. essays, reports etc.), engagement in real or simulated professional roles (e.g. laboratories, fieldwork, training roles in organisation, virtual practice environments, etc.), public performance (e.g. in giving presentations, playing music or sport, etc.), self reflection (e.g. in portfolio construction), and reporting on prior experience. This isn't an exhaustive list. Given the imperative for personalised learning and authentic assessment, it is likely that assessment will reflect increasingly local and personal context. Interoperability can be thought of as an expression of the need for such pedagogical imperatives to be reconciled with the needs for broad metrological validity and accountability.
• Contemporary accounts of assessment stress the intimate relationship between its summative and formative functions. There is a need for an integrated design of any learning experience with a recognition of the 'backstream' effects of any summative task on the individual's pattern of study, interaction with support functions and learning. Mechanisms for interoperability should reflect this integrated reality.
• It is interesting to consider where a desire for interoperability is visible in the present or future market place. Assessment is profoundly culturally related with nations, institutions and individuals insistent on retaining their control over summative assessment content, process and accreditation. The level at which this is exerted tends to vary with level and is most acute at the tertiary level, where the concept of the self directing university and academic is dominant, at least in areas where there is not a strong external driver (e.g. a professional accreditation agency such as a professional body or a powerful vendor, e.g. Cisco). It follows that there is major reluctance to recognise or engage with the exchange of assessment content, process or even information.
• A related issue is the sharing of understanding or documentation of levels of achievement. In spite of long standing attempts to codify local norms and generate broadly recognised benchmarks, genuine progress has been limited. The extent of the failure to arrive at operationally effective understandings is often concealed as it is potentially embarrassing at political, institutional, directorial and individual levels. How can interoperability at summative level operate effectively without a shared understanding of achievement and the description of achievement?
• Similar arguments surround the pedagogy within which the assessment lies. There are major differences in expectation of and judgements about teaching and learning styles.
• It is possible to have greater optimism about interoperability in assessment for learning, not least because there is a long history of interoperable assessment use in end of chapter questions and other prosaic resources. Formative assessment has fewer issues of ownership. The main issues are perceived relevance and benefit, not least in increasing the efficiency of the teaching process.
• Within the narrower CBA/CAA domain, there are several possible future models of assessment practice with different issues for interoperability. It is unclear whether a model in which assessment is distributed and implemented on a variety of platforms is sustainable. Instead we might expect assessment to be a service or solution delivered via networks from a central source, just like a map or an auction. This model, in limited but widely adopted form, is common in the United States where it is seen as an adjunct to publishing. For example the uptake of Wiley Plus with its well developed formative assessment capability is growing rapidly.The challenge may be to offer a richer service, which, for example, includes the wider functions of a community of learning, while retaining cultural diversity and flexibility. Moodle is an example of an open source initiative that embodies such an aspiration.
• The roles and levels of competence of those involved in generating and delivering assessment will affect interoperability. As standards, specifications and authoring aspirations increase in complexity, in order to reflect more complex assessment strategies, it becomes increasingly difficult to envisage a process in which either the 'teacher' has the technical competence or a 'learning engineer' has the pedagogical skills to deliver an appropriate solution. The community will be driven towards team approaches.
• There is a need to expand the concept of validity to incorporate systematic influences on educational systems, including all previously considered aspects of validity (and reliability).
• There are many challenges facing performance assessment that should be addressed, such as: limits on the generalization of our approaches, value in indicating broader constructs, development of new psychometrically based cognitive diagnostic models, the addressing of new equity issues, investigation of the problem of large error in ability estimates due to fewer items, and overcoming problems with current DIF analysis models to include constructed response items.
• There is a need to develop improved multidimensional psychometric models to accommodate the needs of performance assessment tasks administered in computer-based environments.
• Work is required to establish procedures to maintain a valid bank of items for test construction, with the appropriate statistics, and procedures for item production and item exposure control, and including other security considerations.
• Research is needed to establish automated scoring of the complex item types that are or will be typical of performance assessments.
• The development of multidimensional latent trait models that capture performance modifiability will provide a means of increasing the impact of assessment results on the topic of assessment in support of the learning process.
• There is a need to develop assessment procedures, including selective and diagnostic testing, that optimize skill acquisition in complex task performances.

Drivers

The following drivers and obstacles were seen as being key to the promotion of assessment interoperability.

• Useable tools, particularly free and open source tools which are adequately supported by their developers.
• Training and support for teachers.
• An effective and robust infrastructure.
• Useable and reusable content, perhaps within a Creative Commons or similar license arrangement..
• Effective metadata that allows users to identify items or tests of value.
• Government or international policy could force major change in the market., e.g. the requirement for UK school-level exam boards to introduce on-screen assessment
• Government might also provide incentives, e.g. funding for development tied to the use of interoperability standards, and the sharing of open source outputs. These may be motivated by a wish to achieve demonstrable savings in resource or time.
• Standards and specifications that are robust, stable, and extensive enough to encompass practice.
• The need for vendors and customers to address accessibility needs enshrined in supporting legislation.
• The economics of coping with demands for personalised learning with timetable flexibility are increasing. Imported solutions offer one means of coping with such demands.
• At a relatively trivial level, increases in communication bandwidth allow a greater complexity of assessment task and more effective links into the 'community of learning'. This makes it more realistic to plan for a geographically dispersed student body within world-wide mega-universities.
• Plagiarism is a major concern at all levels. The quality accreditation process may increasingly require conformity with agreed (inter)national standards of both assessment system design and plagiarism checking.
• The fear of vendor lock-in encourages awarding bodies to pursue interoperability.

SOME ITEMS DELETED HERE AS THEY AREN'T DRIVERS
Obstacles
• Competing commercial products prefer to add features rather than focus on compatibility with interoperability standards.
• The standards domain is opaque. The specifications are complex and not user friendly.
• A lack of combined expertise in technical skills and pedagogical awareness leads to difficulty in recruiting appropriate personnel to deliver interoperable systems or content.
• Fragmented ownership of the validation and accreditation processes.
• Powerful and 'monopolistic' vendors whose interest is in training that is narrowly based around their products/services and who have limited incentive to create interoperable systems
• The lack of a broadly shared epistemology of assessment. Without such a 'map' it is difficult to generate the momentum for the large reforms that would create a climate more conducive to interoperability.
• Limited levels of expertise in authoring computer marked questions that test higher level outcomes. Limited range of question types that do not include, except at an experimental level, more open ended student responses, e.g. essays.

4. Standards

It is helpful to look at key parts of the lifecycle of an assessment to see where "standards" fit in and where interoperability might be an issue.
Assessment Creation and Re-use
To exchange, edit, or disaggregate assessment items in a variety of different tools, the items need to be described in a common format. The IMS Question and Test Interoperability Specification is used to describe the content of common assessment types, such as multiple choice, true/false, etc. A number of vendor's tools implement the import and/or export of assessment in "QTI format." The IMS QTI v2 specification addresses many issues raised by implementers over the last 3-4 years of use of the QTI specification. This includes but is not limited to more complex question types and support for mathematical representations. The current V2 specification addresses Item interoperability. The next addition to the V2 specification, currently in development, will address updates to QTI's Test interoperability. IMS QTI has been designed for multiple language settings. Whether its language support is sufficiently adequate or not requires more investigation.

Assessment Discovery

If you maintain a searchable collection of assessments for internal or outside use, then the assessments need to be tagged just as a library assigns card catalog information to books to enable their discovery. The IEEE Learning Object Metadata Standard, also known as "LOM," defines the names and meanings of the tags used to describe electronic information used in learning and training, including assessments. The Dublin Core metadata specification, developed by the Library community, is commonly used as well in elearning.

If you need to describe the competency definition or educational objective, then the IMS Reusable Definition of Competency or Educational Objective (RDCEO) specification can be used.
Assessment Ordering
Once an assessment is discovered, the HR-XML Assessment specification supports ordering assessments in a standard way that a tool could facilitate thereby automating parts of the order fulfillment process.
Assessment Distribution
Distributing an assessment to a teaching or training organization or business partner is much like sending a physical product. You need a way to package up all of the related files, label the assessment, and send it in a way that the receiver knows how to inspect and use the contents. The IMS Content Packaging specification is designed to support the electronic distribution of learning and training materials. Most elearning tools support this specification, and it is the basis for work from the US Department of Defense's Advanced Distributed Learning (ADL) effort most commonly known as SCORM (Sharable Courseware Object Reference Model).

Although SCORM does not support IMS QTI, meaning it is not possible to "look inside" a SCORM-compatible piece of content and see individual assessment items; IMS has described the appropriate way to include QTI-based assessments in an IMS Package in the QTI V2 specification.
Assessment Reporting and Status
Standards for reporting assessment status and outcomes have generated the greatest level of interest. They enable the management of training without requiring assessment developers to expose their intellectual property that makes up the assessment. There are a number of standard ways to report status and/or results. Only simple results and status are described. This includes IMS QTI, Aviation Industry CBT Committee AICC standard, SCORM, HR-XML, and the Schools Interoperability Framework SIF (from the K-12 community). In the final case, the reporting supports the US Government No Child Left Behind initiative.

Besides defining "what" to exchange, the aviation industry has also defined "how" to move assessment results and status between systems. SCORM uses a modified version of the AICC approach and it is a cornerstone to SCORM's ability to provide an interoperable content model that separates content from Learning Management Systems. The SCORM version of the AICC work has been standardized within the IEEE.

For content developers using simulation-based content, the IMS Shareable State Persistence (SSP) specification was created to work in conjunction with the AICC (IEEE CMI) specification to enable more detailed reporting of simulation data that might include assessment information or data that could be used for assessment purposes. The specification does not provide any semantics about the detailed data and therefore the utility of the specification is limited to single companies or trading partners that share the internal structure of their information that is exchanged according to the SSP approach.

5. Trends and Issues

Granular Learning Objects
There is an increasing level of interest in learning object repositories, either local or distributed. Today's bulky (low granularity) content objects almost assume an integrated content and assessment model with summative assessment reporting to an LMS/VLE. Should eLearning content repositories become wide spread, then more detailed assessment of learning may lead to separation of assessment from content with the reduced need for complicated pedagogical sequencing and thereby assume a more just in time and targeted assessment to learning or remediation approach.
Gaming, Simulation, and Collaboration
It is a common refrain that it is a good thing that the basic training model of 20 years ago has, to a sufficient degree, been captured in open specifications. At the same time, most would say that it is critical to create learning opportunities that deliver additional learning outcomes. Current open content and assessment models do not support the approaches that are being adopted to cope with these new aspirations. For example, gaming and simulation, whether single or multi-player, are increasingly looked at as trend setters and as key to the next generation of learning strategies. With a new generation of learners completely at ease with SMS and online gaming, learner expectations and their ability to use these types of environment for educational purposes can be expected. Assessment approaches must keep up to accommodate testing within a context sufficiently similar to the learning context.
Personalization
With the advent of ePortfolios, the much awaited goal of personalized learning becomes feasible in a much more interoperable manner. Likewise, the corporate organizations who need to be efficient in re/training workers in a fast moving economy demand greater understanding of a worker's competencies. Again, ePortfolios provide an important means of collecting information to make these business outcomes a reality. What is missing is agreement on ontologies and vocabularies. Assessment approaches need to be able to sufficiently integrate with the demands of ePortfolio applications in these personalized learning and training contexts.

New assessment tasks
The overwhelming majority of current electronic assessment tasks involve closed responses to circumscribed questions, MCQs. Such questions have limited authenticity and relevance outside the broad sciences. Is there a market for tasks involving more open responses. The technology of such questions is moving forward steadily but this is not matched by standard and specification development, e.g. in QTI?

6. Is there a market that requires interoperability? If so, where?

Eassessment interoperability will only be implemented comprehensively if there is a market and there is no strong evidence for such a market at present. Market growth will depend on the perception of the value of interoperability which in turn will depend on the linkage to the wider learning process and therefore on the wider issue of interoperability in learning objects. A demonstration of an implemented interoperable assessment system with tangible benefit would be of immense value in creating market momentum. Where might such progress be made. We can analyse drivers and obstacles in deploying summative and formative assessment across three education/training sectors as follows.

School Sector - Formative Assessment
Drivers.
• Lack of practitioner time and the consequent need to share resources.
• The school sector is a large volume activity with some consistency of relevant task. This enhances market potential.
• There is a limited local assessment resource base and imited authorship skills. This tends to focus authorship on a core group.
• Formative assessment resources can allows teacher to choose learning path.
• The cost of generating high quality items is high and therefore there is incentive to acquire these from centralised sources.

Obstacles.
• High cost of quality assured questions and limited purchasing power at local level.
• National teaching practice varies and therefore international usage is inhibited.
• The tendency to 'teach to the test' tends to drive formative assessment into resembling summative assessment with consequent issues of non-standardisation.

School Sector - Summative Assessment

Drivers
• There is a political imperative for shared understanding of achievement.

Obstacles
• The metrological calibration depends on the population and therefore there are fundamental limitations on interoperability.

University Sector

Several of the drivers mentioned above apply to this sector. In addition, there are the following sector specific factors.

An obstacle is created by the tendency for universities to favour bespoke locally generated solutions, even though this limits the feasibility of using complex approaches and may require a large investment of academic time. The strong local ownership of the accreditation process militates against interoperable summative assessment. There is a lack of a credible business model supporting interoperable university level summative assessment systems

Professional Training

Specific issues here include the large well defined markets the limited local authorship base, and the strong emphasis on summative processes. The presence of influential and often monopolistic stakeholders, who often link summative assessment outcomes with freedom to practice, erodes interest in interoperability.

The above analyses, though incomplete, suggest that the most promising area in which interoperable assessment systems might demonstrate value is formative assessment for the school sector. We will not elaborate here on the definitions of formative and summative but, for the present discussion, will consider formative assessment to be that which has no substantive role in the summative evaluation of performance and therefore is not affected by issues of confidentiality etc. Of course, summative processes have strong formative influence but this is not relevant to the pragmatic market sector definition we are adopting. In general there are greater opportunities for interoperable systems in assessment as part of a learning process rather than in summative processes. This is due to the added complexity of summative assessment which needs a more careful stakeholder analysis in order to establish more precise interoperability needs.

Further analysis of the school sector and formative assessment suggest a possible model for an item bank based system that has the potential to win sector support.

Primary authorship of an item might be separated from 'item dressing' which locates the item within local learning pathways. For example, an initial author, guided by a specification might produce a 'bare item' that includes all high cost production elements and captures the pedagogical aspirationin devising complex assessment tasks that address higer order outcomes. To be useful, the item should carry metadata linking it to local curricula and to summative tests. This may be easier in some subjects, e.g. computing, maths, science, than others involving more subjective discourse. Subsequent authors could takes the 'bare item' and refine it in the light of the local context, e.g. by translating the text, linking to the local curriculum, providing appropriate feedback etc. This process of refinement should be simple and would be facilitated by authoring tools. This sequential authorship model would pose new issues in interoperability.

There are several options within this model. For example, there would be a significant diifference between a small volume high production value bank and a massive item bank with inbuilt item redundancy. Qualitative feedback would add value to the collection.

Central coordination would be required alongside appropriate and authoritative quality assurance mechanisms. One authorship mechanism would involve the pooling of items from a network of authors with benefit in generating ownership. Public funding could generate content but this might be held to undercut the market. A demonstrator project would be less contentious in that it would have a market development function.

RECOMMENDATIONS

The following recommendations were the consensus view of a group of eassessment experts who met in Madrid under the auspices of EIN on May 22, 2006.

• Develop demonstrator project(s) that shows that a market can be created (see above).

• Create a forum for further development of the above projects with more detailed specification. This could be facilitated by EIN.

• Assuming that a market exists, develop authorship tools that assist localisation of QTI implemented items.

• Build capacity in item authorship and interest in item usage which may help to develop market.

• Build awareness of importance of compliance (via IMS and localised interoperability testing mechanisms).

• Facilitate further discussions on how educational (assessment) service delivery could be developed and sustained with distributed input.

• Restrict further developments of QTI specification until there is a demonstrable need. Drivers for further QTI development include:

o aggregation of qualitative and quantitative feedback on assessment items so as to enrich future usage.
o upgrade QTI validation tool(s).
o constructing a QTI Lite Version 2 which facilitates interoperability at lower cost.