Citolab.QTI.ScoringEngine 1.4.0

Citolab QTI Scoring Engine

Scores QTI assessment results: it runs the response processing of your items and the outcome processing of your test, and writes the outcome variables into the assessment results it returns.

• Reads QTI 2.x and 3.0 items and tests, and QTI Results Reporting 2.x assessment results.
• Works on XDocuments, so nothing is read from or written to disk.
• Not every expression of the specification is implemented, but everything in 3.2 Simple Items is scored correctly. What is supported lists the rest.

Contents

• Installation
• Quick start
• Usage
  • The context
  • Options
  • What the engine changes
• What is supported
  • Cardinality
  • Base types
  • Response processing templates
  • Rules
  • Expressions
  • Lookup tables
  • How the operators behave
• Records and QTI_CONTEXT
• Custom operators
• Errors and logging
• Processing a package

Installation

dotnet add package Citolab.QTI.ScoringEngine

The package targets net9.0 and netstandard2.0.

Quick start

using Citolab.QTI.ScoringEngine;

var scoringEngine = new ScoringEngine();

var scoredAssessmentResults = scoringEngine.ProcessResponsesAndOutcomes(new ScoringContext
{
    AssessmentItems = assessmentItemXDocs,      // List<XDocument>
    AssessmentTest = assessmentTestXDoc,        // XDocument
    AssessmentmentResults = assessmentResultXDocs, // List<XDocument>
    Logger = logger                             // optional
});

scoredAssessmentResults holds the assessment results with the calculated outcome variables.

Usage

IScoringEngine has three methods:

public interface IScoringEngine
{
    List<XDocument> ProcessResponses(IResponseProcessingContext ctx, ResponseProcessingScoringsOptions options = null);
    List<XDocument> ProcessOutcomes(IOutcomeProcessingContext ctx);
    List<XDocument> ProcessResponsesAndOutcomes(IScoringContext ctx, ResponseProcessingScoringsOptions options = null);
}

The context

Pass one of ScoringContext, ResponseProcessingContext or OutcomeProcessingContext - or your own implementation of the matching interface.

Options

Options are passed as the argument of the method, not through the context:

scoringEngine.ProcessResponses(ctx, new ResponseProcessingScoringsOptions
{
    StripAlphanumericsFromNumericResponses = true,
    QtiContextFields = new Dictionary<string, string> { { "environmentIdentifier", "ENV_1" } }
});

What the engine changes

The engine scores copies: the XDocuments you pass in are not modified. Use the list that is returned - the AssessmentmentResults of your context is replaced by that same list.

Per item result:

• Outcomes that the response processing writes (qti-set-outcome-value, qti-lookup-outcome-value) are reset to the default of their declaration - 0 for a number without one - and recalculated.
• Outcomes that the response processing does not write are left exactly as they are, so an outcome like numAttempts or a human scored outcome survives a rescore. An outcome variable marked external-scored="human" is never overwritten.
• An item that has no itemResult is skipped; an item result that has no candidate response still gets the outcome variables of the response processing, with their default value.

Per test result:

• A testResult is added when the result does not have one yet.
• The outcomes that the outcome processing writes are reset to 0 and recalculated.

What is supported

The lists below use the QTI 3.0 element names. QTI 2.x items and tests are converted to the 3.0 spelling while they are read, so a 2.x <responseCondition> is the qti-response-condition below, and a 2.x attribute baseType is base-type. Assessment results keep their 2.x Results Reporting spelling.

Cardinality

single, multiple, ordered and record (see Records and QTI_CONTEXT).

Base types

Response processing templates

An item that refers to one of the standard templates is scored by that template, so it does not need response processing of its own:

• match_correct
• map_response
• map_response_point

Rules

Both kinds of processing:

• qti-set-outcome-value
• qti-lookup-outcome-value

Response processing only:

• qti-response-condition, qti-response-if, qti-response-else-if, qti-response-else
• qti-exit-response - ends the response processing of the attempt; the outcomes that were already set are kept
• qti-response-processing-fragment - processed in place. Fragments in a separate file are not pulled in, because qti-include is not resolved.

Outcome processing only:

• qti-outcome-condition, qti-outcome-if, qti-outcome-else-if, qti-outcome-else

Not supported: qti-include.

Expressions

Supported:

* qti-equal only supports tolerance-mode="exact".

Outcome processing only: qti-test-variables, qti-number-selected.

Not supported:

• qti-number-correct - an item is left untouched rather than scored 0: what the delivery engine calculated stays in the result.
• qti-number-incorrect, qti-number-presented, qti-number-responded
• qti-outcome-maximum, qti-outcome-minimum
• qti-random-float, qti-random-integer

Lookup tables

qti-lookup-outcome-value reads the table of the outcome declaration it writes to:

• qti-interpolation-table with qti-interpolation-table-entry - matched on an exact source-value, not on a range.
• qti-match-table with qti-match-table-entry.

How the operators behave

• The operators that take numbers (qti-product, qti-divide, qti-power, qti-math-operator, ...) are NULL when a child is NULL, is not a number, or when there is no finite result - dividing by zero, the square root of a negative number. qti-sum and qti-subtract predate this and count a child that is not a number as 0.
• qti-integer-divide rounds down: integerDivide(7, 3) = 2, and the remainder of qti-integer-modulus goes with it: integerModulus(7, 3) = 1.
• qti-round-to takes rounding-mode="decimalPlaces" or "significantFigures"; significantFigures is the default, as it is for qti-equal-rounded.
• qti-math-operator supports sin, cos, tan, secant, cosecant, cotangent, asin, acos, atan, atan2, sinh, cosh, tanh, exp, log (base 10), ln, sqrt, abs, floor, ceil, signum, toDegrees and toRadians. Angles are in radians.
• qti-stats-operator supports mean, median, popVariance, popSD, sampleVariance and sampleSD.
• qti-pattern-match matches the pattern against the whole value, the way XML Schema patterns work, and gives up on a pattern that runs for more than a second.
• qti-repeat evaluates its children at most 1000 times, so a number-repeats that is read from a variable cannot build an endless container.

Records and QTI_CONTEXT

A record is a set of named fields, each with its own base type. qti-field-value reads one field of a record:

<qti-field-value field-identifier="environmentIdentifier">
    <qti-variable identifier="QTI_CONTEXT"/>
</qti-field-value>

A record can come from:

• QTI_CONTEXT, the built-in record below.
• A response or outcome variable of the assessment result declared cardinality="record". Its values carry a field identifier and their own base type: <value fieldIdentifier="floatValue" baseType="float">1.5</value>.
• A custom operator that returns a BaseValue with Cardinality.Record and its Fields set.
• The qti-default-value of a record declaration, read with qti-default.

qti-set-outcome-value can write a record to an outcome of record cardinality: it is added to the item result as one value element per field.

qti-field-value is NULL when the record has no such field, so qti-is-null is how to check whether a field is there.

QTI_CONTEXT

QTI_CONTEXT can be read with qti-variable in response and outcome processing. The engine fills what the assessment result knows:

Everything else is only known to the delivery engine, so fields can be passed in - and the ones above overridden - through the options:

scoringEngine.ProcessResponses(ctx, new ResponseProcessingScoringsOptions
{
    QtiContextFields = new Dictionary<string, string>
    {
        { "environmentIdentifier", "ENV_1" }
    }
});

A field without a value is left out of the record instead of being set to an empty string.

Custom operators

Custom operators are usually specific to a delivery engine, so they are passed to the scoring engine. The key of the dictionary is the definition attribute of the element:

<qti-custom-operator definition="depcp:Trim">
    <qti-variable identifier="RESPONSE"/>
</qti-custom-operator>

public class Trim : ICustomOperator
{
    public BaseValue Apply(List<BaseValue> values)
    {
        var value = values.FirstOrDefault();
        if (value?.Value != null)
        {
            value.Value = value.Value.Trim();
        }
        return value;
    }
}

var ctx = new ResponseProcessingContext
{
    // ...
    CustomOperators = new Dictionary<string, ICustomOperator>
    {
        { "depcp:Trim", new Trim() }
    }
};

An operator is handed the values of its child expressions and returns the value of the expression - including, if that is what it computes, a record.

Three operators are built in, each under the prefixes depcp:, questify: and qade::

An operator you pass in overrides a built-in one with the same key.

Errors and logging

ScoringEngineException

Thrown when the engine is called with nothing to work with:

• a context that is null,
• response processing without AssessmentItems,
• outcome processing without an AssessmentTest.

Other exceptions

Items, tests and results are indexed by identifier for fast lookups, so a duplicate identifier throws:

• the same item reference twice in a test,
• more than one itemResult for the same item in an assessment result,
• more than one outcome or response declaration with the same identifier in an item.

Two assessment items with the same identifier are an exception to this: that is logged as a warning and the first one is used.

Logging

Every step is logged as information, which makes it possible to follow how a score came about. Problems that do not stop the scoring are logged as a warning or an error, and usually end in a score of 0 or in a NULL value. For example:

• a value that does not fit its base type, such as a float of 'hello',
• a lookup that finds nothing in an interpolation table or a mapping,
• an expression or a rule the engine does not know.

Processing a package

Console.ScoringEngine is a worked example that scores the assessment results of a QTI package: it reads the imsmanifest.xml of the package, hands the items, the test and the results to the engine, and writes the scored results to a processed folder next to them.

dotnet run Console.ScoringEngine c:/mypackage.zip c:/assessmentResults

The first argument is the package, the second the folder holding the assessment results. Both can also be set as AppSettings:PackageLocation and AppSettings:AssessmentResultFolder in appsettings.json.