PhoenixmlDb.Xslt 3.0.0

• net10.0

  • PhoenixmlDb.Core (>= 2.6.0)
  • PhoenixmlDb.XQuery (>= 3.0.0)

• net8.0

  • PhoenixmlDb.Core (>= 2.6.0)
  • PhoenixmlDb.XQuery (>= 3.0.0)

Requires PhoenixmlDb.XQuery 3.0.0 and PhoenixmlDb.Core 2.6.0. Tail calls, a large drop in W3C
failures, package checks, and typed results. W3C XSLT: 132 failing of 11,539, measured on a host
in UTC (2.9.0: 216). One of the 132 is a known defect in the test (accumulator-073). date-014
assumes UTC, and it fails on a host in another timezone.

### Security: CPU use of XPath compilation with untrusted expressions (GHSA-6342-6rv2-xj9j)

Only hosts that compile stylesheets or `xsl:evaluate` expressions from untrusted parties are
affected. With PhoenixmlDb.XQuery 3.0.0, a chain of `=>` arrows compiles in linear time, and a
long chain of predicates compiles much faster. See
[GHSA-6342-6rv2-xj9j](https://github.com/phoenixmldb/phoenixmldb-xslt/security/advisories/GHSA-6342-6rv2-xj9j)
for the details and workarounds.

### Breaking — read this before upgrading

- **The namespace sets of three AST types are read-only.** `XsltElement.InScopeNamespaces`,
 `XsltAttribute.InScopeNamespaces` and `XsltLiteralResultElement.NamespaceDeclarations` are
 `IReadOnlyDictionary<string, string>`, not `Dictionary<string, string>`. Equal sets in a module are
 one shared object (#403), so a write to one would change every instruction that shares it. Code
 that reads them, or sets them with a `Dictionary`, compiles as before; code that changes them, or
 takes them as a `Dictionary`, does not, and the change is binary-breaking.
- **`as="element(N, T)"` checks the type annotation** (#438): a stylesheet that declares a wrong
 type in this form can fail where it did not. See "Types and validation".
- **A `document-node(element(N, T))` pattern tests the type of the document element**
 (xquery#276, #412): a rule for a schema type does not match a source that is not validated. See
 "Types and validation".
- **PhoenixmlDb.XQuery 3.0.0 has its own behaviour changes**, for example XPDY0130 for an expression
 nested too deeply for the stack, and FODC0005 for a colon in the first segment of a relative
 `fn:doc` reference. See the PhoenixmlDb.XQuery 3.0.0 notes.

### Packages

- **`xsl:override`: an overriding function or template belongs to the using package** (#441), so
 its `key()`, `format-number()` and output names resolve there.
- **A global variable or parameter of a library package has no context item** (#444): a path from
 it is XPDY0002.
- **`default-mode` is a use of the mode it names** (#440): with `declared-modes="yes"`, an undeclared
 mode is XTSE3085.
- **A used package's template is private unless an `xsl:accept` rule raises it** (#454): such an
 initial template is XTDE0040.
- XTSE3087 compares the declared types of `xsl:global-context-item`, and whitespace in `as` is not
 significant; XTSE3051 covers a function that is accepted and overridden (#451).
- XTSE3022 for `xsl:expose component="*"` with an explicit name; XTSE3060 for overriding a private
 or final variable (#452).

### Types and validation

- **`as="element(N, T)"` checks the type annotation** (#438) for local variables, global `select`
 variables and parameters, and excludes a nilled element for `element(N, T)`. A template with `as`
 keeps the annotation on its result. **Stylesheets that declare a wrong type in this form can fail
 where they did not.** `attribute(N, T)` keeps the untyped test (known issue #439, open).
- XTTE0950 for namespace-sensitive content under `validation="preserve"` (#437).
- **A `document-node(element(N, T))` or `document-node(element(*, T))` pattern tests the type of
 the document element** (xquery#276, #412). 2.9.0 ignores the type, so such a rule matches every
 document, also one that is not validated. In 3.0.0 a document that is not validated does not match
 a rule for a schema type, and the transformation can give other output. Validate the source
 (`XsltTransformer.SourceValidation`, or the CLI option `--validation strict`) to get the typed
 match. W3C match-142, 143, 159, 160, 161, 163, 201 and 202, with a validated source.
- **`xsl:copy-of copy-accumulators="yes"` with validation keeps the source's values** (#443).

### Static expressions

- **Static expressions keep XPath number types** (#450): literals keep their type, and arithmetic
 uses the XPath rules, so `1 div 0` in a shadow attribute is FOAR0001. An arithmetic or cast error
 in a static expression is a static error with that code. A static variable or parameter with
 `select` and content is XTSE0620.

### Streaming

- **`deep-equal` and `index-of` operands that read the input are buffered** (#447).
- **Attributes keep their namespace in nodes built from the streaming pass** (#456): a streamed copy
 of `xml:lang` stays `xml:lang`.

### Other behaviour

- **A source too deep for the stack is a catchable error** (#458).
- **A one-slash `file:` schema location is absolute** (#396).
- **Accumulators: a failure inside a node is the after-value of its ancestors** (#442).
- XPTY0004 for an empty `xsl:map-entry` key (#446).
- XTSE0020 for an invalid literal `schema-aware` value on `xsl:evaluate`, and XTDE0030 for a computed
 one (#453).
- FOTY0015 for `fn:deep-equal` on function items, including inside arrays and maps; FOXT0002 for an
 unreadable `fn:transform` stylesheet-location; XTSE0220 for an inline `xs:schema` that does not
 compile (#455).
- MessagePack 2.5.307 in the language server, for GHSA-qhrr-8q5h-9q3h (#408).

### Also in this release

- **Tail calls of stylesheet functions** (xquery#296). A call to a stylesheet function in tail
 position of a stylesheet function's body (the select of the xsl:sequence the body evaluates
 last, also in the chosen xsl:choose / xsl:if branch, and there the then/else of an if) runs as
 the caller's next iteration instead of nested, when the callee has the caller's `as` type.
 Tail recursion, direct or mutual, is no longer limited by the recursion depth (XTDE0000 at
 1,200): a million iterations run in about 1.4 s. Not tail calls, and still limited: a call
 whose result is used, one inside xsl:try, a call of or from a memoised (`cache="yes"`,
 `new-each-time="no"`) or streamable function. A string walked with `substring($s, 2)` is passed
 as a slice where the parameter is untyped or `xs:string`: a 100,000-character walk takes about
 0.6 s. The trace listener reports each iteration as a call, as before. An infinite tail
 recursion now runs until the transformation is cancelled, as a loop does.

- **XPath evaluations reuse their query context** (xquery#294): each used to make a new one, with
 its stacks and scope; a small pool now keeps them for the next evaluation, reset, and never one
 a function item keeps. `iso_dsdl_include` over NY.EMS.v350: 4.2 GB -> 2.3 GB, time 0.93x;
 W3C evaluate-010: 685 -> 370 MB. Output unchanged.

- **A template parameter declared `as="xs:string"` also holds a slice** (xquery#293): the walk
 of xquery#287 is linear for it too (a 100,000-character walk: 3.9 s and 10 GB -> 0.16 s and 438
 MB). A type derived from xs:string still casts the value.

- **The element test inside `document-node()` in a step is applied** (xquery#292). In a step such
 as `self::document-node(element(N))` or `child::document-node(element(*, T))`, the element test
 is matched against the document element, with the prefixes of the stylesheet. In 2.9.0 it was
 ignored, and the step matched every document node. An undeclared prefix in that inner test is
 XPST0081.

- **A template walking a string one character at a time is linear, not quadratic** (xquery#287).
 An `xsl:call-template` parameter bound to `substring(...)`, without an `as` type, holds a slice of
 the original string instead of a copy of its rest; `substring`, `string-length` and `starts-with`
 read the slice, and any other use of the parameter sees an ordinary string. The ISO Schematron
 skeleton's `sch-check:xpath-wf` walks every XPath this way: `iso_dsdl_include` over NY.EMS.v350
 allocates 4.2 GB instead of 21.4 GB and runs in 1.9 s instead of 8.0 s; output byte-identical.

- **New: `XsltTransformer.DocumentValidation`** (`XsltTransformOptions.DocumentValidation`).
 With `Strict` or `Lax`, the documents a transformation reads with `fn:doc`, `document()` and
 `fn:collection` are validated against the transformer's schemas as they are read, as Saxon's
 `-val` does: their nodes have the schema's type annotations, and a document that is not valid
 fails the transformation (XTTE1510, XTTE1515). Null, the default, reads them untyped as before;
 `SourceValidation` still governs the principal source only. W3C validation-2001, validation-2002.

- **`xsl:evaluate` parses each target expression once per transformation** (#405): an instruction
 that evaluates the same text again reuses the expression it parsed and checked, where its
 namespace bindings are its own (no `namespace-context`). W3C evaluate-010, which evaluates two
 strings 100,000 times, takes 1.2 s, not 2.7 s; the output is the same.

- **In `xsl:override`, an `xsl:param` may override an `xsl:variable` of the used package, and an
 `xsl:variable` an `xsl:param`** (XSLT 3.0 §3.5.3.4): the overriding parameter takes the value
 the caller supplies. It was looked for among the parameters only and dropped. An overriding
 variable or parameter must declare the type of the one it overrides: XTSE3070. W3C
 override-v-007, override-v-008, override-v-009.

- **A stylesheet function's parameter applies the function conversion rules to `xs:string` and the
 types derived from it**: a node is atomized, and an untyped value cast to the declared type, so
 `as="xs:NMTOKEN"` holds an xs:NMTOKEN and `as="xs:string"` an xs:string, not the node. A value of
 xs:string itself for a derived type is XPTY0004, as the rules (and Saxon) say.

- **`xsl:merge-source` with `validation` or `type` validates each document `for-each-source` reads**
 (XSLT 3.0 §15.3), and the merged items have the schema's type annotations. Both attributes were
 refused as "schema validation not supported". Without `for-each-source` they are XTSE0020. A
 validated source is read into a tree, not streamed. W3C merge-049, merge-051, merge-054.

- **A variable with no `as` whose content is a validated document or element keeps the type
 annotations** (XSLT 3.0 §9.4: the content of the implicit document node is copied with its
 annotations unchanged). Its content was written as text and read again, untyped, so
 `<xsl:variable name="d"><xsl:document validation="strict">…</xsl:document></xsl:variable>` gave
 untyped nodes. W3C si-copy-101/102/105, si-copy-of-102/105, si-document-101/102/105, streamable-009.

- **A literal result element built with `validation="strip"`, the default, strips the type
 annotations of everything in it** (XSLT 3.0 §25.4.1), including a child given a type with
 `xsl:type` and validated before it. The child kept its annotation, so `element(*, T)` matched it
 inside an untyped parent. W3C match-173.

- **An element or attribute validated against a union type has the union as its annotation** (with
 PhoenixmlDb.Core's change), so `element(*, xs:decimal)` no longer matches an element declared as
 a union of xs:integer and another type: the rule for the union is chosen. W3C match-195,
 match-197, match-211.

- **A variable or parameter whose `select` gives an untyped node or a string, with `as` a type derived
 from xs:string (`xs:NMTOKEN`, `xs:ID`, `xs:token`, `xs:Name`, ...), holds a value of that type.** The
 value matched the xs:string the type is reduced to and was kept as a plain string, so
 `$v instance of xs:NMTOKEN` was false. W3C as-1902.

- **A template with `as="schema-element(N)"` or `as="schema-attribute(N)"` returns the element or
 attribute.** The type was taken for an atomic one, so the node was atomized and its value written
 as text: an attribute a template returned did not reach its parent element. W3C as-1812, as-2905,
 as-3504, as-3601, as-3701.

- **A sequence constructor whose type is a sequence of atomic values gives one value for each text
 node it makes** (XSLT 3.0 §9.4): a variable, a parameter or a template with `as="xs:string*"`,
 `as="xs:anyAtomicType+"` and the like. Its text was written into one buffer, so two `xsl:value-of`
 gave one value of the two strings joined, and a template's values were written with no space
 between them. W3C as-1801, as-1803, as-2601.

- **`schema-element(E)` in a pattern matches the elements of E's substitution group** (XPath 3.1
 §2.5.5.3). The pattern was matched by the name E, so a member of the group with another name
 never reached the test of its declaration. W3C match-179, match-199, match-200.
- **The default priorities of kind tests are those of XSLT 3.0 §6.5**: 0.25 for `schema-element(E)`,
 `schema-attribute(A)`, `element(E, T)` and `attribute(A, T)`; 0 for `element(*, T)` and
 `attribute(*, T)`; and for `document-node(...)` around an element test, that test's. The first
 four were 0, `element(*, T)` and `attribute(*, T)` were −0.25, and `document-node(element(E))` was
 −0.5. A rule with no `priority` attribute may now be chosen over another rule where it was not.
- **A pattern step with the parent axis is XTSE0340** (XSLT 3.0 §5.5.2): `match="/.."` and
 `match="parent::a"`. Such a rule was compiled and matched nothing. `..` in a predicate is allowed.
 W3C match-213.

- **The indentation pass grows its buffer in chunks** (#401). Sized for the whole result, it was one
 large-object-heap array, which only a full collection frees: the committed heap of the
 Schematron compile of GA.EMS.v350 rose by 12 MB on Linux, and the peak private commit by 1.9 MB
 on Windows, against 2.9.0. It is back to the 2.9.0 level.
- **`on-no-match="deep-copy"` keeps type annotations**: its built-in rule is
 `xsl:copy-of select="." validation="preserve"` (XSLT 3.0 §6.7.2). An element or attribute it
 copied into a variable was untyped. W3C match-262, match-263.
- **In a mode with `typed="strict"` or `typed="lax"`, a declared element name in a pattern means
 `schema-element(E)`** (XSLT 3.0 §6.6.1): only an element validated against that declaration
 matches the first step of the pattern. It was matched by name. W3C match-220 and match-221, with
 the XQuery change for elements validated against a type.
- **The in-scope namespaces of each stylesheet element are worked out once** (#401). Each instruction
 and XPath expression walked its ancestors' attributes for them. For a generated Schematron
 validator of 2.3 MB, a compile now allocates 419 MB, not 465 MB (-10%).
- **`suppress-indentation` with the html method matches names as HTML does** (#400, which stays open). An element in no
 namespace or the XHTML namespace is matched by local name without case; any other element by its
 expanded QName, its prefix resolved through the namespace declarations of the output. The names
 were matched by local name only and with case, so `suppress-indentation="div"` also suppressed
 `x:div` in another namespace, and `"DIV"` did not suppress `<div>`.
- **`indent="yes"` for the xml and xhtml methods is one streaming pass** (#400, part 1; the issue stays open). The result
 was read into an `XDocument` and written again, then the text was passed over once more for
 `suppress-indentation`. It now goes once through an `XmlReader` and `XmlWriter`, with the same
 bytes; the pass allocates 56% less (12.6 MB against 28.7 MB for a 1.8 MB result).
 `suppress-indentation` now names elements by expanded QName (Serialization 3.1 §7.1): it matched
 local names only, so `suppress-indentation="p"` also suppressed `x:p` in another namespace. An
 element keeps the prefix it has in the result tree; the `XDocument` pass could change it where
 two prefixes were bound to one namespace.
- **The `xslt` CLI can validate the source: `--validation strict|lax`** (also `--val:strict`,
 `--val:lax`), which sets `XsltTransformer.SourceValidation` (#398). The source's nodes then have
 the type annotations of the schemas the stylesheet imports; an invalid source is XTTE1510 (strict)
 or XTTE1515 (lax). Without the option the source stays untyped, as before.
- **A validated copy of a document keeps its type annotations in a temporary tree.**
 `xsl:copy-of` of a document node with a `validation` attribute was written as text and read
 again, so the copy was untyped and `id()` and `idref()` found nothing in it. With
 `validation="strip"` or `"preserve"`, a copy now keeps the is-id and is-idrefs properties
 (XSLT 3.0 §25.4.1); it cleared them. W3C copy-5031..5034.
- **A function value runs as the module that made it.** `unparsed-text`, the other text
 functions, `static-base-uri`, `stream-available` and `fn:transform` read the stylesheet's
 current position. So a closure, a partial application or a named reference made in one module and
 called in another resolved a relative location against the caller's base URI, and
 `ResourceRequest.ModuleUri` named the caller. Requires the XQuery change of the same name.
- **An `xml:base` in a stylesheet module that is not a URI is XTSE0020**; it left the
 transformer as a `UriFormatException`. `file:/path` (one slash) is read as `file:///path`.
- **`fn:transform` turned off by the resource policy raises FOXT0004** ("the XSLT
 transformation has been disabled", F&O 3.1); it raised FOXT0001, which means no suitable
 XSLT processor.