NovoNordisk.SpecTrace 1.0.15

NovoNordisk.SpecTrace

SpecTrace is a CLI tool designed to provide traceability between user requirement specifications as code and the executed test results. It facilitates the generation of various test reports (test report, validation reports, and custom reports), ensuring that your software development process meets and documents the predefined requirements.

Whether you adhere to the default configuration or customize the tool to fit your specific workflows, SpecTrace ensures a seamless integration into your CI/CD pipeline automation flow.

Installation

To install the latest version of SpecTrace as a NuGet package, run the following command:

dotnet tool install --global NovoNordisk.SpecTrace

Prerequisites

SpecTrace, by design, does not include aggregators for various test frameworks such as xUnit, NUnit, Playwright, etc. Instead, it leverages the capabilities of the Allure Reports aggregator.

Pick your framework(s):

• https://allurereport.org/docs/xunit/
• https://allurereport.org/docs/reqnroll/
• https://allurereport.org/docs/playwright/
• ... or any other (nunit, pytest, cypress, ...) framework that Allure supports

SpecTrace does not use the "allure generate" command or any other Allure commands, except for accessing the actual test result files automatically generated by Allure. If you are curious why we use Allure Reports, then please read more in the Why Allure Reports?

Commands

Command: Extract requirements

spectrace extract requirements --name=mymodule --path='./requirements/**/**.md' --verbose

# Output: dist/mymodule-requirements.json

The requirements command is used for extracting requirements and mapping them to the SpecTrace Requirement domain model. It accepts the following options:

Option	Type	Required	Default	Description
--name, -n	string	true	-	Name prefix to the file that will be generated as an output.
--path	string	true	-	Path to the file location. Supports globbing.
--tags	array	false	["URS/URS.AC", "URS/DS/DS.AC", "URS/FS/FS.AC"]	Tags hierarchy for requirement definitions.
--pattern	regex	false	Auto-detected based on the extension of the --path defintion (.md or .adoc)	Used when extracting the requirement tags from the defined files in --path. Not recommended to update.
--context	string	false	Current directory	Specifies the base directory. If not provided, the current directory is used as the base.
--dist, -d	string	false	"dist"	Location for generated files to be stored.
--verbose, -v	bool	false	false	Increase logging verbosity.

How to write Requirements as Code

SpecTrace supports both markdown and asciidoc when writing the requirements as code.

Example: ./requirements/create-account.md

`@URS:CreateAccount`
# URS: Create a new user account
As a user, I want to be able to create a new account on the platform using my email address and password.

## Acceptance criteria

`@URS.AC:Min10CharPassword`
- URS.AC: Password should contain minimum 10 characters

Rules

• All root tags should be unique (in default SpecTrace configuration the root tag prefix is "URS")
• Tags within the same root (URS) are scoped, which means that they should be unique within the same URS/scope

INCORRECT tag structure:

- URS:CreateAccount
   - URS.AC:Min10CharPassword
- URS:CreateAccount // duplicate root-tag, will fail.
  - URS.AC:Example

CORRECT tag structure:

- URS:CreateAccount
   - URS.AC:Min10CharPassword
- URS:ResetPassword
  - - URS.AC:Min10CharPassword // URS.AC:Min10CharPassword is unique _within_ the same URS scope, which is approved.

Command: Extract test results

spectrace extract testresults --name=unittests --path='src/unittests/allure-results/*-result.json' --attachments-path='src/unittests/allure-results/*-attachment*' --verbose

# Output: dist/unittests-testresults.json
# Output: All the matched attachments will be stored in: dist/attachments/...

The testresults command is used for extracting test results and mapping them to the SpecTrace Test Result domain model. It accepts the following options:

Option	Type	Required	Default	Description
--name, -n	string	Yes	-	Name prefix to the file that will be generated as an output.
--path	string	Yes	-	Path to the executed test results location. Supports globbing.
--attachments-path	string	No	-	Path to executed test result attachments. Supports globbing.
--exclude-tags	string	No	["draft"]	Define tags to exclude from the report.
--aggregator	string	No	"Allure"	Test result aggregator to parse executed test runs to SpecTrace domain models.
--check-testresults-exists	bool	No	true	Fail if no test results could be found based on the provided path.
--check-attachments-exists	bool	No	true	Fail if no test attachments could be found based on the provided path.
--context	string	false	Current directory	Specifies the base directory. If not provided, the current directory is used as the base.
--dist, -d	string	false	"dist"	Location for generated files to be stored.
--verbose, -v	bool	false	false	Increase logging verbosity.

How to Link the test case with the requirement tag

In this example, we are using xUnit and the xUnit aggregator from Allure.

There are different ways of creating the link from the test definition to the requirement.

Rules:

• There should always be a label named "req".
  • This should always refer to the "root" requirement tag (with default SpecTrace configuration, this would be the "URS:" tag.)
• Optional: Add a label named spec if you want to refer to any sub-requirements within the same URS tag.
  • You can add several spec-labels, if the same test case covers several sub-requirements (within the same URS tag).

using Allure.Xunit;
using Xunit;

[AllureLabel("req", "URS:CreateAccount")] // Tagging, the requirement
public class UserAccountTests
{
    [AllureLabel("spec", "URS.AC:Min10CharPassword")] // Tagging, the individual acceptance criteria
    // Optinal: Possibel to assign the same test definition to several sub-requirements within the same URS:CreateAccount
    // [AllureLabel("spec", "URS.AC:Max40CharPassword")]
    [Fact]
    public void Password_MinimumLength_Validation()
    {
        // Not important for SpecTrace...
    }
}

Command: Merge requirements & test results

# Output from 'extract requirements' command: dist/mymodule-requirements.json
# Output from 'extract testresults' command: dist/unittests-testresults.json

spectrace merge --name=requirements --requirements-path='dist/*-requirements.json' --testresults-path='dist/*-testresults.json' --verbose

# Output: dist/requirements-merged.json

The merge command is used to merge and validate the provided requirements with test result execution. It accepts the following options:

Option	Type	Required	Default	Description
--name, -n	string	Yes	-	Name prefix to the file that will be generated as an output.
--requirements-path	string	Yes	-	Path to SpecTrace extracted requirements. Supports globbing.
--testresults-path	string	Yes	-	Path to SpecTrace extracted test results. Supports globbing.
--context	string	false	Current directory	Specifies the base directory. If not provided, the current directory is used as the base.
--dist, -d	string	false	"dist"	Location for generated files to be stored.
--verbose, -v	bool	false	false	Increase logging verbosity.

Command: Generating Reports

# Test Report
spectrace report --name test-report.html --dist reports --source 'dist/requirements-merged.json' --template '<TEST_REPORT_HTML>' --params-file params.spectrace --verbose

# Output: reports/test-report.html
# Output: reports/attachments/... (all attachments has been copied from the specified path)

# Validation Report:
spectrace report --name validation-report.adoc --params-file .spectrace --source 'dist/requirements-merged.json' --template '<VALIDATION_REPORT_ASCIIDOC>' --verbose

# Output:

The report command is used for generating reports (implementation report and test report) based on the merged requirements and test results. It accepts the following options:

Option	Type	Required	Default	Description
--name, -n	string	Yes	-	Name including the extension.<br>Example: --name my-test-report.html
--source, -s	string	Yes	-	File path to the source data used as input to the report template.
--template, -t	string	Yes	-	Path to the liquid template, or built-in SpecTrace template reference.
--attachments-path	string[]	No	-	Path to attachments directories. Supports globbing.
--params-file	string	No	-	Path location to the SpecTrace params file.
--params, -p	string	No	-	Additional params to be included in the report.<br>Expected syntax: --meta status=draft
--context	string	false	Current directory	Specifies the base directory. If not provided, the current directory is used as the base.
--dist, -d	string	false	"dist"	Location for generated files to be stored.
--verbose, -v	bool	false	false	Increase logging verbosity.

Passing Parameters to SpecTrace Report CLI

When passing parameters to the SpecTrace report CLI, there are two methods available for interacting with the template engine (Read: How does the template engine works?).

1. Using a Parameter File: To start, you can utilize a file, typically named params.spectrace, which contains static variables. All dynamic parameters can then be passed as CLI parameters. Within the params.spectrace file, dynamic variables can be referenced. For instance:

productName=My Awesome Product
release=Release tag {tag}

2. Override Parameters: If you need to override specific parameters, you can achieve this by simply passing the --params tag=1.0.0 as a parameter to the spectrace report command. This approach allows for the straightforward adjustment of individual parameters as needed.

How does the template engine works?

The template engine used by SpecTrace operates as a liquid template parser behind the scenes. When utilizing the CLI, any data passed with --source (e.g., file.json) is sent to the specified template indicated by --template (e.g., mytemplate.liquid). This process allows for the dynamic generation of output based on the provided data and the template's structure.

SpecTrace uses https://github.com/sebastienros/fluid as the template engine based on the Liquid template language.

SpecTrace built-in templates

• Template: Test Report
  • --template <TEST_REPORT_HTML>
• Template: Validation Report
  • asciidoc:
    • --template <VALIDATION_REPORT_ASCIIDOC>
  • markdown:
    • --template <VALIDATION_REPORT_MARKDOWN>

Parameters:

Parameter	Required	Example	Description
productName	Yes	--params productName="SpecTrace"	Used to show the Product Name in the Test Report
pipeline.link	No	--params pipeline.link="https://dev.azure.com/orgname/project/_build/results?buildId=buildid"	Link to the pipeline from where the report has been generated.
pipeline.displayName	No	--params pipeline.displayName="Build ID #123"	Used to show the Product Name in the Test Report
git.commitId	No	--params git.commitId="commitId"	Used to show the Product Name in the Test Report
git.requirementLink	No	--params git.requirementLink="https://dev.azure.com/orgname/project/_git/gitrepo/?version=GC{git.commitId}\u0026path={spectrace:filePathRelativeToRoot}"	Creates a link for each requirement in the test report that links to the specific commitId and file location where your requirement is defined.<br>Note that {spectrace:filePathRelativeToRoot} is important and will be replaced runtime.

(Only for Novo Nordisk employees: Find the SpecTrace team in our Developer Portal, to get the verified internal-only validation report template.)

FAQ

Why Allure Reports?

Allure Reports provides a flexible and comprehensive way to aggregate test results from numerous testing frameworks. By using Allure Reports, SpecTrace ensures compatibility with a wide range of existing frameworks without the need to develop and maintain individual aggregators.

Using the Allure Reports aggregator ensures that SpecTrace can seamlessly integrate with your existing test frameworks to extract test results while focusing on providing high-quality, detailed compliance reports.

Test Result Aggregator Flexibility

SpecTrace is designed to be tool-agnostic. This means that while we currently use Allure Reports to extract test results, the underlying architecture of SpecTrace allows for flexibility in changing or adding aggregators as needed.

Markdown & Asciidoc Requirement example

Requirement definition in markdown:

`@URS:CreateAccount`
# URS: Create a new user account
As a user, I want to be able to create a new account on the platform using my email address and password.

## Acceptance criteria

`@URS.AC:Min10CharPassword`
- URS.AC: Password should contain minimum 10 characters

Equivalent definition in asciidoc:

[[URS:CreateAccount]]
= URS: Create a new user account
As a user, I want to be able to create a new account on the platform using my email address and password.

== Acceptance criteria

[[URS.AC:Min10CharPassword]]
- URS.AC: Password should contain minimum 10 characters

Supported --pattern

--pattern is dynamically set based on the --path extension:

Markdown (example: --path ./requirements/*.md)

^`@(?<draft>DRAFT::?)?(?<parent>PARENT::?)?(?<tag>(?<tagId>[^:].+):{1}(?<tagTitle>[^:].+))`\r?$(\n^(#|\*|-|\+|1\.)+\s(?<title>.+))?$

Asciidoc: (example: --path ./requirements/*.adoc)

^\[\[(?<draft>DRAFT::?)?(?<parent>PARENT::?)?(?<tag>(?<tagId>[^:].+):{1}(?<tagTitle>[^:].+))\]\]\r?$(\n^(=|\*|-|\.|1\.)+\s(?<title>.+))?$

Create your own custom "--pattern"

SpecTrace uses regex for pattern matching and can support various types of flat files. You can extend the "pattern" with your custom pattern.

Ensure that your custom regex pattern includes these specified groups:

• <parent>
  • The pattern to find parents (in case you split requirements into several files)
  • Example: PARENT::
• <tag>
  • The entire tag
  • Example: URS:MyExample
• <tagId>
  • The ID of the tag must match the tags defined in "requirements.tags"
  • Example: URS
• <tagTitle>
  • The title of the tag
  • Example: MyExample
• <title>
  • The title of the requirement