NovoNordisk.SpecTrace 0.10.24

NovoNordisk.SpecTrace

SpecTrace is a CLI tool designed to provide traceability between user requirement specifications and executed test results. It facilitates the generation of various test reports (test report, validation reports, and custom reports), ensuring that your software development process meets predefined requirements.

At the core of SpecTrace, it essentially revolves around a configuration file that contains references to the paths where the requirements are defined and where the test reports are stored.

SpecTrace is inteded to run as part of your CI/CD pipeline automation flow.

Features

• Extract Requirements
  • Automatically extract requirement specifications written as code.
• Extract Test Results
  • Retrieve results from unit, integration, and end-to-end (e2e) tests written in various programming language and frameworks.
• Map Requirements & Test Results
  • Establish a connection between requirements and their corresponding test results.
  • The connection is established by tagging the requirements and then linking it from the actual test.
• Validate
  • Ensure that all requirements are met and tested.
• Generate Reports
  • Create detailed compliance reports, including test reports, implementation/validation reports, and custom reports.

Installation

To install SpecTrace as a NuGet package, run the following command:

dotnet tool install --global NovoNordisk.SpecTrace

Prerequisites

SpecTrace, by design, does not include aggregators for various test frameworks such as xUnit, NUnit, Playwright, etc. Instead, it leverages the capabilities of the Allure Reports aggregator.

Pick your framework(s):

• https://allurereport.org/docs/xunit/
• https://allurereport.org/docs/reqnroll/
• https://allurereport.org/docs/specflow/
• https://allurereport.org/docs/playwright/
• https://allurereport.org/docs/nunit/
• https://allurereport.org/docs/cypress/
• https://allurereport.org/docs/vitest/
• https://allurereport.org/docs/pytest/
• https://allurereport.org/docs/pytestbdd/
• ... or any other framework that Allure supports

(We do not use the "allure generate" command or any other Allure commands, except for accessing the actual test result files generated by Allure.)

Why Allure Reports?

Allure Reports provides a flexible and comprehensive way to aggregate test results from numerous testing frameworks. By using Allure Reports, SpecTrace ensures compatibility with a wide range of existing frameworks without the need to develop and maintain individual aggregators.

Using the Allure Reports aggregator ensures that SpecTrace can seamlessly integrate with your existing test frameworks to extract test results while focusing on providing high-quality, detailed compliance reports.

Aggregator Flexibility

SpecTrace is designed to be tool-agnostic. This means that while we currently use Allure Reports to extract test results, the underlying architecture of SpecTrace allows for flexibility in changing or adding aggregators as needed.

Getting started with SpecTrace

Step 1: Start writing your requirements

Create a file: requirements/first-requirement.adoc

[[URS:CreateAccount]]
= URS: Create a new user account
As a user, I want to be able to create a new account on the platform using my email address and password.

== Acceptance criteria

[[URS.AC:Min10CharPassword]]
- URS.AC: Password should contain minimum 10 characters

If you'd rather use markdown, that option is also available.

`@URS:CreateAccount`
# URS: Create a new user account
As a user, I want to be able to create a new account on the platform using my email address and password.

## Acceptance criteria

`@URS.AC:Min10CharPassword`
- URS.AC: Password should contain minimum 10 characters

Step 2: Link the test case with the requirement

(Read the Installation guide first)

In this example, we are using xUnit and the xUnit aggregator from Allure.

using Allure.Xunit;
using Xunit;

[AllureEpic("URS:CreateAccount")] // Tagging, the requirement
public class UserAccountTests
{
    [AllureLabel("spec", "URS:CreateAccount")] // Tagging, the individual acceptance criteria
    [Fact]
    public void Password_MinimumLength_Validation()
    {
        // Dummy example - not important for SpecTrace...
        // Arrange
        var email = "test@example.com";
        var password = "P@ssw0rd";
        
        // Act
        var result = UserAccountService.CreateNewAccount(email, password);

        // Assert
        Assert.True(result.Success);
        Assert.NotNull(result.AccountId);
    }
}

Step 3: Create the SpecTrace configuration file

Generate a file named "spectrace-config.yaml"

requirements:
  - path: ./requirements/pages/urs/**/*.adoc

testResults:
  - path: ./src/UnitTests/**/allure-results/*-result.json

params:
  productName: 'StudyHub - Study Start-Up'
  projectLink: 'https://dev.azure.com/my-org/example-name' 
  pipeline.link: '{projectLink}/_build/results?buildId={pipeline.buildId}&view=results'
  pipeline.displayName: 'Pipeline Execution'

reports:
  - outputFile: test-report.html
    templatePath: <TEST_REPORT_HTML>

Step 4: Run SpecTrace

spectrace report --config ./spectrace-config.yaml --param pipeline.buildId=insertValueFromPipeline

The test report will be created at ./dist/test-report.html

SpecTrace Configuration

The SpecTrace configuration provides complete flexibility to customize the entire behavior, while also incorporating sensible defaults to assist in most scenarios.

The configuration is divided into different sections:

• global config
• "validation" config
• "requirements" config
• "testResults" config
• "params" config
• "reports" config

You can omit any options with default values from the spectrace configuration. Include them if you want to be explicit about the value, even if it's the same as the default, or if you intend to override the default setting.

Global config options

showLogs: true
showConfigLogs: true
dist: dist

Validation config options

SpecTrace has built in validation verification that you can enable/disable based on your context.

validation:
  checkRequirementsHaveTests: true
  checkTestsWithBrokenLinks: true

Requirement config options

SpecTrace supports two different file types out of the box:

• Markdown files
• Asciidoc files

requirements:
  - path: documentation/modules/requirements/pages/urs/**/*.adoc
    pattern: ^\[\[(?<parent>PARENT::?)?(?<tag>(?<tagId>[^:]*):{1}(?<tagTitle>[^:]*))\]\]$(\n^(=|\*|-)+\s(?<title>.+))?$
    tags: [ 'URS/URS.AC', 'URS/DS/DS.AC', 'URS/FS/FS.AC' ]

PATTERN_ASCIIDOC:

^\[\[(?<parent>PARENT::?)?(?<tag>(?<tagId>[^:]*):{1}(?<tagTitle>[^:]*))\]\]\r?$(\n^(=|\*|-|\.|1\.)+\s(?<title>.+))?$

PATTERN_MARKDOWN:

^`@(?<parent>PARENT::?)?(?<tag>(?<tagId>[^:]*):{1}(?<tagTitle>[^:]*))`\r?$(\n^(#|\*|-|\+|1\.)+\s(?<title>.+))?$

Extend with custom file types

SpecTrace uses regex for pattern matching and can support various types of flat files. You can extend the "pattern" with your custom pattern.

Ensure that your custom regex pattern includes these specified groups:

• <parent>
  • The pattern to find parents (in case you split requirements into several files)
  • Example: PARENT::
• <tag>
  • The entire tag
  • Example: URS:MyExample
• <tagId>
  • The ID of the tag must match the tags defined in "requirements.tags"
  • Example: URS
• <tagTitle>
  • The title of the tag
  • Example: MyExample
• <title>
  • The title of the requirement

TestResults config options

Specify the location where the executed test results are stored, and include a list of test results if applicable.

testResults:
  - path: '**/allure-results/**/*-result.json'
    # attachmentsPath: '**/allure-results/**/*-attachment.*'
    excludeTags: ['draft']
    reporter: Allure

Params config options

A key-value dictionary that will be passed further to the liquid template files as parameters.

params:
  key1: 'value'
  key2: 'value combined with {key1}'
  key3: 'value'

SpecTrace has some magic keywords that will be replaced runtime:

Reports config options

Define where the executed test results are stored. Include list of test results if relevant.

reports:
  - outputFile: test-report.html
    templatePath: <TEST_REPORT_HTML>
  - outputFile: validation-report.adoc
    templatePath: <VALIDATION_REPORT_ASCIIDOC>
  - outputFile: my-custom-report.html
    templatePath: my-custom-report-template.liquid

Magic templatePaths that will be replaced runtime:

Real case example

showLogs: true #default
showConfigLogs: true #default
dist: dist #default

validation:
  checkRequirementsHaveTests: true #default
  checkTestsWithBrokenLinks: true #default

requirements:
  - path: documentation/modules/requirements/pages/urs/**/*.adoc
    tags: [ 'URS/URS.AC', 'URS/DS/DS.AC', 'URS/FS/FS.AC' ] #default

testResults:
  - path: src/backend/StudyOps.AcceptanceTests/allure-results/*-result.json
    attachmentsPath: src/backend/StudyOps.AcceptanceTests/allure-results/*-attachment.*
    excludeTags: ['draft'] #default
  - path: src/backend/StudyOps.UnitTests/allure-results/*-result.json
    attachmentsPath: src/backend/StudyOps.UnitTests/allure-results/*-attachment.*
    excludeTags: ['draft'] #default

params:
  productName: 'StudyHub - Study Start-Up'
  projectLink: 'https://dev.azure.com/my-org/example-name' 
  git.repoName: 'studyops'
  git.requirementLink: '{projectLink}/_git/{git.repoName}/commit/{git.commitId}?refName=refs/heads/main&path={spectrace:filePathRelativeToRoot}&_a=contents'
  git.commitId: '4096e7b92ab651d8f3469162129c375d3c19f83f'
  pipeline.link: '{projectLink}/_build/results?buildId={pipeline.buildId}&view=results'
  pipeline.displayName: '{pipeline.number}'
  pipeline.number: '#2024.09.18.6-1532519'
  pipeline.buildId: '1532519'

reports:
  - outputFile: test-report.html
    templatePath: <TEST_REPORT_HTML>  #default
  - outputFile: validation-report.adoc
    templatePath: <VALIDATION_REPORT_ASCIIDOC>  #default