Skyline.DataMiner.Utils.GQI.Extensions 1.0.0

Skyline.DataMiner.Utils.GQI.Extensions

About

Utilities to facilitate the implementation of GQI extensions.

About DataMiner

DataMiner is a transformational platform that provides vendor-independent control and monitoring of devices and services. Out of the box and by design, it addresses key challenges such as security, complexity, multi-cloud, and much more. It has a pronounced open architecture and powerful capabilities enabling users to evolve easily and continuously.

The foundation of DataMiner is its powerful and versatile data acquisition and control layer. With DataMiner, there are no restrictions to what data users can access. Data sources may reside on premises, in the cloud, or in a hybrid setup.

A unique catalog of 7000+ connectors already exist. In addition, you can leverage DataMiner Development Packages to build you own connectors (also known as "protocols" or "drivers").

Note See also: About DataMiner.

About Skyline Communications

At Skyline Communications, we deal in world-class solutions that are deployed by leading companies around the globe. Check out our proven track record and see how we make our customers' lives easier by empowering them to take their operations to the next level.

How to use the GQI optimizer

1. Define a field mapping

Map GQI columns to data source exposers using the fluent FieldMap API:

var nameColumn = new GQIStringColumn("Name");
var createdAtColumn = new GQIDateTimeColumn("Created At");
var idColumn = new GQIStringColumn("ID");
var scoreColumn = new GQIDoubleColumn("Score");

var fieldId = new FieldDescriptorID(Guid.NewGuid());

IFieldMap<DomInstance> fieldMap = FieldMap.For<DomInstance>()
    .Map(nameColumn).ToName()
    .Map(idColumn).ToInstanceId()
    .Map(createdAtColumn).To(DomInstanceExposers.CreatedAt)
    .Map(scoreColumn).ToField(fieldId)
    .Build();

The field mapping can be stored as IFieldMap<T> and reused to create a new optimizer for every data source instance.

2. Create an optimizer

The optimizer should be created once per data source instance before the IGQIOptimizableDataSource.Optimize method is called:

var optimizer = fieldMap.CreateOptimizer();

3. Optimize operators (in IGQIOptimizableDataSource.Optimize)

public IGQIQueryNode Optimize(IGQIDataSourceNode dataSourceNode, IGQICoreOperator nextOperator)
{
    return _optimizer.Optimize(dataSourceNode, nextOperator);
}

The Optimize extension method returns dataSourceNode unchanged if the operator was fully optimized, otherwise it appends the operator.

Both filter and sort operators are supported. Mapped sort fields are converted into IOrderBy elements in the built query.

4. Build the optimized query

Use the accumulated filters and sort orders when fetching data:

var query = _optimizer.BuildQuery();
// Pass query to your data source, e.g. DomHelper.DomInstances.Read(query)

A base query can be provided to extend with additional static filters:

var query = _optimizer.BuildQuery(baseQuery);

Built-in filter converters

The FilterConvert class provides ready-made converters for common scenarios.

FilterConvert.StringToGuid

Converts a string column value to a Guid for the exposer:

.Map(idColumn).To(DomInstanceExposers.Id, FilterConvert.StringToGuid())

// Or with a custom format:
.Map(idColumn).To(DomInstanceExposers.Id, FilterConvert.StringToGuid("N"))

FilterConvert.Value

Transforms the filter value using a function that returns a Result<T>. Returning default signals a failed conversion:

.Map(valueColumn).ToField(fieldId, FilterConvert.Value<double?, long>(filterValue =>
{
    if (!(filterValue is double doubleValue))
        return default;

    var longValue = (long)doubleValue;
    if ((double)longValue != doubleValue)
        return default;

    return longValue;
}))

FilterConvert.Range

Maps a filter value to a FilterRange<T> for ordinal comparisons. This is useful when a single column value represents a range of underlying values (e.g. a rounded "days" column mapped to a DateTime exposer):

var converter = FilterConvert.Range<int?, DateTime>(filterValue =>
{
    if (!(filterValue is int days))
        return null;

    var minTime = referenceTime.Subtract(TimeSpan.FromDays(days + 1));
    var maxTime = referenceTime.Subtract(TimeSpan.FromDays(days));

    return Range.Between(Bound.Exclusive(minTime), Bound.Inclusive(maxTime));
});

.Map(daysColumn).To(DomInstanceExposers.CreatedAt, converter)

Ranges are created using the Range factory methods and Bound helpers:

Bounds are created with Bound.Inclusive(value) or Bound.Exclusive(value).

FilterConvert.StringDiscrete

Maps discrete string display values to underlying data source values using a DiscreteMap. This handles Equals, DoesNotEqual, Contains, DoesNotContain, MatchesRegex, and DoesNotMatchRegex filter methods:

var stateMap = DiscreteMap.Define<string, int>()
    .WithValue("Initial", 0)
    .WithValue("In progress", 1)
    .WithValue("Completed", 2)
    .WithDefault("Unknown");

.Map(stateColumn).ToField(fieldId, FilterConvert.StringDiscrete(stateMap))

You can also create a discrete map from an enum and transform its values:

var colorMap = DiscreteMap
    .FromEnumNames<ConsoleColor>()
    .Map(color => (int)color);

.Map(colorColumn).ToField(fieldId, FilterConvert.StringDiscrete(colorMap))

List filter converters

The ListFilterConvert class provides converters for columns backed by string-joined list values.

ListFilterConvert.StringJoined

For columns where the cell value is a list of strings joined by a separator (e.g. "A, B, C"):

.Map(tagsColumn).ToField(fieldId, ListFilterConvert.StringJoined(", "))

ListFilterConvert.StringJoinedGuid

For columns where the cell value is a list of GUIDs joined by a separator:

.Map(idsColumn).ToField(fieldId, ListFilterConvert.StringJoinedGuid(", "))

Custom filter converters

When none of the built-in converters fit, you can write a custom FilterConverter. A FilterConverter is a delegate that takes a GQIFilterMethod and the column value, and returns a FilterConversion<T> (or default if the conversion is not possible):

.Map(ageColumn).To(DomInstanceExposers.CreatedAt, (method, value) =>
{
    if (!(value is TimeSpan age))
        return default;

    var createdAt = referenceTime.Subtract(age);
    switch (method)
    {
        case GQIFilterMethod.IsLessThan:
            method = GQIFilterMethod.IsGreaterThan;
            break;
        case GQIFilterMethod.IsLessThanOrEquals:
            method = GQIFilterMethod.IsGreaterThanOrEquals;
            break;
        case GQIFilterMethod.IsGreaterThan:
            method = GQIFilterMethod.IsLessThan;
            break;
        case GQIFilterMethod.IsGreaterThanOrEquals:
            method = GQIFilterMethod.IsLessThanOrEquals;
            break;
    }

    return Filter<DateTime>.Value(method, createdAt);
})

Note MatchesRegex and DoesNotMatchRegex filters cannot be optimized for DOM queries yet. Even if a custom converter returns a result for these methods, the filter will not be pushed down to the data source and will always be applied in memory by GQI instead.

Sort optimization

By default, every mapped column supports sort optimization with the sort direction passed through unchanged (SortOptimization.Direct). This can be changed by calling WithSortOptimization(...) after To(...).

Inverted sort

When a column's sort direction is the inverse of the underlying field's order (as in the age example above — ascending age means descending timestamp), use SortOptimization.Inverted to ensure sort operators are optimized correctly:

.Map(ageColumn).To(DomInstanceExposers.CreatedAt, (method, value) =>
{
    // ... converter as above ...
})
.WithSortOptimization(SortOptimization.Inverted)

Disabling sort optimization

To prevent a mapped column from being included in sort optimization — causing GQI to apply the sort in memory instead — use SortOptimization.None:

.Map(nameColumn).To(DomInstanceExposers.Name)
.WithSortOptimization(SortOptimization.None)

This is useful when the data source sort order for a field does not match the semantics of the GQI column.

Note Sort operators are all-or-nothing: if a multi-column sort contains any column with SortOptimization.None (or any unmapped column), the entire sort operator falls back to GQI in-memory sorting.

WithSortOptimization(...) can be chained with further Map(...) calls to continue building the field mapping.

Partial optimization

By default, returning a Filter<T> from a converter marks the filter as fully optimized (the original filter will no longer be applied in GQI). When the server-side filter is a superset of the original filter (e.g. Contains mapped to a broader match), mark it as partially optimized so GQI still applies the original filter on the results:

.Map(firstNameColumn).To(DomInstanceExposers.Name, (method, value) =>
{
    if (!(value is string name))
        return default;

    if (method != GQIFilterMethod.Equals)
        return default;

    return Filter<string>.Value(GQIFilterMethod.Contains, name).PartiallyOptimized();
})