Stardust.Paradox.Data.InMemory 1.0.0-preview.2

Stardust.Paradox.Data.InMemory

A high-performance, TinkerGraph-inspired in-memory Gremlin database implementation for testing, development, and rapid prototyping. Perfect for unit testing, integration testing, and AI agent development with graph data.

?? Quick Start

Basic Connector Creation

using Stardust.Paradox.Data.InMemory;

// Create an empty connector
var connector = InMemoryGremlinLanguageConnector.Create();

// Create with custom options
var connector = InMemoryGremlinLanguageConnector.Create(options =>
{
    options.EnableDebugLogging = true;
    options.AutoGenerateIds = true;
    options.ValidateEdgeVertices = true;
});

Creating with Pre-built Scenarios

The most efficient way to get started is with pre-built scenarios:

// Create with social network data already loaded
var connector = InMemoryConnectorFactory.CreateSocialNetwork();

// Create with e-commerce data
var connector = InMemoryConnectorFactory.CreateECommerce();

// Create with organizational hierarchy
var connector = InMemoryConnectorFactory.CreateOrganization();

// Create with user/role management data
var connector = InMemoryConnectorFactory.CreateUserManagement();

?? Loading Scenarios After Creation

Method 1: Using Factory with Scenarios

// Load single scenario
var connector = InMemoryConnectorFactory.CreateWithScenario("BasicSocialNetwork");

// Load multiple scenarios
var connector = InMemoryConnectorFactory.CreateWithScenarios(
    new[] { "BasicSocialNetwork", "SimpleECommerce" });

Method 2: Manual Data Population

// Create empty connector
var connector = InMemoryGremlinLanguageConnector.Create();

// Add vertices directly
connector.AddVertex("person", new Dictionary<string, object>
{
    ["name"] = "John",
    ["age"] = 30,
    ["email"] = "john@example.com"
}, "john");

connector.AddVertex("person", new Dictionary<string, object>
{
    ["name"] = "Jane",
    ["age"] = 28,
    ["email"] = "jane@example.com"
}, "jane");

// Add edges
connector.AddEdge("knows", "john", "jane", new Dictionary<string, object>
{
    ["since"] = "2020-01-01",
    ["strength"] = 0.8
});

Method 3: Using ImportData with Definitions

var connector = InMemoryGremlinLanguageConnector.Create();

// Define vertices
var vertices = new[]
{
    new InMemoryVertexDefinition("user1", "user", new Dictionary<string, object>
    {
        ["name"] = "Alice",
        ["role"] = "admin"
    }),
    new InMemoryVertexDefinition("user2", "user", new Dictionary<string, object>
    {
        ["name"] = "Bob",
        ["role"] = "editor"
    })
};

// Define edges
var edges = new[]
{
    new InMemoryEdgeDefinition("manages", "user1", "user2")
};

// Import all at once
connector.ImportData(vertices, edges);

?? Available Pre-built Scenarios

Scenario Data Examples

BasicSocialNetwork

var connector = InMemoryConnectorFactory.CreateSocialNetwork();

// Pre-loaded data includes:
// - Users: john, jane, bob with profiles
// - Posts: various social media posts
// - Relationships: friendship connections
// - Interactions: likes, comments

// Immediate queries work:
var friends = await connector.ExecuteAsync("g.V('john').out('knows')", new Dictionary<string, object>());
var posts = await connector.ExecuteAsync("g.V().hasLabel('post').has('author', 'john')", new Dictionary<string, object>());

SimpleECommerce

var connector = InMemoryConnectorFactory.CreateECommerce();

// Pre-loaded data includes:
// - Customers with purchase history
// - Products with categories and pricing
// - Orders with line items
// - Purchase relationships

// Example queries:
var customerOrders = await connector.ExecuteAsync("g.V('customer1').out('placed')", new Dictionary<string, object>());
var productPurchasers = await connector.ExecuteAsync("g.V('product1').in('contains').in('placed')", new Dictionary<string, object>());

?? Advanced Configuration and Population

Custom Scenario Creation

public class CustomBlogScenario : InMemoryScenarioProviderBase
{
    public override string ScenarioName => "BlogPlatform";
    public override string Description => "Blog platform with authors and posts";

    protected override (InMemoryVertexDefinition[] vertices, InMemoryEdgeDefinition[] edges) GetScenarioData()
    {
        var vertices = new InMemoryVertexDefinition[]
        {
            new InMemoryVertexDefinition("author1", "author", Props(
                ("name", "Alice Writer"),
                ("email", "alice@blog.com"),
                ("posts_count", 15)
            )),
            new InMemoryVertexDefinition("post1", "post", Props(
                ("title", "Getting Started with Graphs"),
                ("content", "Welcome to graph databases..."),
                ("published_date", "2024-01-15")
            ))
        };

        var edges = new InMemoryEdgeDefinition[]
        {
            new InMemoryEdgeDefinition("authored", "author1", "post1")
        };

        return (vertices, edges);
    }
}

// Register and use
InMemoryScenarioRegistry.Register(new CustomBlogScenario());
var connector = InMemoryConnectorFactory.CreateWithScenario("BlogPlatform");

Fluent Configuration

var connector = InMemoryGremlinLanguageConnector.Create()
    .WithScenario("BasicSocialNetwork")
    .WithScenario("SimpleECommerce");

// Clear and reconfigure
connector.ClearScenarios()
        .WithScenarios("UserRoleManagement", "OrganizationHierarchy");

Runtime Data Population

var connector = InMemoryGremlinLanguageConnector.Create();

// Start with a scenario
connector.ImportData(InMemoryScenarioRegistry.GetScenario("BasicSocialNetwork"));

// Add more data programmatically
await connector.ExecuteAsync("g.addV('company').property('name', 'TechCorp')", new Dictionary<string, object>());
await connector.ExecuteAsync("g.V('john').addE('works_for').to(g.V().hasLabel('company').has('name', 'TechCorp'))", new Dictionary<string, object>());

// Check current state
var stats = connector.GetStatistics();
Console.WriteLine($"Total vertices: {stats.VertexCount}, edges: {stats.EdgeCount}");

?? Testing Integration Patterns

Unit Testing with Scenarios

[Test]
public async Task Should_FindUserConnections()
{
    // Arrange - Start with known data
    var connector = InMemoryConnectorFactory.CreateSocialNetwork();
    
    // Act - Test your logic
    var mutualFriends = await connector.ExecuteAsync(
        "g.V('john').out('knows').where(in('knows').hasId('jane'))", 
        new Dictionary<string, object>());
    
    // Assert
    mutualFriends.Should().NotBeEmpty();
}

AI Agent Testing

// Perfect for AI agents that need predictable graph data
var connector = InMemoryConnectorFactory.CreateSocialNetwork();

// AI can immediately start querying without setup
var socialGraph = await connector.ExecuteAsync("g.V().hasLabel('person').limit(10)", new Dictionary<string, object>());
var networkDensity = await connector.ExecuteAsync("g.E().count()", new Dictionary<string, object>());

// Add AI-specific test data
await connector.ExecuteAsync("g.addV('ai_agent').property('name', 'GraphBot')", new Dictionary<string, object>());

Performance Testing with Large Datasets

var connector = InMemoryGremlinLanguageConnector.Create(options =>
{
    options.TrackStatistics = true;
    options.EnableDebugLogging = false; // Disable for performance
});

// Load multiple scenarios for comprehensive testing
connector.ImportData(InMemoryScenarioRegistry.GetScenario("BasicSocialNetwork"));
connector.ImportData(InMemoryScenarioRegistry.GetScenario("SimpleECommerce"));
connector.ImportData(InMemoryScenarioRegistry.GetScenario("OrganizationHierarchy"));

// Now test complex queries
var complexResult = await connector.ExecuteAsync(
    "g.V().hasLabel('person').out('knows').out('knows').dedup().count()", 
    new Dictionary<string, object>());
    
Console.WriteLine($"Query consumed {connector.ConsumedRU} RU");

?? Known Issues

This section documents current limitations and known issues in the InMemory Gremlin database. The implementation currently has 330 passing tests out of 341 total tests (96.8% success rate).

?? Gremlin Query Limitations

Complex Aggregation Operations

• Issue: Some complex group() and groupCount() operations may not return expected keys
• Impact: Advanced grouping queries might fail or return unexpected results
• Workaround: Use simpler grouping approaches or direct database access methods
• Example Problem: g.V().groupCount().by('property') may return serialized objects instead of property values

Parameter Binding Edge Cases

• Issue: Some parameterized queries with complex data types may not bind correctly
• Impact: Certain queries with mixed parameter types might fail
• Workaround: Use direct string interpolation for complex scenarios (with proper escaping)

Advanced Traversal Steps

• Issue: Some advanced Gremlin steps are not fully implemented:
  • match() step
  • choose() with complex predicates
  • optional() in certain contexts
  • coalesce() with multiple alternatives
• Impact: Complex traversal patterns may need alternative approaches
• Workaround: Break complex traversals into simpler steps or use direct database methods

?? Scenario Framework Limitations

Scenario Data Consistency

• Issue: Some edge cases in scenario data loading may result in missing relationships
• Impact: Rare cases where expected graph connections don't exist
• Workaround: Verify data after loading and add missing relationships manually

Large Dataset Performance

• Issue: Very large scenarios (>10,000 vertices) may experience slower query performance
• Impact: Performance degradation with extensive test data
• Workaround: Use smaller, focused scenarios for unit tests; reserve large datasets for integration tests

?? Parser-Specific Issues

TinkerGraph Parser Fallbacks

• Issue: The TinkerGraph parser may fall back to simpler parsers for some valid queries
• Impact: Reduced performance for complex queries, but correct results
• Status: This is expected behavior - the system has multiple parser layers for reliability

Regular Expression Parsing

• Issue: Very complex nested queries may not parse correctly with the regex-based parser
• Impact: Some advanced query patterns require manual query construction
• Workaround: Use the direct database access methods for complex operations

?? Data Type Handling

Mixed Property Types

• Issue: Aggregations on properties with mixed data types (strings, numbers) may produce unexpected results
• Impact: sum(), max(), min() operations on heterogeneous data
• Workaround: Ensure consistent data types within properties used for aggregation

Date/Time Operations

• Issue: Limited support for date/time comparisons and operations
• Impact: Temporal queries may require custom implementation
• Workaround: Use epoch timestamps for date operations

?? Testing Framework Integration

xUnit Collection Fixtures

• Issue: Some xUnit collection fixture scenarios may not share data correctly
• Impact: Potential test isolation issues in certain configurations
• Workaround: Use fresh connector instances for each test class

?? Concurrent Access

Thread Safety Edge Cases

• Issue: Very high concurrency scenarios may experience rare synchronization issues
• Impact: Potential data inconsistency under extreme load
• Workaround: Use single-threaded access for critical test scenarios

?? Performance Considerations

Memory Usage with Large Graphs

• Issue: In-memory storage grows linearly with graph size
• Impact: Memory consumption for very large test datasets
• Workaround: Clear connectors between test suites: connector.Clear()

Query Optimization

• Issue: Some query patterns may not be optimally executed
• Impact: Slower performance for certain traversal patterns
• Workaround: Use direct database access methods for performance-critical operations

??? Reporting Issues

If you encounter issues not listed here:

1. Check Version: Ensure you're using version 1.0.0-preview.1 or later
2. Enable Debug Logging: Use options.EnableDebugLogging = true to see query execution details
3. Minimal Reproduction: Create a minimal test case that reproduces the issue
4. Report: Create an issue at GitHub Issues

Include in your report:

• InMemory connector version
• .NET framework version
• Complete error message and stack trace
• Minimal code example that reproduces the issue
• Expected vs. actual behavior

? Workaround Patterns

Direct Database Access

var connector = InMemoryConnectorFactory.CreateSocialNetwork();

// Instead of complex Gremlin query, use direct access
var vertices = connector.GetVerticesByLabel("person");
var filtered = vertices.Where(v => v.Properties.ContainsKey("age"));

Manual Data Verification

var connector = InMemoryConnectorFactory.CreateSocialNetwork();

// Verify scenario data loaded correctly
var stats = connector.GetStatistics();
Assert.True(stats.VertexCount > 0, "Scenario should load vertices");
Assert.True(stats.EdgeCount > 0, "Scenario should load edges");

Performance Optimization

var connector = InMemoryGremlinLanguageConnector.Create(options =>
{
    options.EnableDebugLogging = false; // Disable for performance
    options.TrackStatistics = false;   // Disable if not needed
});

?? Key Features for Developers

Immediate Query Execution