Imagibee.Gigantor
3.0.0
dotnet add package Imagibee.Gigantor --version 3.0.0
NuGet\Install-Package Imagibee.Gigantor -Version 3.0.0
<PackageReference Include="Imagibee.Gigantor" Version="3.0.0" />
paket add Imagibee.Gigantor --version 3.0.0
#r "nuget: Imagibee.Gigantor, 3.0.0"
// Install Imagibee.Gigantor as a Cake Addin #addin nuget:?package=Imagibee.Gigantor&version=3.0.0 // Install Imagibee.Gigantor as a Cake Tool #tool nuget:?package=Imagibee.Gigantor&version=3.0.0
Gigantor
Boosts regular expression search/replace performance including support for gigantic files and streams
It solves the following problems:
- search/replace of gigantic files that exceed RAM
- CPUs are under-utilized
- main thread is unresponsive
- searching streams
- searching compressed data
The approach is to partition the data into chunks which are simultaneously processed on multiple threads. Since processing takes place on worker threads, the main thread remains responsive. Since the chunks are reasonably sized it does not matter if the whole file can fit into memory.
RegexSearcher
RegexSearcher
is the class that boosts regular expression search/replace performance for gigantic files or streams. Search was benchmarked at about 2.7 Gigabyte/s which was roughly 4x faster than the single threaded baseline. It depends on a System.Text.RegularExpressions.Regex to do the searching of the partitions. It uses an overlap to handle matches that fall on partition boundaries. De-duping of the overlap regions is performed automatically at the end of the search so that the final results are free of duplicates. Performance can be further enhanced by simultaneously searching multiple regular expressions or files for use cases that have these dimensions.
// Create a regular expression to match urls
System.Text.RegularExpressions.Regex regex = new(
@"/https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b([-a-zA-Z0-9@:%_\+.~#()?&//=]*)/",
RegexOptions.Compiled);
// Create the searcher
Imagibee.Gigantor.RegexSearcher searcher = new("myfile", regex, progress);
// Do the search
Imagibee.Gigantor.Background.StartAndWait(searcher, progress, (_) => { });
foreach (var match in searcher.GetMatchData()) {
// Do something with the matches
}
// Replace all the urls with stackoverflow.com in a new file
using System.IO.FileStream output = File.Create("myfile2");
searcher.Replace(output, (match) => { return "https://www.stackoverflow.com"; });
LineIndexer
LineIndexer
is the class that creates a mapping between line numbers and file positions for gigantic files. Once the mapping has been created it can be used to quickly find the position at the start of a line or the line number that contains a position. Index creation was benchmarked at about 2.5 Gigabyte/s which was roughly 4x faster than the single threaded baseline.
// Create the indexer
LineIndexer indexer = new("myfile", progress);
// Do the indexing
Imagibee.Gigantor.Background.StartAndWait(indexer, progress, (_) => {});
// Use indexer to print the middle line
using System.IO.FileStream fs = new("myfile", FileMode.Open);
Imagibee.Gigantor.StreamReader reader = new(fs);
fs.Seek(indexer.PositionFromLine(indexer.LineCount / 2), SeekOrigin.Begin);
Console.WriteLine(reader.ReadLine());
Input Data
The input data can either be uncompressed files, or streams. Files should be used when possible because they were benchmarked to be faster than streams. However, one notable use case for streams is searching compressed data without decompressing it to disk first.
Examples
Benchmarks
Testing
Prior to running the tests run Scripts/setup
to prepare the test files. This script creates some large files in the temporary folder which are deleted on reboot. Once setup has been completed run Scripts/test
.
License
Versioning
This package uses semantic versioning. Tags on the main branch indicate versions. It is recomended to use a tagged version. The latest version on the main branch should be considered under development when it is not tagged.
Issues
Report and track issues here.
Contributing
Minor changes such as bug fixes are welcome. Simply make a pull request. Please discuss more significant changes prior to making the pull request by opening a new issue that describes the change.
Product | Versions Compatible and additional computed target framework versions. |
---|---|
.NET | net5.0 is compatible. net5.0-windows was computed. net6.0 is compatible. net6.0-android was computed. net6.0-ios was computed. net6.0-maccatalyst was computed. net6.0-macos was computed. net6.0-tvos was computed. net6.0-windows was computed. net7.0 is compatible. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 was computed. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. |
.NET Core | netcoreapp3.0 was computed. netcoreapp3.1 is compatible. |
.NET Standard | netstandard2.1 is compatible. |
MonoAndroid | monoandroid was computed. |
MonoMac | monomac was computed. |
MonoTouch | monotouch was computed. |
Tizen | tizen60 was computed. |
Xamarin.iOS | xamarinios was computed. |
Xamarin.Mac | xamarinmac was computed. |
Xamarin.TVOS | xamarintvos was computed. |
Xamarin.WatchOS | xamarinwatchos was computed. |
-
.NETCoreApp 3.1
- No dependencies.
-
.NETStandard 2.1
- No dependencies.
-
net5.0
- No dependencies.
-
net6.0
- No dependencies.
-
net7.0
- No dependencies.
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.
Version | Downloads | Last updated |
---|---|---|
3.0.0 | 476 | 4/5/2023 |
2.0.1 | 196 | 4/3/2023 |
2.0.0 | 190 | 4/3/2023 |
1.0.2 | 183 | 3/30/2023 |
1.0.1 | 201 | 3/25/2023 |
1.0.0 | 215 | 3/24/2023 |
0.8.2 | 245 | 3/8/2023 |
0.8.1 | 216 | 3/8/2023 |
0.8.0 | 238 | 3/6/2023 |
0.7.1 | 233 | 3/6/2023 |
0.7.0 | 232 | 3/5/2023 |
0.6.3 | 230 | 3/1/2023 |
0.6.2 | 231 | 2/21/2023 |
0.6.1 | 234 | 2/18/2023 |
0.6.0 | 259 | 2/18/2023 |
0.5.0 | 248 | 2/13/2023 |
0.4.1 | 255 | 2/10/2023 |
0.4.0 | 270 | 2/8/2023 |
0.3.5 | 373 | 2/7/2023 |
0.3.4 | 242 | 2/7/2023 |
0.3.3 | 250 | 2/6/2023 |
0.3.2 | 271 | 2/6/2023 |
0.3.1 | 260 | 2/6/2023 |