Encoding-Converters-Core 0.9.9

Encoding Converters Core (EncConverters / SIL.Transduction)

A 20+ year old transduction platform, developed by SIL, that converts text between character encodings and — more recently — between languages. It exposes a single stable API, IEncConverter, behind which many different conversion engines ("transducers") can be plugged in interchangeably.

It was originally built to solve one very specific problem for Bible translation and literacy work: converting text out of legacy 8-bit "Ansi-hack" fonts (custom fonts that mapped, say, Devanagari glyphs onto code page 1252) into Unicode. That problem is largely solved today — everyone has either migrated to Unicode or already has a copy of the sibling GUI project SILConverters to do it for them. What keeps this repo alive and under active development is that the same plug-in architecture turned out to be a convenient place to also host AI-driven machine translation engines (Azure OpenAI, Google Vertex AI/Gemini, Google Translate, Bing/Microsoft Translator, DeepL, and a self-hosted Meta NLLB model), so client apps that already know how to call IEncConverter get translation "for free."

Client applications include Paratext, FieldWorks, LibreOffice/MS Office plugins, and various in-house SIL tools, on both x86 and x64 hosts, via .NET, native C/C++, and COM.

Architecture

                        [Client Applications]
        Paratext · FieldWorks · MS Office · LibreOffice · in-house tools
                                  │
                                  ▼
                  IEncConverter / IEncConverterConfig
                (src/ECInterfaces — the stable public contract)
                                  │
                                  ▼
              EncConverters dispatcher ("SilEncConverters40.dll")
                          (src/EncCnvtrs)
        - discovers implementations via HKLM\SOFTWARE\SIL\SilEncConverters40
        - resolves named mappings via the mappingRegistry.xml repository
        - instantiates the right transducer by ProgID / assembly version
                                  │
        ┌─────────────────────────┴─────────────────────────────┐
        │                                                        │
   Legacy encoding / font transducers                  AI machine-translation transducers
   (native C/C++ engines, P/Invoked)                    (src/EcTranslators, online + self-hosted)
   ├─ Consistent Changes  — src/CcEC                    ├─ Azure OpenAI     — AzureOpenAi
   ├─ TECkit              — built into EncCnvtrs         ├─ Google Vertex AI — VertexAi (Gemini)
   ├─ ICU convert/regex/  — src/IcuEC                    ├─ Google Translate — GoogleTranslator
   │  transliterate                                      ├─ Bing/MS Translator — BingTranslator
   ├─ Python script       — src/PyScriptEC (Py2 native    ├─ DeepL            — DeepLTranslator
   │  or Py.NET for Py3)                                  └─ Meta NLLB        — NllbTranslator
   ├─ Perl expression     — src/PerlExpressionEC             (self-hosted Docker service;
   ├─ Windows code page   — src/CodePageEC                    supports private fine-tuned models)
   ├─ Font↔ISCII (Indic)  — src/SilIndicEncConverters
   └─ AdaptIt KB / guesser— src/AIGuesserEC, DriveAiEncConverter
                                                          (Note: "AI" in these two names means
                                                           "AdaptIt", a legacy tool — not AI/LLM)

Every transducer, native or managed, ultimately implements IEncConverter (or the native COM equivalent in src/EncCnvtrs/lib/ECEncConverter.h), so the dispatcher and all client apps only ever code against one interface.

Repository layout

Building

Windows

msbuild "EncConverters 2019.sln" /p:Configuration=Debug /p:Platform=x86
msbuild "EncConverters 2019.sln" /p:Configuration=Debug /p:Platform=x64

NuGet restore is required first (nuget restore "EncConverters 2019.sln"). The solution cannot be built AnyCPU — several native DLLs (CC, TECkit, ICU, Python, GuesserEC, SilFont2Iscii) are pinned to a specific bit width, so x86 and x64 must be built as fully separate configurations.

Linux

sudo apt-get install automake g++ python-dev libicu-dev mono5-sil icu-dev-fw libteckit-dev
(. environ && ./autogen.sh && make)      # first build
(. environ && make)                      # subsequent builds

Tests run under Mono:

export EC_COMMON_APPLICATION_DATA_PATH="$(pwd)/ec-common"
export MONO_REGISTRY_PATH="${EC_COMMON_APPLICATION_DATA_PATH}/registry"
cp -a src/RunTests/bin/x64/Debug/RunTests.exe output/x64/Debug/ &&
  (. environ && mono output/x64/Debug/RunTests.exe)

See also ReadMe_linux.txt and ReadMe_windows.txt.

NuGet package

The published package Encoding-Converters-Core is built from Package.Debug.nuspec / Package.Release.nuspec. To cut a new version: run Release builds for both x86 and x64, bump the version in the nuspec, then nuget.exe pack.

Configuration & discovery

Converters are looked up two ways at runtime:

1. HKLM\SOFTWARE\SIL\SilEncConverters40\ConvertersSupported — every installed transducer implementation self-registers here (COM ProgID, priority, per-assembly-version overrides).
2. The mapping registry — an XML file (mappingRegistry.xml, schema in src/ECInterfaces/SILMappingRegistry.xsd) holding named, user-configured conversion mappings, normally under %ProgramData%\SIL\Repository (Windows) or /var/lib/encConverters (Linux), overridable via EC_COMMON_APPLICATION_DATA_PATH.

Several transducer config dialogs embed an HTML help/setup page via WebBrowserAdaptor (src/EncCnvtrs), which can use Internet Explorer (WinForms WebBrowser), GeckoFX/XULRunner (bundled, Linux's only option), or WebView2/Edge (the modern default on Windows). See Handover.md for the plan to consolidate this.

A host app's compile-time references don't cover its runtime dependencies

Because converters are resolved dynamically by ProgID/assembly name at runtime (see above), a host application typically only compiles against SilEncConverters40.dll (the IEncConverter contract and dispatcher) and never references the individual transducer assemblies at all. That's enough to get the app running, but it is not enough to make any given converter actually work — each transducer implementation still has to be loadable from the host's own output/bin folder at the moment EncConverters.InstantiateIEncConverter tries to instantiate it, compile-time reference or not.

This matters most for src/EcTranslators (the AI/LLM translation engines), which pull in a large, fast-moving set of third-party packages (Newtonsoft.Json, the Google.Api.*/Google.Cloud.* family, the Grpc.* family, Azure.AI.OpenAI, Azure.Core, OpenAI, various Microsoft.Extensions.* and System.* compatibility shims — see SharedItems.props at the repo root for the authoritative list). If a host app wants any EcTranslators-provided converter (Azure OpenAI, Vertex AI, Google Translate, Bing, DeepL, or NLLB) available at runtime, it needs that same package set sitting alongside its own binaries — regardless of whether it references EcTranslators.csproj/EcTranslators.dll directly.

The projects in this repo that can end up hosting a dynamically-loaded converter (EcTranslators itself, ECFileConverter, TestEncCnvtrs, RunTests) all import $(SolutionDir)SharedItems.props directly rather than duplicating its PackageReference list — this is deliberate: it's the single source of truth for "what a host needs to use EcTranslators converters," shared not just by these few in-repo consumers but by dozens of host EXEs in the separate SILConverters solution. If you're building a new host application against this NuGet package and want to use any EcTranslators converter, import or replicate that same package list rather than guessing at a subset from compile-time references alone.

Further reading

• Handover.md — current-state assessment and modernization roadmap for anyone picking up maintenance of this repo.
• SILConverters developer notes — hints on consuming EncConverters from your own application.