GSoC Proposal: Implementing a Plugin Development Kit for Tremor

The minimum common denominator to define binary interfaces most times is C. This would allow Tremor plugins to be written in any language, but the complexity of this project would increase tremendously. Types required for the interface like tremor_script::Value (currently used by some traits), error handling, and other advanced features like lifetimes or templates would have to be simplified to the C ABI. Not only would this be an incredibly tedious and complicated task, but it’d also introduce a small overhead for every function in the plugin.

Thus, the only viable solution is a Rust to Rust FFI, using dylib libraries for the plugins. The problem in this case is that Rust doesn’t have a stable ABI, and it doesn’t plan to define one in the future^[2]. This means that programs or plugins compiled with different versions of rustc won’t be compatible at all. This could be less of a problem with alternatives like abi_stable_crates, which provide unofficial stable ABIs, but they don’t follow Rust’s objectives nor are very popular, so these libraries might end up unmaintained or just not worth it.

Interface improvements may also happen in the future, where a method might be deprecated, removed, or added. The same could happen with Tremor; different API versions wouldn’t be backwards compatible either.

Knowing this, it’s important to correctly handle version upgrades. This means that some kind of interface versioning is needed. Tremor could disallow loading plugins that would inevitably break after a version upgrade, or perhaps it could show a message informing the user that they should upgrade their plugin before the deprecations turn into actual changes. Most Plugin Development Kit interfaces include some way of providing metadata about the plugin^[3]. This may include the following fields:

The Linux Kernel Module system for example inserts this in the .modinfo section of the binary, where the fields can be read as a key-value c-string. C extensions for Python, on the other hand, use a PyModuleDef struct with all the fields. The last option is by exporting globals with known names, which is the most straightforward.

The next step after declaring a basic interface is somehow importing the dynamic objects into the core of Tremor, in Rust. The main library that can help with this is libloading, which seems to be somewhat mature and well maintained. Although it does offer OS-dependent features, its main usage is wrapped to be cross-platform, following the same behaviour on all platforms. libloading would allow Tremor to load symbols from an external module, like functions or variables. Here’s an example of how it’s used. After loading the dynamic library, the Tremor core would wrap its exported functions so that they can be used without unsafe and unnecessary boilerplate.

Something to consider is where the plugins are going to be loaded dynamically from, or how that can be configured. Tremor currently has quite a few ways to handle configuration via repositories and registries.

One way to approach this would be to have a directory inside /etc/tremor/ — or any user-configurable path when launching Tremor — with all the available plugins. Tremor would read its contents when starting and load all the plugins it finds. It should also be possible to load new plugins on demand, so the repository could be given the path of a library to include at runtime, for example with the tremor CLI tool.

It’s important to make the plugin system as easy to use as possible, and to avoid breaking already existing installations of Tremor. For this, the default Tremor configuration should include the current libraries in the default dynamic library location.

The proposal establishes the following components of Tremor as the main deliverables:

After defining its interfaces and implementing the architecture, these features would be turned into plugins, which would allow them to be developed independently, and thus wouldn’t have to be in the main tremor-runtime repository. For simplicity, though, it might be best to keep them there. Or at least group them up in less repositories to avoid having too many. Some components are quite simple and it might not be necessary to create a new repository for them, with a copy of the CI, README…​ But this is up to the core contributors of Tremor, after all.

The requirements also list a few optional targets, which can be implemented if there’s remaining time after the main goals are fully finished:

The most likely to be implemented of these is the fifth, as documentation is important for this new breaking feature. It also looks like the easiest one, or at least seemingly more flexible, considering there most likely won’t be that much extra time after the main goals, if any.

The “development tooling” part would also be inevitably developed as the project progresses, since I’ll need them anyway to move the existing Implementations to the plugin system. Said resources could be contained in a separate tremor_plugin crate, with all kinds of utilities to make plugin development easier, including traits or even procedural macros if necessary, which are a very interesting part of Rust, and I’m looking forward to on working on as well, and I’ve already done in the past.

As the code is written, documentation and the tests also will. Tests are a great method to make sure a feature really works while developing it, and a solid way to move on to another feature when coding is to sum it up with documentation before forgetting more about its details; I consider it a bad idea to forget these points until the very end.

There are five main objectives proposed for the initial target, to be distributed in 9 weeks. Some will take more effort to implement, so here’s an estimate: