Plugins in Rust: The Technologies

Many of the technologies that can be used to implement a Plugin System work with unsafe code, like FFIs or IPC Memory Sharing. This isn’t necessarily a problem if it’s self-contained and thoroughly reviewed, but we lose some safety guarantees provided by Rust, increasing the maintenance cost of the library.

It takes considerably more work to make sure the implementation is sound, even with tools like MIRI, which I plan on integrating into Tremor if I do end up having to use unsafe.

Rust doesn’t protect its users from leaking memory. In fact, it’s as easy as calling mem::forget. The thing is that if a plugin leaks memory, the entire process also does, meaning that Tremor’s performance could be affected by incorrectly developed plugins.

This doesn’t happen just with memory leaks; a plugin could abort or panic and crash the entire thing very easily.

Ideally, Tremor could detect plugins that aren’t performing well and stop them before it’s too late. The core of the program could continue running even when a plugin fails, and perhaps warn the user about its malfunctioning, for optimal error resilience.

This was a problem with Internet Explorer, for example. It used COM and ActiveX, which implemented no sandboxing at all and were running directly on the user’s machine, allowing malware to be included in plugins with the ability to execute arbitrary code on the host machine. This can be less of a problem when installing only trusted plugins via digital signatures, but it’s still a huge risk.

In the case of Tremor plugins this is a similar problem. The end user — those who will add plugins to their configuration — is a developer, who should be more wary of what they’re including in their own projects. But the truth is that it isn’t any better.

I would compare the situation more specifically to how package managers like npm work (or anything that deals with dependencies, really). The entire infrastructure is usually based on trust; noone’s really stopping you from uploading a malicious package/crate for remote code execution or to steal data [1] [2]. Plugins are like dependencies in this case; they have full access to the host’s computer, and thus shouldn’t be trusted by default.

An improved approach to Node + npm would be something like Deno, which is a secure by default JavaScript/TypeScript runtime. This is enabled thanks to sandboxing, and requires the developer to manually toggle, e.g., access to the filesystem or to the network. This is no panacea because people might end up enabling whatever permission is required by the dependency they want to install without thinking about it, but it’s similar to unsafe: at least it makes you aware that you might mess up.

One could argue that their program is realistically going to be running in a Virtual Machine or a container most of the times, where this may not matter so much. But should you really assume that? Should user safety rely on the fact that the machine is running on an isolated kernel? Containers, by the way, introduce a much bigger overhead in comparison to some sandboxing methods. And even if the system as a whole is isolated, there is still a possibility for internal leaks: the Postgres plugin has access to whatever is being used by the Apache Kafka plugin, which may have sensitive logs you don’t want anyone to see.

The easiest fix for backward compatibility issues I can come up with is to serialize and deserialize the data with a flexible protocol rather than using its binary representation directly. If something like JSON is used to communicate between the core and its plugins, adding a field to a message won’t break anything, and removing one can be done via a deprecation process. Unfortunately, this introduces some overhead the application may not be interested in.

Other more involved methods for binary representations include [3]:

There are times when breakage is inevitable. Tremor may want to rewrite part of its interface or finally remove a deprecated feature without being afraid of breaking all the plugins previously developed.

For that, the plugins must embed a bit of metadata about the different versions of rustc/interface/etc it was developed for. Then, when it’s loaded by Tremor, it may check if they’re compatible, rather than breaking in mysterious ways.

Plenty of projects use scripting languages to extend their functionality at runtime, like Python, Ruby, Perl, Bash, or JavaScript. Most notably, Vim created its own scripting language, Vimscript, in order to be fully customizable, and NeoVim is now pushing Lua as a first-class language for configuration. Even Tremor itself has the interpreted query language Troy for configuration.

Lua is seen in game development; it’s a quite simple language with a very performant JIT implementation, which in any case I think would be the best option here. It could be embedded into our runtime (it’s only 281 kB compiled! [4]) and used to load plugins at either start-time or run-time. And knowing it’s used in games, which are obsessed with performance, it might not be that much of a big deal in that regard.

Rust has libraries like rlua which provide bindings for interoperability with Lua. rlua in particular seems to focus on having an idiomatic and safe interface, which is rare for a bindings library and good news, though it seems to be currently semi-abandoned, and forked by mlua. Unfortunately, after digging a bit the Rust ecosystem for Lua bindings doesn’t seem to be as mature as I’d like for a project this big; there’s still some work to do to reach more stability.

Lua gets extra points in safety. It’s possible to sandbox it by blocking whichever functions you don’t want users to access [5] (though it’s “tricky and generally speaking difficult to get right”). Similarly, one can also handle resource exhaustion issues within Lua programs. Not to mention that this wouldn’t require unsafe at all with an optimal set of bindings.

Anyhow, the main deal-breaker with scripting languages in this case is that it would be extremely complicated to port everything in Tremor so that it can be used from Lua. For new projects this could perhaps be something interesting, but not if the entirety of the already existing plugins have to be rewritten.

Still, it’s a pretty interesting option for new projects, as you have ABI stability guarantees, solid safety overall, and it’s very straightforward to use.

Another possibility for plugins is to define a protocol for Inter-Process Communication, turning your program into a server that extends its capabilities by connecting to external plugins. For instance, most text editors use this method to support the Language Server Protocol, which uses JSON-RPC.

There are of course multiple ways to do IPC, which I’ll briefly list below. Performance-wise, this graph shows a comparison of the overhead of each of them [6]:

This is possibly the least weird way to implement a Plugin Development Kit, i.e., it’s the most popular method I’ve seen outside of Rust. A Foreign Function Interface can allow us to directly access resources in separately compiled objects, even after the linking phase with dynamic loading. It’s one of the fastest options available because there’s no overhead at all after dynamically loading the library.

The main library for this is libloading. There’s also the less popular dlopen and sharedlib, with some small differences [9]. It seems to be a lower-level implementation for any kind of FFI that requires unsafe for almost everything — what I was expecting. Based on it there’s dynamic_reload, which is very interesting in order to “live reload” the plugins when they are recompiled. This would be useful for the development process of the plugins, since it also handles unloading the same plugin seamlessly, but that’s not a goal for this project, so I don’t plan on using it. And the dlib crate provides macros to make the library loading simpler, based on libloading.

Creating an internal interface for Tremor through C is a lot of work. But it’s unfortunately the only way to do it, since Rust-to-Rust FFI is not stable. It has awful safety by relying heavily on unsafe with plenty of caveats [10] [11], including subtle differences between Operating Systems [12] — although dlopen seems to be better in that regard [13]. Since it doesn’t have a sandbox, plugins can abort Tremor’s core execution when panicking/leaking memory/similars.

I recently discovered the abi_stable crate, which guarantees stability in Rust’s ABI unofficially and helps a lot with the binary compatibility aspect.

It includes FFI-safe alternatives to many of the types in the standard library, and even external ones (namely crossbeam, parking_lot and serde_json). This works by implementing a StableAbi trait that guarantees its FFI-safety, which may be done automatically with one of its procedural macros. Internal ABI stability is also guaranteed with macros like #[sabi(last_prefix_field)], which would allow Tremor to add fields to existing structs without breaking backwards compatibility.

Fortunately, this crate has a few very detailed examples one can look at to better understand how it works, and it’s exceptionally well documented. If I’ve understood it correctly, some of its inconvenients are the following:

While it’s a fascinating concept and look forward to seeing how it evolves, I’m not a big fan of having to resort to it. And the fact that it’s unofficial and not that popular doesn’t give me full confidence that this will still work in 5 years, or that it won’t be outdated/abandoned.

More people have tried writing Rust FFIs in the past, so we can take a look at existing tutorials in order to see their experience:

So, more or less:

Now, this is what I wanted to emphasize in this article! Turns out WebAssembly isn’t limited to web development anymore; it’s slowly evolving into a portable binary-code format. As far as I know, this should be like a mix between Dynamic Loading and Scripting Languages, with a stronger focus in stability and portability. Here’s what Wikipedia has to say about it:

So to clear it up, Wasm is an assembly language, and WASI is a system interface to run it outside the web. The latter is extremely well explained in this article by Mozilla, I suggest giving it a read for more details. This one is also very nice to read and explains the isolation system it provides, specifically.

The two main points WebAssembly offers are, in a nutshell:

WASI is just a standard, so it has multiple runtimes available. The most popular ones are coincidentally written in Rust: wasmtime and wasmer. Both use the Cranelift backend to generate the WebAssembly machine code, though wasmer seems to also support LLVM and others. Then, the runtime can be used to run the generated .wasm binary as a CLI or a library. This also means that plugins could be written in any language that compiles to WebAssembly, though it is frequently harder than compiling to native code. Many Rust libraries don’t support WebAssembly, so it could be tricky depending on the situation.

The differences between the two runtimes aren’t that big. You can read this wiki article for more details, including examples, but I particularly liked this quote:

The article also includes a not reliable at all benchmark, which can serve us as a way to compare its performance with the native code you’d get with e.g., Dynamic Loading. It estimates that Wasm is a bit less than an order of magnitude slower than native code, and the same applies to memory usage. A more thorough benchmark was done in libsodium that shows better results: Wasm can be just about 3 times slower than native code. Do note that this depends on the runtime that’s being used, and it may improve in the future, as WebAssembly is just 4 years old.

There’s a whole series on how to make a Plugins System with Wasmer here, which will come in handy to know what to expect. The usability doesn’t actually seem to be that good, since by default you can only use integers, floating-point or vectors [15] as parameters when calling Wasm plugins. For more complex types, you have to resort to encoding and decoding via a crate like bincode, although most of the boilerplate can be reduced with procedural macros or a wrapper like wasm_plugin, and this opens up the possibility of using serialization with support for backwards compatibility within Tremor. The last part of the series is the most interesting one, as it includes a real-world example, with a version of the final code in this repository.

All in all, WebAssembly seems to win against Dynamic Loading in terms of security by not needing unsafe at all and including sandboxing by default, at the cost of efficiency. This is up to the managers of the project and what they consider more important.