Discreet Log #18: libCwtch in Rust

LibCwtch is our interface (API) library to Cwtch, which was started over 3 years ago as a Go language project. Since then we have started using Flutter and Dart to build our front end UI which meant we needed a more language agnostic interface to Cwtch so we created libCwtch, which is some wrapper functionality around Cwtch and it compiles to a C library that any language can access via FFI. The first language binding we made for libCwtch was Dart which is currently built into our cwtch-ui. At some point we’d like to find the time to break the Dart bindings out into a standalone Dart package that anyone can use to access libCwtch from Flutter/Dart.

Since starting work with Go 3 years ago our interest in working in Rust has been growing so it was the logical next language we would look at to create bindings for.

To get started we first had get a Rust FFI interface to libcwtch. Rust has built in FFI functionality in std::ffi. The first step, a simple proof of concept calling StartCwtch() in the library was easy enough to write.

extern crate libc;

#[link(name = "Cwtch")]
extern {
    fn c_StartCwtch(app_dir: *const u8, app_dir_len: u32, tor_path: *const u8, tor_path_len: u32) -> u32;
}

pub fn start_cwtch(app_dir: &str, tor_path: &str) -> u32 {
    return unsafe { c_StartCwtch(app_dir.as_ptr(), app_dir.len() as u32, tor_path.as_ptr(), tor_path.len() as u32) }
}

As you will have gleaned from reading other Discreet Log posts from us, we like automation. Manually writing a full Rust copy of the interface methods exposed in libCwtch and having to maintain that seemed like a lot of hassle with room for mistakes to easily creep in. Thankfully Rustcean’s appear to agree and, with a quick search, I found Bindgen which takes a .h file and automatically creates Rust functions for each call exposed in the API. A dynamic library and a C header file being the outputs of libcwtch-go’s compilation (go build -buildmode c-shared -o libCwtch.so), we were free to proceed. This left us with an auto generated .rs bindings file with entries for each API call looking like:

extern "C" {
    pub fn c_StartCwtch(
        dir_c: *mut ::std::os::raw::c_char,
        len: ::std::os::raw::c_int,
        tor_c: *mut ::std::os::raw::c_char,
        torLen: ::std::os::raw::c_int,
    ) -> ::std::os::raw::c_int;
}

Which was quick and easy to get to, but a keen observer might note, this is not the most easily usable interface from more native Rust. The very non typical c_char data types for strings, their lengths managed separately (library design choice), and memory management requirements inherited from using a C library in an otherwise managed language (Rust) all added up to this auto generated interface being not very user friendly.

To create a more native Rust interface from this, we would still need wrapper functions for each API call, so the Bindgen automation only got us half way. However, one of the many reasons I’ve been interested in Rust is that it’s doing some work to resurrect and trying to normalize what I would have over a decade ago as a Lisp fan called “metaprogramming”. Many languages have some “macro” facility but most try to hide or pull it out only for the most special of circumstances. It has been a long while between learning Lisp and now since I’ve seen another language try to suggest macros as a normal everyday solution to some problems. Encouraged by this aspect of Rust and its community, I took a look at automating generating Rust native wrappers for the Bindgen generated functions with a Rust macro.

Some functions would be straight forward to handle, with no arguments, or only integer arguments, the generated wrapper function would be pretty straight forward, nearly one-to-one. How ever if some of the arguments were strings, we needed to convert from Rust &str to c_char and their lengths, and manage the memory behind that. Additionally, if we were getting a string result from the function call, we needed to copy the data out, and hand the actual returned memory back to the library. We cover in detail the rules around memory management when using libcwtch in MEMORY.md in the code’s repo. With all that in mind, we created the following Rust macro

// c_bind handles setting up c string arguments and freeing them
// c_bind!( $fn_name ( [ $string_args ]* ; [ $non_string_args : $type ]* ) $c_function -> $return_type? )
macro_rules! c_bind {
    // macro for returnless fns
    ($func_name:ident ($($str:ident),* ; $($arg:ident: $t:ty),*) $bind_fn:ident) => {
        fn $func_name(&self,  $($str: &str, )* $($arg: $t, )*) {
            $(let $str = c_str_wrap::new($str);)*
            unsafe {
                bindings::$bind_fn($( $str.raw, $str.len, )* $($arg,)* );
            }
        }
    };
    // macro for str returning fns
    ($func_name:ident ($($str:ident),* ; $($arg:ident: $t:ty),* ) $bind_fn:ident -> String) => {
        fn $func_name(&self,  $($str: &str, )* $($arg: $t, )*) -> String {
            $(let $str = c_str_wrap::new($str);)*
            unsafe {
                let result_ptr = bindings::$bind_fn($( $str.raw, $str.len, )* $($arg,)* );
                let result = match CStr::from_ptr(result_ptr).to_str() {
                    Ok(s) => s.to_owned(),
                    Err(_) => "".to_string()
                };
                // return ownership of string memory and call the library to free it
                bindings::c_FreePointer(result_ptr);
                result
            }
        }
    };
    // macro for value returning fns
    ($func_name:ident ($($str:ident),* ; $($arg:ident: $t:ty),* ) $bind_fn:ident -> $bind_fn_ty:ty) => {
        fn $func_name(&self,  $($str: &str, )* $($arg: $t, )*) -> $bind_fn_ty {
            $(let $str = c_str_wrap::new($str);)*
            unsafe {
                let result = bindings::$bind_fn($( $str.raw, $str.len, )* $($arg,)* );
                result
            }
        }
    };
}

Which does look a bit unwieldly, but it left us in a position to rapidly wrap the entire set of libcwtch API calls with single lines such as

    c_bind!(start_cwtch(app_dir, tor_path;) c_StartCwtch -> i32);
    c_bind!(send_app_event(event_json;) c_SendAppEvent);
    c_bind!(get_appbus_event(;) c_GetAppBusEvent -> String);
    c_bind!(create_profile(nick, pass;) c_CreateProfile);
    c_bind!(load_profiles(pass;) c_LoadProfiles);

and etc. Between Bindgen and out c_bind! macro, writing a Rust interface to libcwtch became as simple as naming the functions and their arguments and returns types. There are some simple usage caveats, such as all &str arguments needing to be listed first, which conveniently our library already adhered to.

I was left with a fairly enthusiastic impression after this first outing with Rust Macros. I did run into some bumps (currently they don’t support variable name concatenation outside of the nightly version for instance) but I was able to work around them with some creativity. Also as it was my first outing with Rust macros I know that some of the bumps I ran into might be because of the limits of my understanding and newness to Rust macros. For instance, there is still a whole other section/level to dive into and learn, proc macros, which have full token tree manipulation abilities like I was used to from the S Expression language Lisp and had previously expected could only be found in that style of language. I am excited to find a new language that is trying to bring metaprogramming once more into the mainstream and absolutely plan to continue learning more and improve as macros and metaprogramming may be one of the most powerful tools in programming (old and modern).

With the macro written, generating Rust native bindings for libCwtch was basically done! The Rust bindings are now published as a Rust Crate as libcwtch with the code at git.openprivacy.ca/cwtch.im/libcwtch-rs. You can see the how trivial the results are to use in our example echobot which is a simple straightforward Rust implementation of a Cwtch echobot.

As always, we are a small team at Open Privacy Research Society, and sometimes the research takes the form of just getting up to speed on what the state of the art is so that we can do things like bring you Rust bindings for our Go/C library. If you like this work or any of our work and want to support us donations always help and are always appreciated.