RustFest Barcelona – Zeeshan Ali: zbus: Yet another D-Bus library

RustFest Barcelona – Zeeshan Ali: zbus: Yet another D-Bus library


ZEESHAN: Hello? Okay, this works. Hello, everyone. As the MC said, I’m going to present about
zbus. Before I begin, a big round of applause, please,
for the transcriber. They are doing an amazing job and it’s a really
difficult job. [Applause]
So thank you for that. First of all, who am I? My name is Zeeshan Ali, as said. I work for this small company called redhat. I have been working in open source software
for a long time. Nowadays I work with these cloud weird things
in my day job. I’m into flying and I love cats. So background story about this talk, right? It starts with a project called Geoclue. Does anyone who what that is? You, good. It’s a framework for your Linux system. If you have a Linux desktop you probably have
it installed on your machine, if it’s a modern one, and the task of this service is to locate
where you are, so applications, when they need access to your location data, that’s
the service that it needs to talk to. Firefox still hasn’t ported to that. It would be really nice if they do that, if
any one of you works on Firefox. It’s written in C entirely and I have been
the maintainer since a re-write. There was a complete re-write because the
old score was really bad and stuff, and that was 2013 so it has been quite many years that
I have been maintaining it. So I thought: let’s oxidise it. Those of you who don’t know this term, it
means to port it to Rust. But why? Well, it would be singing to the choir but
still: crash reports. I get crash reports all the time. I’m tired of them so I don’t want to deal
with those anymore. Also leaks. Your location is one of your most sensitive
data on your machine. You don’t want to give it to any app or – you
don’t want anything bad to happen to the geodata, this data of yours, so that is another reason,
but most importantly I just love Rust. I don’t do Rust in my day job, at least at
the moment, so in my spare time at least I don’t want to look at C and I want to do something
nice like Rust. So what are the challenges for porting Geoclue
to Rust? First of all, the thing that came to my mind
was this build system called Meson. All the system level services like System
D and related services on your system, they make use of this build system nowadays. Also, all of GNOME and hopeful at some point
maybe KD will use it too if they are not already. I don’t know. And so yes, Geoclue uses it and it has Rust
support in it for a very long time, but doesn’t have cargo support and it looks unlikely there
will ever be any cargo support there. So I thought this would be the biggest challenge,
but we will see later how it will work out. The other thing I thought of was D-Bus, how
do I talk to D-Bus? What is D-Bus? It’s a very efficient interprocess communication
protocol and it’s very popular on the desktop and in embedded systems as well. As I mentioned, the other projects like System
D and related services and GNOME in the same two bubbles, D-Bus also exists and it’s very
popular and I don’t see why it wouldn’t be. It’s very awesome and very efficient. There is a crate, dbus-rs but it relies on
libdbus which is a C code so it relies heavily on C code and also it is not shown to be a
reliable library. There are other issues but I still decided
to use it and not just use it but I also contributed in terms of issues and patches and stuff and
I thought I will just use it. Until Rust GNOME Hackfest that happened in
May this year, and I started doing it, an amazing part turned out to be much, much easier
than I thought. It’s a work-around still but an easy work-around,
I can call Meson from D-Bus easily. It turns out, I wanted to use code generation,
it was very, very complicated. Also, sorry, I have to convey a lot of information
here so if you don’t understand something, that’s fine, I’m doing a bad job, please grab
me after the talk and I will explain further to you. I will be happy to you. That’s okay. D-Bus crate from scratch? I thought maybe that’s an option. So I was like: how hard can it be? [Laughter]
So let’s see what’s involved first. I looked at the pack, there is a D-Bus pack,
a one page which explains everything. It’s a really nicely written pack in my opinion
so I started the low level and lowest level is just message passing on any kind of socket
or any kind of medium. It’s medium agnostic actually, the protocol
and there’s a wire format that you have to implement to be able to talk to D-Bus. It’s also called GVariant because D-Bus was
implemented by non-folks and the main library made use of that from the very beginning and
they realised that the wire format of D-Bus, it can be used in a very generic way so the
API is provided to use it in a generic way outside of D-Bus as well so it could be like
a database and people use it that way. What that is, it’s mostly defining data types
and their encodings. By encoding I mean you have to have alignment. All the data is aligned. If you don’t know what alignment means, again
grab me after the talk and I will explain it to you, what that is all about. D-Bus has natural alignment for the most part. 99% of it. There is one exception. I will talk later about that. Every data type that D-Bus defines has a signature,
which is a string which is a suggested designation for that particular data type. We will see the use of that in a bit. So we start with the basic data types. What are they? They map exactly to the Rust data types. It’s a subset of Rust data types actually. The only difference is that they are encoded
differently, of course, because they need to go on a socket, not in a programming language. Then there’s container data types. Array maps nicely to a vector or array itself. Structure, it’s a bit weird to have like a
dynamic structure with maps actually to struct in Rust, but of course it’s more dynamic so
we can’t just use a struct as there is in Rust. And dict is just short for dictionary and
that maps nicely to hash maps and most importantly there’s another data type called variant which
is basically just generate data. So if you want to put – you can put any of
the data types I mentioned before in this variant and what the variant contains is the
signature of the data and the data itself, so if you want to transfer data in a very
generic way, this is the way and this data type is used a lot in D-Bus. On a high level, you have objects. When you talk to the D-Bus you usually deal
with objects that are exposed on these specific paths. These are shown here, the object paths, and
the way you talk to them is through interfaces. This is a really horrible example, sorry for
that. There’s a one-to-one relationship here between
the objects and the interfaces, but it doesn’t have to be. Any object can implement any number of interfaces
and what that means is that, if they implement an interface, they need to provide specific
API of that interface. You can have methods and they have to have
the data types that I said before. Signals is just a method but in the reverse
direction. If the service needs to notify apps of some
change, they use these signals, and they also have parameters, but only I think in parameters
– also out parameters. We are not sure right now. And of course they can have properties that
they expose and they can be read only, they can be read write. So I thought actually it’s not that hard after
all. Am I going too fast? Okay. So after three days, which includes the Hackfest
and I had an excuse to not do my work-work, I could do Rust and I started implementing
it first as a hack and what I could accomplish in three days, the first day I established
a connection to the D-Bus on the service, and I called a method. That felt so great. Within three days I could do this. So I was like: yes, let’s do this. It doesn’t sound too hard! So zbus was born. Don’t ask about the name. It’s a cool name and you have to accept it. [Laughter]
So let’s start with the hard part, the low level, which is called the GVariant as I mentioned
before, which is dealing with the data types. So I came up with this to represent the data
type that can be encoded to and encoded from D-Bus wire format. It’s pretty simple actually, I think. Encoding is – I tried to make it very efficient. Encoding, I can’t make it efficient because
you have to copy data because the D-Bus encoding is different from the Rust encoding so there’s
no way out of this as far as I know. If you know a way out of it, please do tell
me. I would love to know that. But decoding, I made it very efficient so
that you don’t have to copy the data. Only for container types you have to allocate,
but even that only the container, not the actual data contained in it. Signature, you might see there’s a constant
signature and then there’s a signature. For simple data types it’s exactly the same,
the screen that you represent, it is one corrector, but for complex data types, the container
types, they have to also specify what is included in them so it’s more dynamic, so it’s not
the same. The constant – actually, I have a pointer! But it doesn’t work on this, does it? No. Anyway, so this signature, the constant that
you see on the top and the signature method, they don’t always have the same output. So basic data types as I mentioned before,
the same as Rust, very easy to implement. I did that. Container types were the next. Arrays, as I said, vector, only thing is that
I had a constraint that it can only have the variant type in it. All the other types, the D-Bus support, not
anything else. Structures: yes, I have to create another
data type for that. Captured as simple as I could. You have fields on the structure and you can
create them and then you can use that, and then you implement variant type for it so
that you can encode and decode structures. Of course, the most important one is variant
which as I said contains a signature of what it contains and the data in encoded format. Now, this is the important bit that, it keeps
already in encoded format. It doesn’t encode when it’s needed, but already. I thought that would be more efficient. We will see later how much trouble that gave
me. So yes, this is how you will use variant in
isolation without D-Bus. There is no D-Bus here so this is the type
I came up with. This is my unit test which just shows you,
you create a type that is D-Bus – which implements for a type and then you create a variant from
that and then you can do different tests on it. You can get the data back from it and you
can – once it’s encoded you can also decode it and stuff. Yes. I lied before when I showed the variant type
frame. It has lifetimes on it and I did it to be
as efficient as possible and I was successful in making it efficient, but at the same time
it made life much, much harder. Each time I change anything, lifetimes come
in the way and I have been bugging lots of people, especially Florian over there, about
how to deal with lifetimes. Oops, sorry, I
pressed the wrong button. So back to D-Bus. So this is what the D-Bus API looks like. You just create a connection, there’s two
kind of common connections that you talk to, one is a session connection which is for the
user and then there is the system connection which is used by, for example, Geoclue for
system services, and then you just call a method on it. You specify the – there’s the service, the
path, the interface and the method and parameters which are zero here, so I pass none. Then you get a reply and then you get the
first out parameter and it’s the type of string so I could get the string and display it. All done with variants, right? It’s all done? That’s what I thought and I thought: like,
yes. I didn’t talk about an additional part which
I implemented really late but it’s kind of implemented already, and I thought: I’m done
with variants. But I added new test cases for the dictionary. Actually, these test cases were more complex
and I was like: why is it failing? I thought maybe I had made the wrong test
case or something else wrong with it. And then I had this face. This is an actual picture of me actually. That’s how I felt. [Laughter]
Variant alignment is all wrong. It’s completely wrong. Why? D-Bus has a few strange rules. Turns out, D-Bus is not that easy after all. [Laughing] So first of all alignment of data
when you encode it is based on the whole message, it’s not individual levels. So if you put an integer in a structure the
alignment is not based on the parent structure but the whole message when you put all the
data together, which is okay on its own, but that’s not the only – and also if you see
at first what I came up with, there is no way with this API, I don’t pass to encoder
how many bytes were before, so this API cannot accomplish this but I modified it soon after
and added 10 bytes before so then the encoder knew how many bytes were before so it knew
how much padding to add for alignment and stuff like that so that was an easy problem
to solve on its own, but variants and contained value do not need alignment, which is okay
but – surprising, but okay, we will deal with this. It’s no alignment, better, but the problem
is its grandchildren do. If you have, for example, a structure in a
variant that structure does not need padding nor alignment but all the fields in the structure,
they do, so how do you pass that along, all the information? As I showed you before, variant already keeps
things in encoded format so when you create a variant in isolation then you put it in
a message the encoding is not correct and that’s what was wrong in my whole thing, so
it turns out my whole structure, the whole API I built in three months was fundamentally
wrong. So I have been on this for a month now. Mind you, I only work on this in my spare
time which is very limited, so one month is not that long in reality. So I still have not solved this problem. I came up with multiple solutions. They all had some problem in the end after
all and I couldn’t solve it. Now I have a solution but before I implement
it I have to kill the lifetimes from the very end. Because that makes – when I said I came up
with multiple solutions and many of them failed because of this lifetimes because I just couldn’t
satisfy the lifetimes, so the lesson I learned from all this practice is that efficiency
is not a religion. I was being so focused on making it very efficient
that I wasn’t making it more efficient than I would be in C. In C, I would be using something
called GObjects from GLib API and they use reference counting all the time. That’s what they are based on, and their reference
counting is by default atomic so it’s much more heavier than the RC reference counting
type in Rust, so why not use it? So with our team I would really like to make
use – to share data between different points of the code and then it will work out. I have a solution. It will work. If not, I’m in this conference for three more,
four more days, so I will grab someone to help me. That’s my hope at least. Anyway, once I solve that problem, which I
will, looking forward, I want to separate out the variant crate into its own crate. As I said, it’s useful in a generic way and
there is one application in Rust that the maintainers said yes, if you make a crate,
I will use it, so yes. I have to do that. Then in D-Bus, we have an API for receiving
messages, I have to implement signals which should be really trivial actually. “Should be”, in quotes. Async, I need to add asynchronous API, and
I’m hoping that actually the API will just be asynchronous by default and hoping to use
the new async STD, or how do you pronounce it, I don’t know, crate. It looks pretty awesome, that crate, and also
it’s good that I was stuck on the variant part until now so that all the async stories,
all in stable, so I can just use the async from stable. High level API, then I need to add. That shouldn’t be extremely hard but I need
to be careful in creating a good API. Fortunately, GLib has a really good high level
API and I can copy that; not copy, but I can learn a lot from that so I don’t need to figure
out a lot of things. Code generation: most of the D-Bus code that
I see out there, they use code generation. You give it like – you can define your D-Bus
API in XML which people do and you can give it to code generator and it generates the
code so I need on implement that, and maybe also macros for making it super easy. If you don’t want to do code generation, some
people, I don’t know why, but they don’t like code generation, so sure for them there should
be some macros. And a lot more of easy stuff. Yes, that’s it from me. If you have any questions. [Applause]
No questions? Everything was clear? Yes, there is a question. Do you get a mic? It’s better.>>Tech team, please enable microphone 2.>>[Question off mic]. ZEESHAN: Yes, when I said receiving messages,
that’s the server side, yes. That’s coming. I need it myself for Geoclue. That’s the main use for me, so it will definitely
be there. Someone else?>>It’s working, so I have a magic hand. So you were saying that performance shouldn’t
be an issue, but did you compare it to native implementation in GNOME or other implementations? How does it fare? ZEESHAN: I haven’t done any actual comparisons
but I could see that I’m not allocating anything or copying anything, so yes, I made sure of
that, so that’s the way I know it’s as efficient as it would be, but no, I haven’t done any
real comparisons. Once I’m ready, once it’s done, then I can
look at that.>>Any more questions? No? One more.>>Could you explain alignment? ZEESHAN: [Laughing] So some CPUs require you
to start your messages and to be able to read specific data types from on a particular boundary,
so the memory address has to start with some multiple – or it has to be a multiple of 8
bytes, something like that. That’s as far as I know, so that’s why you
need alignment. If you generate protocol that’s CPU or architecture
agnostic you need to make sure of the padding and the alignment. The way to align is that you do something
called padding, and padding means you just add zero bytes before the data starts so make
it so that that data actually starts at a boundary that is defined for that data type. So, for example, as I said, natural alignment
in D-Bus, what that means is that, if it’s a 4 byte integer it has to be aligned with
4 byte value, so you have to have 4 bytes of padding if it doesn’t start at a multiple
of four address. Things like that. Okay? So thank you so much for attending and I hope
you understood some of it at least. And if you don’t, as I said, let me know. I’m very happy to explain things to you. I’m wearing the green one, so you can do that. [Applause]

Leave a Reply

Your email address will not be published. Required fields are marked *