Challenges in High Performance Robotics Systems

[upbeat music] Right.

Good afternoon, everyone. And really excited to be here today. Really exciting stuff so far. So many models, so many new ideas. And today, I wanna talk about what happens between the controller and the wire.

Now, we have seen so many policies that work that control robots, but again, with that, we need to get that data to the actuators. We need to get that data from sensors and feed the whole system.

And what happens if your carefully crafted policy does not work as expected? Like, is this issue in the policy, or is it in the software system? So today, we look at a lot of instances where the issue will look like it's the policy, but it's actually the software system.

And along the way, we'll try to design a very, uh, small toy robotics robot.

So why this talk? Again, well, robots are complex, so many systems, so many different software components, and yet we're focused on, like, one big question. When things go wrong on the robot, when you don't see that motor move, what's the root cause?

Is the policy that is not giving the command, or is it the software system? And this is a question that I grapple almost every day, and so I wanna talk about what I've seen so far, and how to diagnose these issues on the robot.

So let's go to the buildup. Let's try build a very small, uh, toy robotics general architecture, right? Like, this is what a general robot would look like. You'd have some actuators, a CPU, maybe a hybrid accelerator, and then a sensor.

Perfect. Now, one of the most critical aspects is the communication protocol. So for our, our talk, we'll use CAN. CAN is great. CAN is open source. Everyone can use CAN.

It's cheap, it's affordable, and it has enough data rate and enough compatibility for a lot of components out there. So we'll stick to CAN, and, uh, we'll see how that influence a lot of the design, design decisions down the line.

All right. So let's also start simple with the code. We'll start with receiving the data, giving that to the policy, and basically sending it back out. Nothing, nothing happening, nothing fancy, right?

And let's assume that we have approximately two milliseconds for our policy, and this is what we should expect to see, right? Our loop's running every two millisecond, we are able to see our policy output.

We read data, we send it out. Standard. But as soon as we deployed on the robot, this is what happens. There's a gap. Every two milliseconds, there's a gap.

Wait, what's going on? Well, let's look at the loop again. So at the edge of the loop, we have question marks. We see that we are transmitting and receiving CAN data.

So let's look at the CAN bus. Maybe we'll find some hints there.

Okay, so let's say we have a hundred bits per message, and we have about ten messages, five to be sent out, five to be received. That gives us a total of thousand bits.

And for a CAN bus that's operating at one megabit per second, that's about point one milliseconds per message or one milliseconds for ten message. You can see, like, how even a small number of messages are saturating the CAN bus to the point that the s- loop time, the how much our system takes to run, is on the

same order as the transmission time. And this explains the one millisecond gap. So great. But then what to do about it? It's like, it's almost unavoidable, right? Like, we cannot go around this one millisecond gap.

Well, that's solution number one. You just accept the delay. Hopefully, it's three milliseconds, and that's not too bad. But again, a system would not be high performance if we let that stop us.

So we'll multithread, and we'll pipeline. We'll try to figure out how we can work around that one milliseconds, and see how we can sort of organize our tasks differently to still get that two millisecond loop time.

So here, we'll take a, take a moment to pause and see that, you know, the loop, it has multiple components broken down into three now: TX, RX, and the policy, and we're running the communication in a different thread and the policy in a different thread.

And now we'll see how we'll take this simple building block and stagger it so that we can actually achieve faster loop times.

And this is it. So what we do, we seed the policy the first time. We get some data, we feed it to the policy. But before we conclude the policy, we start receiving the next set of data, and that's for the next iteration.

When the next iteration starts, we tr- transmit the data from the last policy, and we continue resuming the ne- this iteration of this policy. Essentially, we have parallelized our RX and TX, but we're still receiving data for the same policy at the same cadence.

So this is great. We might have solved our problems.

Let's move on. So we deployed the system on the robot, and now we see new problems. Our system is stuttering. Our actuators are making j- sounds like catching up or like we are seeing weird motions on the actuator.

This has to be policy. There's no way this can be software. Well, let's investigate more. Let's get some more data from the CAN bus.

So again, like here, we have our CAN bus again, and we see our CPU, GPU, all our accelerators. And what we'll try to do is get an external transceiver.

These are, again, very cheap, very open source products that you can get anywhere, and we connect it to the CAN bus, and we get data off the CAN bus.

We take this data, we feed it to another host computer, let's say a laptop, and on there, we can run utilities like candump, which will actually give you a timestamp data of what message was seen at what time.

So once we get this raw data off the bus, we can start plotting it. And this is what we should expect, that every two milliseconds, we have a message on the bus that is being sent out.

Right? It should be very nicely spaced, and it should reach the actuators in time. And this, if we see this on the bus, we're really happy. Now, what happens a lot of the times in systems is you'll not see this, you'll see something like this.

Here, we'll see, like between message number three and four, there's almost no gap. What happened there? And between two and three, there's four milliseconds of gap. It's almost like message number three was just late, and four was on time, and because of that, we had this weird, weird jitter where the actuator would try to catch up or

have tried to command, like try to follow two commands at the same time.

Okay, same thing happened with seven and eight. So let's take a deeper look, but first let's try plot this differently. So there's this plot called a cycle time plot, and where what we plot here is the time since last message.

Time since last message is just a way to say like, "Hey, last message came in at two milliseconds interval. This one should also come at two milliseconds, so we should see a straight line around the two millisecond mark."

But here we see some messages jump at four milliseconds, and the one after that comes to zero. This is expected because if a message is delayed, the cycle time for that would be late.

But then for the next one, it would be much closer to zero because that one was not late, and the difference between the last message and the current one is basically nothing.

Okay, so now we characterize the system, we know what's going on, and we can start solving it.

But this is what's going on with the TX side. So let's see. So we missed sending the data and queued it. Why would that happen? Well, policies are not very real time.

At times they can take longer, at times they can take shorter. And what happens if a policy takes longer? Well, you miss the time when you were supposed to send it out.

So all you can do is just queue it somewhere, you can store it. But that cannot be sent out anymore. And when the next iteration comes around, that's when you send both the last message and the current message.

So you'll see two messages just go on the bus at the same time. And this can also happen if our TX and RX threads start desynchronizing. But this is one of the issues that is very commonly seen with like a multi-threaded system, and it's very important to have, uh, synchronization in the systems.

But let's say we do synchronize it, and we are able to fix our TX side. Well, we see some improvement. We don't see that like everything is solved, we see some improvement.

Okay, but now this has to be policy. Our graphs are looking fine, everything is on the bus is fine. This has to be policy. There's no way this systems.

Well, there's one last, one last issue that we have to check, and that is what happens if we desynchronize in the RX side. What happens if a thread is delayed?

Well, now our policy will, will not get the new data and it will work with the last data. And because of that, the output will also be based on the last data.

And so in policy number two or iteration number two, we'll actually have an old command still, like, which is relatively older. And in policy number three, we'll directly jump.

We'll skip one of the data processings. And because of that, we'll see a sort of skip of caching up behavior on the motors, which will sound like almost like a jitter.

Okay, so how do we resolve these two things? Well, there are synchronization primitives. Look, you can with condition variables, semaphores. These are like very low level system things that are widely used in robotics and should be used as well for this TWiSE system.

But again, if these are not available, which is sometimes the case, like we're not working with Linux-based system, we'll work with like a real-time OS or like a microcontroller where we may not have all these semaphores, we can just add padding.

Just have some cushion, right? Like have some cushion so that if some desynchronization happens, you still have the same RX going into the right policy and coming out the other way in like, in a timely manner.

We don't miss messages. Okay, perfect. So this, this, this makes our system fairly robust, fairly high-performing, but there are a few other relative problems which will happen with a system like this, which we should also talk about.

So let's talk about logging. Logging is benign, right? We just log that, hey, that message is coming in. We wanna just log that this is the data that we got, this is the output.

It's fine, right? But if we log too much, at some point, we have to send those logs to disk, and that is very costly. Imagine what happens if your main control loop starts logging and decides just one day that, "Hey, I'm done.

I'll just start putting this on the, on the hard disk." Well, your robot would stay frozen for thirty milliseconds, as we saw on the Raspberry Pi with an SD card.

So, well, that's bad. How do we fix that? Well, we just throw more CPU at it. We just add another CPU, and now all our logging is handled by that third CPU.

Cool. Okay, so now we have like-- we're seeing how multi-threaded is slowly getting baked into the system, how the robot is operating in a real-time deadline guarantee, and how we are able to, like, avoid the pitfalls.

Perfect. Let's talk about something a little more low level again, like microcontrollers. Microcontrollers are fairly simple, and their logging doesn't actually go through a whole disk and file system way.

They just log to some other peripheral. That takes time. In fact, for UART, it can be on the order of millisecond, depending on how much we are logging. So here's an interesting problem.

Let's say we drop a me- packet and we log that, hey, we dropped a packet. Well, that log itself would take enough time that will drop the next packet, and then you will keep drop-- Because you drop the next packet, you log again.

And so basically, just keep logging, and you see a complete blackout on the CAN bus. And it's very hard to debug, like, why am I getting logs and seeing packet drops but no data?

So these are mysterious things that, and in my experience, like it's really good to, like, know about the pitfalls beforehand before we dive in the system and really figure out that, hey, this can also be a problem, just a log statement.

Finally, there's also priority inversion. So in the kernel, in the Linux kernel, there are ways in which data is received by the user process. It's not direct, like it takes a while between the interrupt, the kernel process handling, and then it goes to the user process.

In robotics, we tend to just boost the priority of all our processes so high that we start just blocking the kernel almost. Like if the kernel doesn't run, we won't get the data, but we're trying to get the data, and we're blocking the very thing that will give us the data.

Well, this is inversion in action, and it will see your system again drop out for like seconds almost at a time.

So again, this is something we, we fix by just making sure we know the parts of the pipeline, we fix the right priorities, and we make sure that our whole system as a whole, like it works together well.

So this is how like software and s- robotics have to work together. We have to talk about hardware, the various profiling, the various priority stuff, and actually just take a recap from the top.

So we went over pipeline. We saw how to reduce cycle time beat, how the communication delays. We saw how synchronization can actually cause some unexpected jitter, which are hard to diagnose.

Could be the policy, could be the system. So we wanna make sure that that doesn't happen. Logging strategies, so that we don't block the system while we're trying to tell the user that, "Hey, this is happening."

And finally, priority inversion to avoid starvation. And that's how we start designing high-performance robotic systems, at least on a very basic level. And that's my talk for today, and thank you so much for being here listening. [clapping] [outro music]