Device Handler

Table of Contents

• 1. Overview
• 2. Hotplugging
  • 2.1. Detecting New Connections
    • 2.1.1. Lowcar Verification
    • 2.1.2. A Note On Polling
  • 2.2. Detecting Disconnected Devices
• 3. Lowcar Communication
  • 3.1. Relayer
  • 3.2. relay clean up
  • 3.3. Sender
  • 3.4. Receiver
• 4. Opening a Connection
  • 4.1. Opening a Serial Port (Arduino)
    • 4.1.1. open()
    • 4.1.2. termios
  • 4.2. Opening a Socket (Virtual Device)
• 5. Design Decisions
  • 5.1. Threads and the Sender, Receiver, Relayer
  • 5.2. libusb
• 6. Glossary

1. Overview

The device handler has two core responsibilities. It acts as

1. the Runtime hotplugger, and
2. the interface between shared memory and the Lowcar devices.

By “Runtime hotplugger”, we mean that the device handler is responsible for detecting any new lowcar devices being connected to the Raspberry Pi. Likewise, it is responsible for detecting when a device is disconnected.

As the shared memory and Lowcar interface, dev handler reads any commands from shared memory, encodes it into a lowcar message (see Lowcar Protocol to learn how we encode/decode messages), and sends it to the lowcar device over serial. It also decodes device data messages sent over serial by the lowcar device, and writes the device data to shared memory.

In short, the device handler makes sure all data from the devices are sent to shared memory, and all commands from shared memory are sent to the device. Shared memory is an abstraction barrier; the device handler doesn’t need to know where the commands come from, nor what happens to the device data after we write it to shared memory.

If you just wanted to know what dev handler is supposed to do, you can stop reading here! If you want to really understand how it’s implemented, the rest of this page explores it in detail.

2. Hotplugging

All of PiE’s Arduino devices are connected to the Raspberry Pi via USB. So what happens if we try to plug in a mouse? What about an Arduino with code students wrote themselves? Runtime as a system should communicate with only Arduinos programmed by team Runtime (i.e. “Lowcar” devices).

2.1. Detecting New Connections

On Unix systems, connected devices are treated as files. All (and only) Arduino devices appear as files with the path /dev/ttyACM*, where the asterisk is a non-negative integer. The device handler scans for the existence of file paths between /dev/ttyACM0 and /dev/ttyACM31. (At most 31 because Runtime supports 32 devices). Just by scanning these 32 files, we can detect connected Arduino devices.

Note that virtual devices will appear as a file with the path /tmp/ttyACM*. We poll for both Arduinos and virtual devices. Verification and communication with Arduinos and virtual devices will be the same. The only difference is the way we open a connection. See the sections below to see how we do this.

How can we be sure that we can ignore /dev/ttyACM32 and above? For brevity, in the remainder of this wiki page we’ll say “slot i” is occupied if and only if /dev/ttyACMi is an existing file. When we connect the first Arduino to the Raspberry Pi, it will occupy slot 0 (/dev/ttyACM0 will be an existing file path). It will be the only slot occupied, and no other file path with the prefix /dev/ttyACM exists. When an Arduino device is connected, it is assigned the lowest-numbered unoccupied slot. So if the currently occupied slots are 0, 1, and 3, then the next plugged in Arduino should occupy slot 2. Since Runtime supports up to 32 devices, we need to check only slots 0 through 31.

2.1.1. Lowcar Verification

We now have a way to detect up to 32 Arduino devices, but we also have to verify that they are all specifically PiE Arduino devices. We don’t want students to write their own Arduino devices and connect them to the robot. We solve this problem by using the Lowcar protocol. It is very unlikely that a non-PiE device would send and receive messages conforming to the protocol we decided on ourselves.

We follow these steps to verify that a connected Arduino belongs to PiE (i.e. a lowcar device):

1. Device handler detects a new device connected (via scanning the “slots” mentioned above).
2. Device handler sends a PING to the device.
3. If the device is a lowcar device, it will read the PING and respond with an ACK.
4. Device handler confirms that it receives the ACK from the device. From the device handler’s perspective, if it sends a PING and doesn’t receive back an ACK immediately, then the device is considered not Lowcar and the device handler ignores it.

In summary, in order to properly detect a new lowcar device, we scan for existing files between /dev/ttyACM0 and /dev/ttyACM31. If the file exists, it is a connected Arduino. We write to that file (that device) a PING and wait for an ACK. A proper Lowcar device will respond with an ACK and dev handler can begin transferring shared memory data with it.

2.1.2. A Note On Polling

When scanning for files /dev/ttyACM0 through /dev/ttyACM31, we need to keep track of which devices we have seen before. For example, let’s say we found a device at /dev/ttyACM0 during one of our scans. We spawn threads to communicate with it, and continue scanning. We see /dev/ttyACM0. How do we know whether 1) We already have threads handling this device, or 2) The device disconnected since our last scan, and we should spawn a new set of threads?

We can do this by using a 32-bitmap. For example, if we see slots 0, 1, and 3 are occupied, we’ll turn on bits 0, 1, and 3 in the bitmap and proceed with the PING/ACK exchange with those three devices. Let’s say when we scan those slots again, we see 0, 1, and 2 are occupied. In this case, /dev/ttyACM2 is a new device and /dev/ttyACM3 disconnected; we know /dev/ttyACM2 is new because our bitmap previously had bit 2 off. We will turn on bit 2 and proceed with the PING/ACK exchange with only device 2. We do not need to do anything about /dev/ttyACM3. Bit 3 was turned off by another thread when we properly handle a disconnect with the device, which will be discussed in the next section.

2.2. Detecting Disconnected Devices

We define “disconnect” in three ways. Here's how we handle each case.

• Lowcar device is unplugged from the USB port:
  • Relayer constantly polls that the file exists, so when it doesn’t exist, we clean up.
• Arduino doesn’t send an ACK:
  • We clean up.
• Arduino is unresponsive:
  • Receiver always records the timestamp of the most recent received message. Relayer compares this timestamp with the current time. If they differ too much, then we clean up.

See the next section for how we clean up.

3. Lowcar Communication

When a new device is connected, we open a connection with it. If the connection is successful, we initialize a relay struct. This struct is used to bookkeep critical information to coordinate a connection with the device. This includes information to identify the device and synchronization data structures. After the struct is initialized, we spawn three threads: the "relayer", the "sender", and the "receiver". Each thread has a reference to the same relay struct. The sender and the receiver threads are blocked until the relayer broadcasts to do work.

3.1. Relayer

The relayer verifies that the Arduino is a Lowcar device, connects it to shared memory, then broadcasts to the other two threads to do work. The Relayer will then continuously try to detect if the device is disconnected, and clean up accordingly.

3.2. relay clean up

When we decide to stop communicating with a device (See Detecting Disconnected Devices), we need to properly clean up. Everything that needs to be cleaned up can be referenced through the relay struct. We need to

1. Cancel the Sender and Receiver threads and join them,
2. Disconnect the device from shared memory,
3. Close the serial port,
4. Mark that the device is disconnected by unsetting the bitmap (See Detecting New Connections for how we use it),
5. Destroy mutexes and conditional variables used by these threads, and
6. Free the relay struct

3.3. Sender

After the Sender is broadcasted to work, it continually

• Checks if there is new data in shared memory to write to the device (i.e. a command). If so, package a DEVICE_WRITE and send it,
• Sends a PING at the specified PING_FREQ time interval, and
• Checks if this thread was asked to be cancelled. If so, cancel this thread. Note that we disable thread cancellation at any other point, to ensure that we won’t get cancelled while sending a message.

3.4. Receiver

After the Receiver is broadcasted to work, it continually tries to receive a message. If it receives a valid message, it will update the timestamp of the most recent message received. If the received message is a

• DEVICE_DATA, it will parse it and write it to shared memory.
• LOG, it will preface the message with the device’s name and uid, then log it to the Runtime logger.

4. Opening a Connection

4.1. Opening a Serial Port (Arduino)

This section addresses some of the nitty-gritty details of how we open a serial port, which is not important to learning how dev handler works. These options are more of a “set and forget” thing, and can confuse you. If you’re curious, read on!

4.1.1. open()

Opening a connection with an Arduino is easy on Linux, because the serial port is simply abstracted as a file. Reading and writing to this file will communicate with the device. We open the file /dev/ttyACM* as we would with any other file with open(). We pass in two flags: O_RDWR and O_NOCTTY. The former allows us to read and write.

O_NOCTTY is a bit more complicated, and involves the concept of “controlling terminals”. In brief, when you execute a process from your terminal (ex: ./dev_handler), the terminal that you executed it from is declared as the controlling terminal. If this terminal were to be closed, it sends a SIGHUP signal to the process which kills it.

Runtime systemd (See systemd wiki) spawns dev handler (among others) in the background, so there is no controlling terminal for this process. Interestingly, an Arduino device can act as a terminal! When systemd’s dev handler calls open() on the Arduino (/dev/ttyACM*), dev handler says, “I don’t have a controlling terminal, so let me make this Arduino my controlling terminal.” As a result, when this Arduino gets removed (i.e. the terminal is closed), dev handler receives a SIGHUP signal and it ends! This means if we disconnect a device from systemd-spawned dev handler, dev handler will “crash”. Adding the O_NOCTTY flag to open() will prevent the opened terminal from becoming the controlling terminal. Note that this flag is relevant only for dev handler when it is spawned by systemd, not when you enter ./dev_handler in your terminal. Also note that when a process already has a controlling terminal, another terminal cannot “steal” this status and become the new controlling terminal.

The O_NOCTTY flag was added in Pull Request #143, which solves issue #145 if you’re curious.

4.1.2. termios

After we open a serial port, there are many options that we should set in order to properly transfer data with the Arduinos. Refer to the official documentation at https://linux.die.net/man/3/cfsetspeed for more in-depth information.

Option	Value	Explanation
Baud rate	B115200	We expect to send and receive 115200 bits per second over serial. This same baud rate must be set in the Arduino code with Serial.begin(115200)
c_flag	8-N-1	See Wikipedia https://en.wikipedia.org/wiki/8-N-1. This is a configuration for how to send bytes over serial. This is a standard and is the default configuration for Serial.begin().
cfmakeraw()	N/A	This is to address issue #45. Without this option, the terminal does special processing on certain characters. For example, the byte 0x0D in ASCII is carriage return (\r or CR). We were observing that when we send this character, it would be received as 0x0A, which is the newline (\n or LF) in ASCII. We want full control over our bytes, so we disable this extra processing.
VMIN/VTIME	Variable	See http://unixwiz.net/techtips/termios-vmin-vtime.html for a short article (specifically the last section). At first, we initialize VMIN = 0 and VTIME to TIMEOUT milliseconds. This makes it so that when we read() for an ACK from the Arduino, it blocks for at most TIMEOUT milliseconds. If the read() finishes blocking and we haven’t received an ACK, it’s safe to assume that Arduino is not a Lowcar device. After we have verified that we have a Lowcar device, we change VMIN to 1. This makes it so that read() will block until there is something to read. This is desired because we always want to be ready to read data from a verified Lowcar device.

4.2. Opening a Socket (Virtual Device)

We can read and write to a virtual device by reading and writing to its UNIX socket, which appears a file with the path /tmp/ttyACM*, where the asterisk is a non-negative integer. This is very much like an actual Arduino, except an Arduino appears in the directory /dev instead of /tmp.

To establish a connection with the virtual device, we can connect to the socket like any other socket client would.

Luckily, we don't have to set as many options as we do with serial ports. We just need to replicate the behavior of VMIN and VTIME. That is, we want the initial read() for an ACK to block for at most TIMEOUT milliseconds. This is done by using setsockopt() with the SO_RCVTIMEO option. After the virtual device is verified (i.e. we get an ACK), we set the timeout back to 0, indicating that read() operations should block until we read something.

5. Design Decisions

This is a short section explaining a couple of the design decisions that came up while implementing the dev handler. This may be useful in the event that the dev handler gets reworked in the future.

5.1. Threads and the Sender, Receiver, Relayer

Since we want to be continuously polling for devices, the actual communication with the devices should be done in separate threads. We also want each device to have its own thread(s) so that we can communicate with them all simultaneously. Each device deserves a thread each for sending and receiving because we don’t want situations where reading and processing large DEVICE_DATA messages slows down packaging and sending DEVICE_WRITE messages (or vice versa). The Relayer was added to coordinate these two threads, complete the initial Lowcar verification, and do necessary clean up when the device is disconnected. We have considered removing the Relayer for efficiency (issue #85), but we decided against it in favor of the nice separation of tasks between the threads.

5.2. libusb

We spent a lot of time figuring out how to send and receive data from the Lowcar devices. libusb seems to be a popular library to communicate with USB devices. This library was our choice for a while, but we had problems with receiving messages. Furthermore, it requires obtaining a lot of information about devices we want to work with (ex: bus descriptors, endpoints, port numbers). We found that we can simply read() and write() to the file /dev/ttyACM*, so we stopped trying to get libusb to work. Let’s keep it simple!

arduino-serial, the GitHub repository that we referenced for Arduino communication can be found here.

6. Glossary

• Arduino/Lowcar
  • Arduino: A hardware and software company that develops microcontrollers. “Arduinos” can also refer to their devices.
  • Lowcar: A subset of Arduino devices developed by PiE staff (i.e. Team Electrical) that students can attach to their robots. The code flashed onto these devices is developed by Team Runtime.
  • Lowcar Protocol: The “language” that we use to communicate with a Lowcar device. (Ex: Dev handler sends PINGs to the device and expect an ACK back.) To learn more, see the Lowcar Protocol Wiki.
• Threads
  • Sender: A thread that sends bytes to a specific Lowcar device
  • Receiver: A thread that receives bytes from a specific Lowcar device
  • Relayer: A thread that confirms an Arduino is a Lowcar device, broadcasts to the Sender and Receiver to start work, and properly ends communication with the Lowcar device if it is disconnected or times out.
• Polling
  • /dev/ttyACM*: The file that appears when an Arduino is plugged into a Unix machine. * is some non-negative integer. When an Arduino is plugged in, * will be as low as possible, avoiding duplicates.
  • Polling: from Wikipedia, it’s the action of “actively sampling the status of an external device”. Dev handler continually polls for the existence of /dev/ttyACM* files to see if any new Arduinos are connected.
  • Serial Port: The medium through which we send and receive data with an Arduino. Writing to and reading from a /dev/ttyACM* file is writing to and reading from a serial port.