Design Principles

The architecture of the farm is designed with the following goals:

  • Interactive and CI access to devices
  • Hardware on desk experience
  • Controlled sharing of devices
  • Good reliability and scalability
  • Security isolation between users / devices
  • Network isolation between users / devices
  • Agnostic to device types and their software
  • Agnostic to the methods of hardware control/access
  • Lightweight farm management software

Typical use-cases that are catered for include:

  • Providing developer access to remote devices (e.g. when devices are limited/delicate/expensive/dangerous/dependant) for development of full system software
  • Providing ‘hardware-in-loop’ automated testing for quality assurance
  • Providing temporary access to hardware (e.g. for evaluation)
  • Providing access to a wide range of reference boards
  • Implementing KernelCI, LAVA and similar solutions

The following subsections describe in more detail the architecture choices that have been made:

Avoid Containers and Abstractions

When developing software for embedded devices, its always more effecient to have the hardware on your desk than access to remote hardware. Besides the age-old excitement of seeing an LED light, the ability to see, smell (on unfortunate occasions), touch and manipulate the device can really make a difference to progress.

The nearest we can get to this experience with remote hardware is remote access (SSH, TeamViewer, etc) to a machine that has the hardware connected to it, and additional peripherals (USB relays, etc) that can be used to sense and manipulate the device. However, this solution doesn’t scale well with multiple users and devices. Often users trample over each other, access to USB peripherals (such as USB serial ports) isn’t well structured, board sharing is uncontrolled and administration of the farm is resource intensive.

A common solution to this is to abstract sensing and manipulation behind a framework - for example if you want access to the serial port of your device you run a farm-specific command that gives you a console. However this puts a burden on the farm software to abstract each interface (limiting what you can do), this adds friction and complexity. When engineers develop low-level software, they may want to rely on third-party or vendor provided tools that expects access to the underlying device nodes (for example when flashing via USB), however this isn’t often supported. When engineers face problems with their hardware, such abstraction can be a distraction, especially when the issue may be caused by the abstraction. Abstraction based interfaces can add friction and are best suited to narrower use-cases.

Another solution is to use containers, with the farm controller software passing through requested devices to the container. This stops users trampling over each other and controls access to hardware. Unfortunately software such as as libusb enumerates devices based on output from /sys/bus/usb resulting in software attempting to access devices not shared with the container. Furthermore, many devices that are flashed via USB often change their USB device ID, we’ve found that if the farm controller software doesn’t passthrough the new device quick enough programming may fail. In order to passthrough devices and allow user access to dmesg (to debug their issues) privileged containers are needed, however the security isolation isn’t as strong as it could be.

Use VMs and PCI USB Passthrough

To avoid the challenges of containers and farm abstractions, the TGP Device Farm does the following:

  • Provisions per-user VMs
  • Provides access to devices via PCI USB passthrough
  • Provides lightweight farm control software that allows users to request and release access to devices

The use of per-user VMs provides the developer with the experience of accessing a remote machine that has device hardware directly connected to it. As it’s a per-user VM they have full control over its software, allowing them to install any development tools needed (which is often debugging tools or vendor flashing tools).

All sensing and manipulation of hardware is performed via USB (e.g. power control, access to serial console, access to USB relays for pressing buttons etc). By using PCI cards with per-port PCI-USB controllers, its possible to use PCI passthrough to passthrough a specific USB port (and thus hub which is connected to device sensing and manipulation peripherals) to a users VM. As the USB peripherals are passed through to the VM at a PCI level, the end-user has full access and control - thus making it difficult for ‘farm issues’ to cause any friction. Furthermore the user can make use of dmesg and similar to identify any issues.

This approach provides a ‘hardware on desk’ experience.