https://kevinlynagh.com/newsletter/2023_09_hardware_prototyping/

- Back to Kevin's newslettersPublished: 2023 September 3

I'm traveling this month! Hit me up for a coffee/beer conversation
in:

  * Budapest (Sept. 15-19)
  * St. Louis (Strange Loop conference, Sept 20-23)
  * Montreal (Sept 24-30)
  * San Francisco (Oct 3-?)

This newsletter in two parts: software/UI, then hardware/electronics.

Interfaces for prototyping hardware

While prototyping scientific hardware, I often find myself
simultaneously iterating on both the domain model and its
corresponding user interface.

For example, a software-controlled pump might start with "on/off"
control:

struct Pump {
  is_running: bool
}

Exposing this data model with an HTML <button> or <input type=
"checkbox"> is enough for initial hardware testing.

Say the first experiments show that switching the pump on/off is too
violent (the sudden start/stop shears cells or explodes pipes or
whatever), so we add target speed and acceleration parameters:

struct Pump {
  target_speed: Quantity<AngularVelocity>,
  max_acceleration: Quantity<AngularAcceleration>,
}

These new fields can be exposed in the interface as numeric sliders
or text inputs which accept exact values of various units, e.g., "600
rpm" or "5 radians/sec". This enables more quick benchtop experiments
to determine reasonable pump speeds.

Perhaps we then add some sensors to our apparatus to measure the
current pump speed and system pressure, which enable us to write
closed-loop control algorithms. Only a single control algorithm can
be active at a time, so we model them as variants of an enumeration:

struct Pump {
    current_speed: Quantity<AngularVelocity>,
    current_pressure: Quantity<Pressure>,
    control: FlowControl,
}

enum FlowControl {
    ConstantSpeed {
        target_speed: Quantity<AngularVelocity>,
        max_acceleration: Quantity<AngularAcceleration>,
    },

    ConstantPressure {
        max_pressure: Quantity<Pressure>,
    },

    ConstantFlow {
        target_flow: Quantity<VolumeRate>,
    },

    Brake,

    FreeSpin,
}

A user interface might show the read-only current speed and pressure
as numbers, perhaps next to line charts visualizing the last few
minutes.

The flow control mode could be specified via a drop down (<select>)
or tabs, with the variant-specific fields appearing after selection.
Alternatively, if we prioritize discoverability over saving screen
space, we could show all of the modes (and their associated field
controls) at once, with radio buttons to select the active mode and
gray out the others.

When I've done this sort of data model iteration in the past, I've
just manually rewritten the UI to match. That usually means
duct-taping together something in a web browser to round-trip the
appropriate Serde JSON representation with the backend system.

Handwritten UIs allow for maximum flexibility: I can choose (or
custom-build) the most appropriate input controls for each field,
adding additional domain context like min/max speed validation,
slider step sizes, etc.

However, such UI takes a ton of time to develop and usually incurs
coordination costs (as the UI and data model are often implemented by
separate people and across tech stacks / deployment mechanisms).

I'm curious about other points of the tradeoff space here, especially
in the context of "internal tools" where the implementers are also
the end-users. Such author-operators are jointly optimizing "ease of
use" with "ease of implementation"; they might prefer an ugly, "good
enough" UI built in 10 minutes over a "bulletproof" one built over a
week.

Generating UI from a data model

A potential approach to quick, "good enough" internal tools is to
generate UI automatically from an underlying data model.

I've seen this done in a few places:

  * CRUD frameworks like Django Admin use data models and SQL
    database tables to generate a full web interface for looking up,
    adding, deleting, etc. arbitrary records.
  * Tools like JSON Editor and JSON Forms generate interactive
    controls based on a JSON schema.
  * Python's Gooey library cleverly generates a native UI based on
    command line argument structure.

However, while there's plenty of prior art on the idea of generating
UI from data definitions, empirically it's not something I actually
do.

What properties would I find compelling for prototyping scientific
hardware?

  * Developer ease of use: It should take less than 2 minutes to
    integrate a UI into an existing project. A library that slots
    into existing toolchain (e.g., a Rust crate) would be ideal, but
    if other tools must be involved, they should be quickly
    installable and easily versionable on a project-by-project level.

  * Web UI: I haven't found anything easier to distribute and use. A
    web UI allows sharing links within a team, usage on Mac/Windows/
    Linux, simple dedicated hardware (Raspberry Pi + television,
    tablet, etc.), and ability to adjust zoom at runtime.

  * Units: These are essential and have a number of subtleties.

      + Fields should indicate their default unit but allow operators
        to use alternatives. E.g., render tare value: [ ] kg but
        accept someone typing "11.3 oz".
      + Units should be customizable on a per-field basis; the same
        interface may reasonably report volumes in one place as
        liters and another as microliters.
      + Custom units should be possible. SI and its prefixes are
        great, but there are lots of practical (er, historically
        contingent) reasons where entire subfields use units like mL/
        min (chromatography), millisiemens/centimeter (conductivity),
        or one of a gazillion units for pressure: bar, bar gauge,
        mmHg, psi, atmospheres, and (lol) Pascals.
  * Validation: Simple "fat finger" checks (min/max limits) on a
    per-field level are table stakes. Algebraic data types go a long
    way towards making illegal states unrepresentable, but custom
    validation code may be required to enforce relationships between
    fields, so there should be a mechanism for this code to explain
    why it's rejecting some input.

  * Data density: Author-operators will want to see as much system
    state at once as possible, so the UI should be much denser than
    typical casual, consumer-oriented designs. Ideally many real-time
    readout fields would be visualized with charts.

  * Historical context: The system should default to storing all
    operator commands (how many inputs can a person click/type in a
    day anyway, a few hundred kB worth?). Sensor output may be more
    voluminous (especially at 10 Hz or faster), but ideally the
    system could be configured with a space-bounded lookback window
    so that the default behavior for even the quickest of prototypes
    is "data are saved" rather than "data are discarded". There's
    likely not a reliable way to handle schema changes across data
    model iterations, but ideally data would be consolidated across
    process restarts and deployments for a fixed schema. This
    historical data should also be accessible as JSON and/or CSV, so
    that they can easily make their way into Jupyter Notebooks or
    Excel (those hallowed programs through which all True Scientific
    Data must transit).

UI implementation thoughts

Most of my scientific hardware work has been in Rust, as it is
suitable for full Mac/Windows/Linux systems, microcontrollers, and
can be used for low-jitter applications like real-time control loops.

The standard Rust field definitions with the Units of Measurement
crate's types:

struct Apparatus {
    current: Quantity<Velocity>,
    target: Quantity<Velocity>,
}

don't sufficiently specify the constraints and units our ideal UI
would support. We could try and encode such things with macros:

struct Apparatus {
    #[read_only, display(meters_per_second)]
    current: Quantity<Velocity>,

    #[min("0 m/s"), max("10 m/s"), display(meters_per_second)]
    target: Quantity<Velocity>,
}

which is the approach taken by crates like serde_valid.

The advantage of this approach is that it might neatly into Rust
ecosystem tooling. Ultimately that's an empirical question, and I'm
not sure whether one (read: me) could generate the validation and UI
code at compile time without devolving into an mess of macro and
build.rs cleverness, with all the associated obtuse error messages
and poor IDE support.

And macros strike me as the only sensible expressive-but-concise
route within Rust's code generation facilities -- I shudder to think
of designing generic-filled type machinery like Quantity<Dimension,
MinValue, MaxValue, DisplayUnit> (and then asking people to use it!)

There may be a path using NewTypes -- following "parse, don't validate
" and replacing Quantity<Velocity> with some kind of BoundedQuantity
<Velocity> that guarantees the 0-10m/s constraint, but it's not clear
to me exactly what that would look like. I also worry about the
compile-time costs (generating unique types per field) and backend
ergonomics (having to write state.some_field.value.0.get::<Thing>()
to extract the value you actually care about from the types).

We can, of course, step outside of Rust's facilities and write a
program that "compiles" our data model into the necessary Rust and UI
(HTML/JavaScript) code. Defining our own lil' DSL would maximize
expressiveness, though with some initial cognitive overhead compared
to "vanilla" Rust.

In this case, I'd be inclined to explore building on something like
Clojure's Malli library, which does validation, error messages, and
can even generate conforming data (useful for checking that a schema
actually (dis)allows what you intended; not to mention load and
visualization testing):

(-> [:map
     [:id int?]
     [:size [:enum {:error/message "should be: S|M|L"}
             "S" "M" "L"]]
     [:age [:fn {:error/fn (fn [{:keys [value]} _] (str value ", should be > 18"))}
            (fn [x] (and (int? x) (> x 18)))]]]
    (m/explain {:size "XL", :age 10})
    (me/humanize
      {:errors (-> me/default-errors
                   (assoc ::m/missing-key {:error/fn (fn [{:keys [in]} _] (str "missing key " (last in)))}))}))
;{:id ["missing key :id"]
; :size ["should be: S|M|L"]
; :age ["10, should be > 18"]}

Beyond model and UI definition, there's also a question about how
this'll work at runtime: Should it be more like a "library" (a pile
of functions to be called by your code) or a "framework" (which
defines an entire lifecycle that calls your code)?

Arduino sketches are a wonderful framework: You write setup() and
loop() functions, then some Magic Arduino Stuff happens and your code
is running on a microcontroller.

We'd need some kind of start_webserver() call to kick things off,
with platform specific variations (threads vs. async runtime, std.
vs. no-std, etc.). Then maybe all we need are two channels (message
queues): Incoming<Command> and Outgoing<Status>, with the Command and
Status types being rendered in the UI as controls and data
visualizations, respectively. The API contract is essentially "we'll
render in the UI and log any statuses you give us, and we'll pass
along any valid commands from the UI".

Having the author reify a distinct Command type (rather than
generating such types automatically for every non-read-only struct
field) may be a bit more verbose, but it strikes me as better to make
things explicit using Rust's existing concepts rather than introduce
new options. For example, one could specify whether the UI should
support setting multiple fields at once by making all of the command
fields optional:

struct PumpCommand {
    speed: Option<Quantity<AngularVelocity>>,
    control: Option<FlowControl>,
}

or whether only a single field can be set at a time:

enum PumpCommand {
    SetSpeed(Quantity<AngularVelocity>),
    SetControl(FlowControl),
}

Having this level of detail may be required to appropriately expose
the underlying hardware constraints.

Overall, while there are lots of details to sort out, I suspect this
architecture would have been sufficient support all of the hardware
prototyping needs I've run across so far in my career.

I'll have a go at building something and let you know how it works
out in practice.

Other UI solutions

  * LabView is a graphical (connect-the-boxes) programming
    environment with lots of charts, scientific functions, and
    certified hardware that "just works". I only briefly touched it
    in undergrad, and believe it's expensive.

  * PLCs (Programmable logic controllers) have robust, plug 'n play
    hardware and manufacturers have (usually Windows-only) GUIs for
    reading/writing values. However, they are much more expensive
    compared to DIY Raspberry Pi and microcontroller hardware; even
    cheap families will cost a few hundred bucks for a few GPIO input
    /outputs.

  * An engineer friend recommended Dewesoft's visualization and
    control software, which is free to use with their data
    acquisition hardware (presumably expensive -- "contact us").

  * Electric UI looks like a hardware-focused, well-documented
    React.js framework that a pair of engineers in Australia have
    been working on the past few years.

On hardware

The space of "prototyping hardware" ranges from:

  * Home Automation: off-the-shelf consumer products like switches,
    lights, motion sensors that communicate over wifi or Zigbee and
    combined together with software like Home Assistant. Couldn't
    build a high-speed robot, but could do motorized blinds and
    sensor readings at 1 Hz.

  * PCB breakout boards: Dirt cheap (37 sensor assortment for $21) to
    merely inexpensive (Sparkfun's Qwiic) modules wired to a $30
    single board computer or $5 microcontroller, shipped next-day via
    Amazon and programmed via YouTube or Arduino forum copy/paste.

  * PLCs: See industrial computers for koji for a great overview, but
    tl;dr; this is the stuff designed to mount on DIN-rail in an
    electrical cabinet. Starts at a few hundred bucks for Click PLC
    family and esp32-in-a-box with a few simple input/outputs and
    quickly gets into "contact us", $50k+ territory for the Western
    brands (Siemens, Beckhoff, etc.) that control your local factory
    or power plant.

  * Custom PCBs: JLCPCB can manufacture and ship something to you for
    $50, but unfortunately you've got to learn electrical engineering
    first.

I've multiple friends wish for something between bare PCB breakout
boards and full-on PLC hardware. I'm sympathetic: It'd be great to
have something more convenient and robust than a mess of wires on a
breadboard without having to drop thousands of bucks for an
electrical cabinet full of factory-grade components. Especially since
PLCs are "just" microcontrollers made robust in a plastic housing
with extra protection circuitry, a bit of networking (fieldbus), and
software.

Using the koji fermentation controller as an example, the author
spent about $500 for a Click PLC with a few GPIOs and temperature
sensors. Why that instead of just reading an $8 K-type thermocouple
breakout with a $5 microcontroller? They write:

    In summary, we use PLCs because they are:

     1. Extremely rugged
     2. Customizable, modular and easily repaired or replaced
     3. Serviceable and programmable by any competent technician

    However, PLCs are no more than simple, real-time computing
    systems. Your average PLC is probably weaker in raw compute power
    than a Raspberry Pi, and multiple times the cost.

How much of this physical robustness, modularity, and system-level
convenience can be engineered at a cheaper price point? Is it simply
"wrapping" cheap breakout boards with:

  * more electrically robust (opto-isolated or surge-protected) IO
  * nicer wire terminals (springs rather than screws or tiny JST
    connectors)
  * enclosures that snap together and onto DIN rails
  * a fieldbus interface that can be read over USB / Ethernet / WiFi
    / Bluetooth

?

I mean, yeah, I'd love something convenient like Click or ODot
components for $10^1 rather than $10^2.

If this were feasible, why haven't SparkFun, SeeedStudio, or Adafruit
done it? Perhaps the hardware margins just aren't there. Or those
companies aren't interested in developing the fieldbus and UI
necessary for the value proposition.

It may be doable as an open source labor of love -- like a moth to the
flame, I can't help but sketch it out.

Requirements for a "hobbyist PLC" system

  * Max $5 marginal BOM cost (i.e., adding whatever electrical
    components to turn an existing sensor breakout board into a PLC
    system "block" should cost no more than $5).

  * A robust and easy-to-setup GUI for debugging and manual control
    of individual blocks (I've already nerd-sniped myself with this
    one, thanks).

  * Reference interface libraries in C or no-std Rust suitable for
    embedded use. On top of this should be built more accessible
    Python / Web libraries for folks without hard latency needs.

  * Blocks should reversably snap together, without tools, making
    robust power, data, and physical connections.

  * Blocks should have status LEDs.

  * Block enclosures should be 3D-printable on cheap FDM printers
    (Ender 3 clones) and should not require custom metal components.
    (Ideally no screws or other components at all.)

  * Block PCBs should be manufacturable by hobbyist-accessible
    assemblers (JLCPCB, PCBWay, etc.), including spring terminals or
    other external connectors -- the end-user shouldn't need to do any
    soldering.

  * Open Source designs for both the interfaces (physical and
    electrical specifications) and reference block design files
    (KiCAD, Fusion360/STEP/STL/sliced, etc.); one should be able to
    "fork" the analog input block to use a higher-resolution ADC chip
    and submit that for manufacture in under an hour.

  * Fieldbus should have < 10ms latency and < 1ms jitter for a small
    system (e.g., 10 blocks).

  * Blocks should self-describe over the fieldbus and automatically
    address themselves based on physical connectivity, so one can
    print an ordered list of all attached devices.

  * It should be easy to run a "test suite" to check the entire bus
    performance and block conformance.

  * Reference blocks should exist for the usual needs: digital input/
    output, 0-20mA analog input, etc.

Nice to haves / open questions

  * Fieldbus support for communicating 100m with a single cable --
    perhaps (power over) Ethernet?

  * Sufficient power to drive smaller stepper motors. A great demo
    would be a pen plotter or 3d printer built with the system.

  * Internal block firmware should be upgradable over the fieldbus.
    Not clear to me how hard this is to do in a generic way, nor how
    useful it will be in practice (the whole point is that the blocks
    should Just Work, so it might not be great to encourage frequent
    block reprogramming).

  * Off-grid usage. It'd be very convenient if the system could be
    easily run from solar / cordless tool battery / usb powerbank but
    I don't have a great sense of problem space and associated trade
    offs. I.e., is a block sleep mode worth the extra protocol/
    software implementation complexity, hardware costs, etc.

I'd be curious to hear about other prior art in this broad space. I
only learned about PLCs after years in Arduino "maker" world, and I
wouldn't be surprised if I'm missing entire other adjacent
communities of practice.

There seem to be plenty of electrical engineers designing modular
PCBs (e.g., Hexabitz, PURE modules) but I'm not aware of any attempts
that include PLC-like physical enclosure and fieldbus standards.

The closest "open source 3d-printable" physical standard I've seen is
the Gridfinity storage system.

Roadmap

The ideas I've discussed above seem (to me, anyway) tantalizingly
doable. There's still a ton of details to work out, but in broad
strokes I suspect the technical core could be built in a month or two
of focused work:

  * design the fieldbus protocol
  * design minimal API on both sides (controller and blocks)
  * implement in Rust, test fieldbus using familiar microcontrollers
    (I have ESP32-C3s dev boards laying around with built in LEDs)
  * build basic control UI
  * find cheap, hobbyist-friendly spring terminals and bus connectors
    (pin headers? card edge? pogo pins? clever usage of SIM or SD
    card socket springs?); design prototype enclosures around 'em
  * iterate iterate iterate
  * profit! (er, bask in open source / public good glory)

Please convince me not to start this.

Misc. stuff

  * "Creating capabilities that don't address an immediate problem
    while still setting them on a trajectory to solve a problem is
    incredibly hard."

  * Onboarding

  * "Nutrition Science's Most Preposterous Result: Studies show a
    mysterious health benefit to ice cream. Scientists don't want to
    talk about it."

  * The CIA deployed a 19mm thick satellite text-messaging device in
    the early 1980's

  * Painting a Landscape with Maths

  * Artificial Wombs

  * "Harvard exists to make society less meritocratic, and it does
    that while subsidized by everyone else. Give up."

  * "a recombinant form of the mechanosensitive protein talin was
    incorporated into a monomeric unit and crosslinked, resulting in
    a talin shock-absorbing material (TSAM). When subjected to 1.5 km
    s-1 supersonic shots, TSAMs were shown to absorb the impact and
    capture and preserve the projectile."

  * Shamir Secret Sharing

  * "Squiggle is a minimalist programming language for probabilistic
    estimation. It's meant for intuitively-driven quantitative
    estimation instead of data analysis or data-driven statistical
    techniques."

  * g, a Statistical Myth

  * skull rug

  * A longform audio interview with Katalin Kariko, the mRNA
    technology pioneer

  * How the VIRAL AI Drake Song "Heart On My Sleeve" was Made

  * Free Public WiFi

  * Welcome to Toastermuseum.com - the world largest Online Toaster
    Exhibition...

  * About Focused Research Organizations

  * The Mathematics of String Art

  * Dense, Interlocking-Free and Scalable Spectral Packing of Generic
    3D Objects

  * I am dying of squamous cell carcinoma, and the treatments that
    might save me are just out of reach

  * Learning to Love rbST: How a 30 year old synthetic hormone can
    cut the carbon footprint and land use of dairy production

This was an issue of Kevin's newsletter; sign up or see other issues.

- Back to Kevin's homepage