[HN Gopher] Launch HN: Synnax (YC S24) - Unified hardware contro...
___________________________________________________________________
Launch HN: Synnax (YC S24) - Unified hardware control and sensor
data streaming
Hi Hacker News! We're Emiliano, Elham, and Patrick, co-founders of
Synnax (https://www.synnaxlabs.com). With Synnax, engineers connect
to their sensors and actuators, stream telemetry for control and
post-processing, and query historical data for analysis. Hardware
teams use our platform in areas ranging from firing rocket engines
to operating manufacturing lines. Here's a video:
https://youtu.be/OJtVBfRwooA. Software developers on hardware
teams don't usually have infrastructure to build their custom
software on. They have to work with dozens of data acquisition
devices to get data into a single database. Oftentimes, they have
to use several different solutions to communicate with different
devices: LabVIEW for National Instruments hardware, ROS for
robotics, and SCADA software for PLCs. There's no good "glue" that
lets you coordinate all of these devices together. Developers end
up using tools like Apache Kafka for streaming and InfluxDB for
storage, but these tools are hard to get working with hardware,
plus it's a pain to configure these tools to also record and stream
data whenever commands are sent to control hardware. This forces
developers to repeatedly build adapters to get data off hardware
devices and manage separate systems for real-time streaming, data
storage, and hardware control. I (Emiliano) discovered this
problem while working as a test engineer at an aerospace company.
We used old control software that spit out data in massive 10 GB
CSV or TDMS files. After a long day and night of testing, no one
wanted to go through all the work to review the data. One day, I
was operating the system, and a very expensive component failed,
causing a multi-million dollar test stand to explode. After many
days of data review, we found a small anomaly that indicated a
component defect. I then got fascinated by this problem, and moved
into a software engineering role to improve their data pipeline.
After I left this job and went back to school, I spent most of my
time skipping classes to build Synnax, eventually meeting Patrick
and Elham. Synnax has several main parts. We have a custom time
series-database that was designed to be horizontally scalable and
fault-tolerant. The database can deploy in an OS or in a container.
Every sensor and actuator can fit into a "channel" that can be
written to using our client libraries in C++, Python, and
TypeScript. When writing to a channel the server both persists it
to the database and streams it for real-time consumption for any
type of service, such as a GUI, automated post-processing tools,
and supervisory control sequences. Finally, we've built a React
component library that simplifies the process of making a GUI, a
visualization and control desktop application, and pre-built device
integrations for National Instruments hardware and PLCs via OPC UA.
We think Synnax is unique in that it provides a bridge between
existing solutions like ROS, LabVIEW, and SCADA software and
general purpose tools like Apache Kafka or InfluxDB. Synnax is
source-available on a BSL 1.1 license (GitHub:
https://github.com/synnaxlabs/synnax, documentation:
https://docs.synnaxlabs.com). Usage of the software is free for up
to 50 channels. We aren't yet sure on what pricing to settle on--
we've talked about doing only usage-based implementation or also
adding an implementation cost. If this sounds interesting to you,
please check us out! You can follow through on our documentation
website (https://docs.synnaxlabs.com) to deploy a database and
download the desktop visualization dashboard. We'd really love to
hear your feedback and look forward to all of the comments!
Author : embonilla
Score : 19 points
Date : 2024-08-12 17:55 UTC (5 hours ago)
| candiddevmike wrote:
| This is really neat, I know a lot of folks in manufacturing QA
| that would love something like this. The telemetry aspect of
| industrial equipment is terrible IMO, so many folks are hand
| rolling sensors and triggers and then trying to duct tape an
| extremely fragile monitoring and dashboard system using something
| like graphite. Neat space to be in!
|
| How are you going to interface with the big boys like rockwell? I
| see you have drivers, what about partnerships? I know a lot of
| companies tend to only work with toolsets their provider
| "blesses", so having them on "your team" can help. You may have
| to pick favorites to win early deals/"synergy" (and may help with
| acquisition?)
|
| I've worked with industrial automation in the past and have
| always enjoyed the technical constraints within it. I would be
| interested in helping you with pre or post-sales
| support/training/implementation for your customers if you need
| it. Email is in my profile.
| embonilla wrote:
| It's been a really interesting problem to tackle - we've seen
| so many different ways that companies try to tackle this
| problem, and while theres one or two companies that have made
| really fantastic internal tools, most are ... lacking, to say
| the least.
|
| Our plan so far has been to try to interface with the bigger
| companies through the drivers we make for their hardware. We
| haven't reached out about partnerships yet, but that is a
| really good idea.
|
| Thank you for the offer - will definitely reach out if and when
| we need more help on the implementation side.
| kpmcc wrote:
| Hey! I work at a startup that does industrial automation related
| work and this looks super helpful. Going to take a deeper look
| later, but off the bat I wanted to ask why you felt a custom time
| series database was warranted when there are options like
| timescale or regular old postgres out there?
| elham_ wrote:
| Hey! Great question we get a lot. We've come from/talked to a
| lot of companies that do what you described with stuff like
| timescale and influxdb. They're useful tools and support a
| breadth of applications. We thought by building one to
| specifically leverage the read/write patterns you'd expect with
| sensor-heavy systems, we could achieve better data throughput
| and thus better enable real-time applications. For example,
| we've been able to get 5x write performance for sensor data on
| our DB compared to influxDB.
|
| In general, having built out the core DB, it has been valuable
| in allowing us to expand to the other useful features such as
| being able to write commands out to hardware at sufficient
| control loop frequencies or create smooth real-time
| visualizations.
|
| The other thing we think is really powerful is having a more
| integrated tool for acquiring, storing, and processing sensor
| data & actuating hardware. One common issue we experienced was
| trying to cobble together several tools that weren't fully
| compatible - creating a lot of friction in the overall control
| and acquisition workflow. We want to provide a platform to
| create a more cohesive but extensible system and the data
| storage aspect was a good base to build that off of.
| btown wrote:
| Did you build on any low-level libraries like RocksDB for
| data persistence etc.? Or did you fully hand-roll the
| database? Curious about the tradeoffs there nowadays.
| embonilla wrote:
| The core time series engine, called cesium
| (https://github.com/synnaxlabs/synnax/tree/main/cesium), is
| written completely from scratch. It's kind of designed as a
| "time-series S3", where we store blobs of samples in a
| columnar fashion and index by time. We can actually get 5x
| the write performance of something like InfluxDB in certain
| cases.
|
| For other meta-data, such as channels, users, permissions,
| etc. we rely on CockroachDB's Pebble, which is a RocksDB
| compatible KV store implemented in pure go.
| kpmcc wrote:
| One thing that might keep me from using this is how well
| it integrates with other tools I might want to use for
| data analysis or historical lookback. For example, I
| currently use grafana as a simple, easy way to review
| sensor data from our r&d tests. Grafana has solid support
| for postgres, timescale, influxdb and a number of other
| data sources. With a custom database, I'd imagine the
| availability of tools outside of the synnax ecosystem
| would be rather limited.
| elham_ wrote:
| That's a valid concern! As we're currently a team of 3
| building this out - we still are working on building out
| our integrations with other tools, hardware, etc. We have
| been prioritizing building direct integrations to systems
| that our current users are interested in.
|
| We also value enabling developers to build off of Synnax
| or integrate the parts they are most interested in into
| their existing systems. We've tried to service that end
| by building out Python, Typescript, & C++ SDKs and device
| integrations. We're continuing to look into how we can
| better support developers build/expand their systems with
| Synnax, so if there are any integrations you think are
| important, I would appreciate your take.
| kpmcc wrote:
| Thanks for the reply! That all makes sense, and I can totally
| relate to the "cobbling together several tools that weren't
| fully compatible" experience. There's enough complexity with
| having to support or integrate sensors/actuators with a
| variety of industrial networking protocols. Anything to
| simplify the software portion of the system would go a long
| way. Excited to dig into this a bit more, best of luck with
| ongoing development!
| elham_ wrote:
| Thank you! Happy to get any feedback after a deeper look
| ReadEvalPost wrote:
| FYI your CSS looks to be broken on Firefox.
| embonilla wrote:
| Hey, we're aware and a little embarrassed! We use some nested
| CSS selectors that aren't compatible with some browsers and
| haven't gotten around to fixing it yet, sorry!
| svnt wrote:
| Context: started a company that did essentially this about a
| decade ago. Haven't looked back much. My data may be stale or
| just biased.
|
| > We used old control software that spit out data in massive 10
| GB CSV or TDMS files. After a long day and night of testing, no
| one wanted to go through all the work to review the data.
|
| > We think Synnax is unique in that it provides a bridge between
| <lab/automation DAQ systems>
|
| On the surface it seems like anomaly detection is still the hard
| problem, but you're not setting out to solve it?
|
| Time series databases are state of the art generally in finance,
| not in industrial/InfluxDB, so I don't think saying you're 5x
| influxSB on writes is going to persuade too many people,
| especially given the cost now for a terabyte of RAM. I'll just
| move all of it to an in-memory database before I'll take on the
| switching costs.
|
| The thing I wanted was one solution for something that was always
| two: a properties/metadata database, and a separate time series
| database.
|
| It seems to me like you are maybe building a level too low and
| could get a lot more value working on the problem that you say
| motivated you in the first place. It is hard because of all the
| context required to automatically detect anomalies, but I think
| that is why it is valuable to solve.
|
| The value we had was we rolled in the data/cellular connection
| all the way down to the endpoint, so they could avoid IT
| integration, which was a big hurdle at the time. I don't know if
| IT integration is still a hang up for your customers.
|
| We found that visualization layers tended to reach down just far
| enough into the data intake world that it was really hard to sell
| just another tsdb.
| embonilla wrote:
| > I don't think saying you're 5x InfluxDB on writes is going to
| persuade too many people.
|
| I definitely agree with this. Our early prototype of Synnax
| actually sat on top of a combined Redis/S3/SQL stack and
| focused on those high level features. We found that it was
| challenging to deploy, manage, and synchronize data across
| these services, especially when you're running everything on
| prem.
|
| We've come to believe that a re-architecture of the underlying
| infrastructure can actually unlock the high level workflows.
| For example, to compare a real-time telemetry stream with a
| historical data set you'd need to query across tools like Kafka
| and Influx at the same time. For an experienced software
| engineer this isn't too hard of a task, but they don't tend to
| be the people who understand the physics/mechanics of the
| hardware. We want it to be possible for say, a Turbo machinery
| expert, to translate a Python script they wrote for post-
| processing a CSV into something Synnax compatible without a
| huge amount of work.
|
| In short, we're working on finding a way for subject matter
| experts in hardware to implement the anomaly detection
| mechanisms they already have in their head, but don't have the
| software expertise to implement.
|
| > The thing I wanted was one solution for something that was
| always two: a properties/metadata database, and a separate time
| series database.
|
| What do you think about TimeScale for this sort of use case?
| Haven't run it in production myself, but having your time-
| series data in the same place as a SQL table seems pretty nice.
|
| > We found that visualization layers tended to reach down just
| far enough into the data intake world that it was really hard
| to sell just another tsdb.
|
| This is a good point. We think that focusing exclusively on the
| DB is probably not the right approach. Most of our focus
| nowadays is on building out higher level workflows on top of
| the new database.
___________________________________________________________________
(page generated 2024-08-12 23:00 UTC)