[HN Gopher] We've been thinking about software integration wrong...
___________________________________________________________________
We've been thinking about software integration wrong for the past
50 years
Author : mprast
Score : 20 points
Date : 2024-02-21 18:49 UTC (4 hours ago)
(HTM) web link (nerve-signals.pika.page)
(TXT) w3m dump (nerve-signals.pika.page)
| weego wrote:
| This is full of opaque phrases that I guess make sense to the
| author.
|
| I highly doubt theres a solution to anything in there.
| bemusedthrow75 wrote:
| Weirdly it's basically one long ad for an implementation.
|
| But I agree with you.
|
| Many years ago I worked with an ex-IBM "strategist" who wrote
| and talked in this way. Supposedly something of a visionary;
| hired as an evangelist. Everyone was very impressed by him in
| meetings and customer consultancy.
|
| But I am not sure if he ever said a single thing of substance.
| teaearlgraycold wrote:
| A grifter trained to lead the blind.
| zer00eyz wrote:
| I...
|
| I am dumbfounded.
|
| With the current state of the market why in the name of all that
| is holy would you depend on one vendor to get to another?
|
| I dont need to be at the whim of one vendor to get to the next
| one. What happens when the vendor in-between goes out of business
| or wants to present their own service or...
|
| I have no words for how bad this idea is.
| CharlesW wrote:
| It's the, "Intuitively, there's no reason you shouldn't be able
| to" that kills me.
| bemusedthrow75 wrote:
| Fret not, Charles, you are not alone.
|
| I too am upset that Big EDIC has been thwarting this
| intuition for the better part of fifty years.
| apimade wrote:
| You're thinking like an engineer. Engineers are poor at
| estimations, poor communicators and can't give me a damn
| budget. I'll never understand why engineers don't see that
| abstraction, even through corporations is bad in one area, but
| good in others (libraries, dependencies).
|
| Zapier? Tines? Workato? That's $10B in marketshare that
| disagrees with this, without even considering the order of
| magnitude of revenue present in companies like IBM, Oracle,
| Salesforce, SAP, etc.
|
| And with these tools I know how long it'll take because it's
| 90% already built. The SLA's and BCP they provide allow me to
| delegate risk "enough" - the insurance company agrees - just
| the same as I delegate risk for my authZ/authN to Okta and
| Auth0, and my infrastructure and pretty much the entirety of my
| business to one-or-many cloud providers.
|
| Do I wholly agree with the above? No. I'd sooner we all moved
| to paper-based databases, and basic HTML forms linked up to
| printers. But I have to be realistic.
| datadrivenangel wrote:
| The abstraction is visible. It's also important to remember
| vendor lock-in, and how they can raise rates and start
| squeezing out all of your profit margin.
| johnorourke wrote:
| The Zapiers of this world do not disagree.. they've built the
| hub and all the apps are spokes on their wheel. Engineers
| know that abstractions only help to create good architectures
| and high-level understanding, but they don't actually make
| the magic happen on the ground.
| beaviskhan wrote:
| Seriously, integrating with a single vendor is often
| nightmarish. And this guy is proposing some unholy vendor
| centipede? Harrrrrrd pass.
| Almondsetat wrote:
| Author has discovered (f [?] g)(x) and wants to tell the world
| about it
| Animats wrote:
| It's a clickbait ad for something called "Nerve".
|
| The original idea for "electronic data interchange"[1] was to
| have a standard format for inter-company invoices, purchase
| orders, bills of lading, customs clearance, etc. So you only need
| N implementations, not N^2. This got going in the 1970s. So the
| formats are ancient. This is a flight ticket availability request
| in the international EDIFACT format. UNA:+.? '
| UNB+IATB:1+6XPPC:ZZ+LHPPC:ZZ+940101:0950+1'
| UNH+1+PAORES:93:1:IA' MSG+1:45' IFT+3+XYZCOMPANY
| AVAILABILITY' ERC+A7V:1:AMD' IFT+3+NO MORE
| FLIGHTS' ODI'
| TVL+240493:1000::1220+FRA+JFK+DL+400+C'
| PDI++C:3+Y::3+F::1' APD+74C:0:::6++++++6X'
| TVL+240493:1740::2030+JFK+MIA+DL+081+C' PDI++C:4'
| APD+EM2:0:1630::6+++++++DA' UNT+13+1' UNZ+1+1'
|
| This dates from when data communications were really expensive
| per byte. It's also from when people were thinking in terms of
| forms expressed in machine-readable data, not an API which
| actually did something. There are about five different standards
| bodies for this sort of thing. There's a mini-industry which
| converts this sort of thing to and from other formats.
|
| Anyone involved in modern systems for this sort of thing?
|
| [1] https://en.wikipedia.org/wiki/Electronic_data_interchange
| johnorourke wrote:
| I'm involved in "modern" systems for Purchase Order and Invoice
| transmission between companies - using a standard called
| "cXML". I say "modern" - it dates back to 1999 when we were
| shouting "XML all the things!". 25 years later companies are
| just discovering how to use it to stop manual data entry.
| ivan_gammel wrote:
| Having Salesforce, Zendesk, Hubspot and Pipedrive that share the
| same data in one company I would immediately start cutting costs
| instead of thinking about integration. Even if I had to think
| about it, the only correct approach for data propagation is
| directly from source of truth (or in "colonial" terms - master
| system) to dependent systems, and the only middleware in between
| them being data delivery (simple queue or ESB, workflow
| automation platform like Camunda or Zapier etc).
| ako wrote:
| Not really, usually you have some translation to ensure data
| from different systems is aligned/translated into a unified
| format. It's expensive and cumbersome to do this in every
| dependent downstream system over and over again. Especially if
| upstream systems change, you want to implement that change in
| one central place instead of all downstream systems.
|
| I think there's a lot of value in the snowflake approach: put
| all your enterprise data in a single logical data layer, where
| you can easily share that data with different tenants.
| Transformation and unification can be handled by the data
| layer.
|
| Systems get customized a lot: sap installations have a lot of
| company specific fields and objects, same for most other
| enterprise systems. Having all those customizations effect a
| network of connected systems is hard to manage. Better to dump
| all data into an enterprise wide data layer, and then let every
| tenant take whatever it needs. Also will provide for more
| flexible access to the data (sorting, pagination, filtering)
| and better performance.
| ivan_gammel wrote:
| I fully agree with you and what you said is basically the
| same as my comment. You probably misunderstood it. What I
| meant is that logical data layer (I call it data delivery,
| which of course includes translation) should not serve any
| other function, i.e. be some system with business logic other
| than translation.
| datadrivenangel wrote:
| This works great until marketing builds their own system
| because they can't get data access fast enough and then grows
| it into a competing CRM and you have multiple systems of
| truth...
| ivan_gammel wrote:
| This is the reason why MarTech must be number one priority
| for a CTO of the company where execs are talking more about
| the funnel and CACs than about UX or security. Two CRMs is
| one of the biggest technical debts that you can imagine,
| the one that may require buy-in from the half of the
| company.
| apimade wrote:
| Hey Matt, it's a really neat product idea. I'd really recommend
| taking a niche and building it for that first. Selling a new
| Enterprise-wide integration tool is hard. Selling a new data
| orchestration concept will be harder. I recently moved out of
| that area, but I worked in this space for about a decade.
|
| I can already see the rejections:
|
| Debugging and Support: Rather than a centralised support system
| with levels, to support this you'd need a technical user familiar
| with the integration, or have the field mappings so well-
| documented and available a level 2 support rep could figure it
| out. The latter will never happen.
|
| Brittle: This introduces a lot of risk to the organisation, as
| understanding conditions of failure and what the repercussions
| are in the event one of those services stops working becomes..
| Complicated. There's a reason authN/authZ providers have opted
| for hub & spoke model, and that's because you need a highly
| specialised, often under-resourced team to look after it. Sound
| familiar? If you've worked in integration, it will.
|
| Development and Resourcing: Like any proprietary product, this is
| another thing to hire for, and support. The uplift required for
| someone to understand this concept, and run with it for the use-
| case you've offered which is adopting it across the entire
| organisation requires.. Executive-level buy-in. To put it
| plainly, adopting this would introduce too much risk to the
| business in its current form.
|
| I'd recommend taking a smaller bite with your approach. You're
| one person. Take a niche, and build something people want for
| that - people don't want technology for the sake of technology
| anymore. That money's dried up, and if you're trying to form a
| business off this concept - you'll need deep pockets to develop
| it.
|
| Here's one that could be relatively successful: Take SIEM
| tooling, for example. For small and medium-sized businesses,
| operating a SIEM can be a massive undertaking. Usually the first
| thing an organisation wants to understand when they look to do
| this is: what audit logs do we have, and how can we look at audit
| logs for a user or employee across a range of services. Every
| company operating in a regulated environment, or company wanting
| to ensure they understand what an employee did during their last
| 2 weeks of offboarding would be interested in this.
|
| If you built a product and demo purely to extract user/employee
| audit logs from services and expose them in this interface - I'd
| be lining up to try it.
| bemusedthrow75 wrote:
| > it's a really neat product idea.
|
| The one thing this idea is _not_ is _neat_ , though. It's the
| opposite of it. It entrenches the opposite of neat.
| apimade wrote:
| At the end of the day it's SQL on top of my integration
| layer. I love that. I don't know how it'd work in practice,
| or if it's any better than adopting a data warehouse - but
| even for understanding state management or offering a tool to
| create a view of a "customer" data definition is valuable.
|
| There's value to be had, I don't love the concept the author
| is proposing but the overall thing they're building, provided
| it "just works", certainly has some legs.
| hadlock wrote:
| This might be functional for a startup with an exit horizon of 2
| years and nobody needs to inherit/own the mistakes of green biz-
| ops people who don't know what they're doing.
|
| I've had to integrate new functionality with disasters like this
| and it is always a huge pain to support. Especially when the one
| guy maintaining the zendesk application leaves the company and
| we're like, "what is the purpose of this zendesk app in the
| django monolith doing?" and six months after we update it to
| support the new thing we added, we find out that django app's
| functionality was superceeded by xyz a year ago.
|
| If I found out the mission critical biz ops side of the company
| was run like this I would not accept the offer.
| bradleybuda wrote:
| This is a neat idea, but the unfortunate reality is that every
| link in an "integration chain" is almost guaranteed to be lossy.
| The author notes an obvious version of this (what if one node
| doesn't support a field that the other nodes do) but there are
| countless other types of loss in each link - structure (i.e.
| document data types), precision, limits (max field size, max
| number), other "exotic" datatypes (geography, binary blobs). Even
| mutability can cause lossiness - some of the fields in these
| systems are write-once, and even if creates and updates can be
| propagated, delete propagation is a Wild West in SaaS
| applications.
|
| If you're old enough to remember what happens when you make a
| repeated analog copy of something (fax machines, copy machines,
| VHS cassettes, etc) you know what happens with this kind of data
| flow topology. We advise our customers to do the /exact/ opposite
| of this - follow as few links as possible for any piece of data
| and move towards hub-and-spoke, ideally with the hub as a highly
| configurable, low loss node - in my world, this is a data
| warehouse or data lake, but it might also mean an iPaaS or
| similar.
| jerf wrote:
| It may not be immediately obvious, but this is basically another
| _cri de coeur_ to the effect of "Why can't we make the Semantic
| Web work?"
|
| That's a good question. But you need to start from the fact that
| people have actually tried really, _really_ hard, rather than
| nobody having tried at all, and it in fact hasn 't worked despite
| rather substantial investment. When you start from there, you may
| obtain some useful understanding. Perhaps even useful
| understanding that will lead to solving some of the problem. I've
| got no problem with people jousting with windmills, I just think
| that anyone who wants to take such a task on really ought to
| research previous jousting attempts first if they want to succeed
| (as opposed to just doing it for fun).
|
| But if you start from the blatently false presumption that you're
| the first person to ever dream of data that can just flow between
| systems, well, yeah looking out into the world you're going to be
| confused at what you see.
|
| We have in fact tried the "correct" way, and it turned out not to
| be correct. Whereas what we're doing now, however annoying it
| definitely is, does in fact at least partially work.
| zitterbewegung wrote:
| The semantic web did work. Google was able to enrich their own
| searches and bought a large semantic web called Freebase and
| incrementally enriched their web searches . When they can't
| show you a relative ad they want to capture your search
| regardless and show you relevant info so you don't switch
| providers.
| jerf wrote:
| If the "semantic web" worked, they wouldn't have _had_ to buy
| a startup. You don 't need a startup to read kindly-provided,
| accurate, well-categorized RDF tuples out of a web page.
|
| One of my favorite ironies of the "semantic web" is almost
| every story cited as a success story by its advocates is in
| fact a failure story, in that it involves somebody having to
| do something that wouldn't have been necessary if the
| semantic web actually existed.
|
| By the way, LLMs completely finish killing off the semantic
| web. The semantic web will never take the form of the entire
| world providing nice public RDF snippets. You "just" feed the
| real-world goo into an LLM (or its sequels) and extract what
| you need. It turns out that it was literally easier to solve
| the language comprehension problem than to implement the
| semantic web. (Even if you don't consider the language
| comprehension problem solved yet, we are _well_ on track to
| having it solved before the semantic web is implemented.)
| BrandoElFollito wrote:
| I am always put off by articles titled "we've been doing X wrong
| for Y years" ("and #3 will shock you!" :)).
|
| I have the feeling that this is the article of someone who has a
| vision under the shower and shares it with the wrong-doers to
| bring them wisdom.
|
| Now, I have not read this specific article so it may be
| revolutionary for once.
| lijok wrote:
| We tried this when we were building DAM integrations. It doesn't
| work. Lossy integrations. You convert a datetime from Zendesk to
| your format via Hubspot and it goes from 1943-01-01T14:00:00Z to
| 0 because HubSpot stores that field as a unix timestamp and is
| very lax around bounds. You pull a decimal and you loose
| precision. Floats become ints, lists become dicts, strings become
| xml, binary blobs become urls, etc.
___________________________________________________________________
(page generated 2024-02-21 23:01 UTC)