[HN Gopher] Pipes: A spiritual successor to Yahoo Pipes
___________________________________________________________________
Pipes: A spiritual successor to Yahoo Pipes
Author : sea-gold
Score : 419 points
Date : 2024-07-01 02:03 UTC (20 hours ago)
(HTM) web link (www.pipes.digital)
(TXT) w3m dump (www.pipes.digital)
| dang wrote:
| Related:
|
| _The architecture and software behind Pipes_ -
| https://news.ycombinator.com/item?id=22959013 - April 2020 (12
| comments)
|
| _Show HN: Early-stage Yahoo Pipes spiritual successor_ -
| https://news.ycombinator.com/item?id=14440993 - May 2017 (126
| comments)
| DonHopkins wrote:
| "Put down the Yahoo Pipe and step away from the keyboard!" ;)
| ReadCarlBarks wrote:
| It is on GitHub: https://github.com/pipes-digital/pipes.
| _gtly wrote:
| I want to support this idea, though If you use their service it's
| free to play around with 3 feeds, but then it gets more pricey.
| Maybe worth it for some?https://www.pipes.digital/pricing
| yreg wrote:
| Three free pipes sound reasonable for me. If you have more than
| three use cases in your life, then it's perhaps worth it to pay
| a few bucks.
| ninetyninenine wrote:
| every form of visual programming I've ever seen is a singular
| style of some of a box that does processing and a line that
| connects that box to another box.
| wild_egg wrote:
| Anyone know of an example of any other style? Would love to see
| any half-baked experiments even.
|
| Seems like a space that could really use some innovation
| jdougan wrote:
| Scratch is arguably a style of visual programming
| viraptor wrote:
| Scratch is boxes and arrows with arrows of zero length ; )
| jdougan wrote:
| I know that's a joke, but in that case every screen is a
| grid of (v. small) boxes with arrows of zero length.
| philsnow wrote:
| Blockly (the visual editor used by Scratch) elegantly
| shows hierarchy of control flow. I've never seen a boxes-
| and-lines visual editor where loops are as easily
| understood as Blockly pieces.
| NaOH wrote:
| Keyboard Maestro foregoes the lines between boxes, but it
| still presents the same general format.
|
| https://www.keyboardmaestro.com/img/v11/overview-dark@2x.png
| __MatrixMan__ wrote:
| I don't have any code for this, but I always thought it would
| be better if the boxes represented data types and the arrows
| represented functions.
|
| Then you could use the arrow as a progress bar, and you'd
| have something that's a bit more focused on the inputs and
| outputs.
|
| Also, the data types could show up as a stack of boxes (like
| a deck of cards) so you can see when you've calculated a
| novel dataset (highlight new card on top the deck) or when
| you've recalculated a previous dataset (highlight the nth
| card).
|
| Such a view would resemble a commutative diagram (from
| category theory) and if you let the cards pile up for a while
| you could use their sizes to reason about which functions are
| bijective, which are good candidates for memorization, etc.
|
| When you want to know why a certain datum is the way it is,
| you can use the same view to trace it back to it's inputs (in
| the boxes-as-tasks mode, you end up zooming in on some
| imperative artifact... logs usually... And discerning the
| inputs is left as an exercise to the reader). I think this
| goes by the buzzword "data lineage" and it's usually this
| extra thing but I think it should be the main thing. Which is
| why I want to make the nodes into edges and the edges into
| nodes.
| andyferris wrote:
| This is interesting.
|
| How do you visually identify a function with multiple
| inputs and outputs? I tend to imagine it as a bi-graph -
| with boxes for values and for functions, and arrows in
| between. The stack idea is interesting and can work here
| too.
|
| I could get behind saying "all functions have exactly one
| input value and one output value" but you really need to be
| able to construct and deconstruct structs/records/tuples to
| be able to e.g. pass two values to one function.
| __MatrixMan__ wrote:
| Ah, yeah, you caught me. It's not so simple as a
| transform from nodes to edges. What I actually have in
| mind is a bit stranger, but the node/edge swap was my
| stepping stone towards it, which is why I mentioned it.
|
| The structs/records/tuples approach would work (I think
| this would correspond with "currying"), but I have this
| weird idea that certain types of reasoning have a "shape"
| and that we might be able to more heavily lean on
| analagous reasoning if we could easily show that shape in
| correspondence with explanations of our reasoning. I
| worry that currying everything would force us further
| away from the "natural" shape (if there even is such a
| thing).
|
| I've been working on something called plibs (simPlical
| mad LIBS, since really we're showing a
| https://en.m.wikipedia.org/wiki/Simplicial_complex here).
| The functions are represented as tuples of the same
| arity, and described via sentences with blanks in them
| (e.g. when you run ____ code in ____ environment you get
| ____ from stdout). One with three blanks would then
| appear like a triangle and then I'd fill it in with a
| gradient indicating which sides are inputs and which are
| outputs. Functions sharing a mad lib would appear as a
| regular polygon in the same color (one edge for each
| blank). Or maybe they're sort of starfish shaped so that
| each leg can reach out an touch whatever the data
| representation ends up being.
|
| One could imagine algorithms that generate patchwork
| quilts or 3D structures (like how protein folding is
| represented) out of such things. These could be computed
| only in one direction (from inputs towards outputs) but
| they could be traversed in any direction if one wanted to
| explore the relationships.
|
| I'm hoping it will be a nice way of citing your
| computational sources. You would attach them to any
| computational result (alongside a scientific paper,
| perhaps) as a way if saying "here's how I came up with
| this, rerun it to verify my result". Aside from the
| computational pathway that yielded the result that's
| being scrutinized, you could explore adjacent plibs to
| understand why certain inputs were chosen or to gain
| other contextual hints.
|
| Now that I'm describing it it sounds a bit like that
| awful zooming-through-towers-of-data visualization in
| "Hackers" the movie.
|
| So the "edge" that I was using as a progress bar in my
| previous description is really a series of paths through
| a space made of these things. The details of the
| functions being called (and of the datasets being
| generated) are attached to the polygonal regions that the
| paths connect. The paths may merge or branch. It could
| get unwieldy, so the user would have to be explicit about
| which plibs to show and which to highlight as a chain of
| computation.
|
| As for how the represent the intermediate data and their
| types, I keep changing my mind. Some days I like the
| stacks of cards thing, some days not. Hopefully something
| will sick soon, then I'll try to build it.
| viraptor wrote:
| Depends how literally you mean programming and processing.
| For example UML is a programming diagram. If you don't have
| the definitions anywhere else and always merge the diagram
| with specific methods implementation when compiling, is that
| an good example?
|
| What about ERD? If you materialise that into schema, is that
| programming?
|
| I think you'll run only into those two categories - boxes
| represent literal code you join together, or boxes represent
| something declarative that gets compiled into code but not
| necessarily in a straightforward way.
|
| I guess visual reporting like in MS Access could be thought
| of as a separate category?
|
| Depending on whether prose is programming, there's also
| https://visar.app/
|
| And then there's CUBE
| https://marc.najork.org/papers/vl1992.pdf
| Ringz wrote:
| Thank you for the very interesting links. I used UML
| extensively during my studies in the nineties, back then on
| Unix machines. Interestingly, even then, it was possible to
| work together on a UML concept. I'm not sure anymore if the
| novelty of working with a graphical UML program was the
| motivation, or the opportunity to program in something
| other than Modula. In the end, after my studies, I still
| had to program only in C++. Once accustomed to the relative
| directness, it is difficult to embrace new innovative
| concepts. But I would welcome it.
| ninetyninenine wrote:
| Text itself is a form of visual programming.
|
| We really have only two styles of visual programming:
|
| 1. Some sort language formed by left to right based
| alphabetic symbols in some pattern called "grammar". There's
| an almost infinite amount of possible grammars
|
| 2. Boxes and lines. Where boxes represent state or function
| and lines connect the boxes and state together. There's less
| formalization in this area so the axioms are vague but it can
| form something of a "grammar" similar to the above.
|
| That's it. It's not a space that could really use innovation.
| It's the fact that the entire space has never innovated.
| We've just been drilling down on a very specific and biased
| way of seeing the universe. The concept of a compiler
| literally illustrates how deep it goes and how tied the idea
| of a program is to a "language".
| DonHopkins wrote:
| I disagree that "It's not a space that could really use
| innovation." There's quite a lot of innovation and
| experimentation in the visual programming space, and many
| vastly different approaches to visual programming.
|
| You forgot about block based visual programming languages
| like Scratch, Snap!, and Nassi-Shneiderman diagrams, and
| also the most ubiquitous, widely used, universally known,
| commonly taught, extremely powerful, and easily accessible
| style of visual programming language, which the worldwide
| economy depends on and would collapse if millions of people
| didn't regularly use them every day: spreadsheets.
| okennedy wrote:
| Spreadsheets.
| smegger001 wrote:
| Spreadsheets are to visual box and line programming what
| goto spaghetti-code is to traditional programming
| languages.
| jes5199 wrote:
| I've seen a demo - cannot immediately remember where - where
| instead of data flowing through lines, it actually had a
| physics engine where the data was shot out of the boxes in
| various directions, and did computation with whatever box it
| intersected with next
| CannisterFlux wrote:
| The only other thing I can think of is GB Studio -
| https://www.gbstudio.dev/ - but the visual scripting there is
| like regular coding, but with visual blocks. It doesn't use
| boxes and lines. Instead, the script elements are more like
| blocks within blocks.
| x-complexity wrote:
| > every form of visual programming I've ever seen is a singular
| style of some of a box that does processing and a line that
| connects that box to another box.
|
| Mainly because it's the most straightforward method to:
|
| 1) Visualize a contained piece of code / function (box)
|
| 2) Show where the output of X goes to (line)
|
| There may be other styles, but this style is the easiest for a
| newbie to wrap their head around, and is the cleanest method of
| visualization. The box method can then be used to further wrap
| a bunch of boxes into a larger box when needed.
| chefandy wrote:
| There are other advantages to this method: easily showing
| which wires are active while debugging, giving new
| programmers an intuitive sense of what's happening in control
| structures like for loops, using color, symbols and line
| weights to intuitively convey things about data/object types,
| giving users a complete list of potential methods and
| parameters without having to look for it, being able to use
| more complex code shapes to navigate complex classes, people
| can learn the basics of coding logic without needing to
| memorize syntax simultaneously... There are obviously tons of
| shortcomings to visual code, but I do think a lot of the guff
| they get is unwarranted.
| hermitcrab wrote:
| The subject of textual vs graphical programming comes up so
| often that I attempted to write a summary of the pros and
| cons of each.
|
| https://successfulsoftware.net/2024/01/16/visual-vs-text-
| bas...
| pacificmint wrote:
| I realized at some point that when people at work draw on
| whiteboards it usually ends up as boxes and lines between
| boxes.
|
| But after thinking about it, it feels natural, because usually
| we talk about "Things" and the "Connections" between them.
|
| Could be a companies org chart, a family tree, a network
| diagram, abstract things like processes. It's all "Things" and
| "Connections".
|
| Maybe that's why graph theory has so many applications. Because
| at the end, it's all just graphs with vertices and edges.
| ninetyninenine wrote:
| Well programming itself with text doesn't have boxes and
| lines. So there's other isomorphisms to represent this
| concept.
| yunohn wrote:
| Programming, especially imperative, is essentially boxes
| (statements, function calls) and lines (control flow).
| That's why algorithms can be represented as flowcharts.
| ninetyninenine wrote:
| Right. I call it an isomorphism. You can translate one
| into the other and back. Whatever "other" form of
| programming exists it can and should be translatable to
| boxes and diagrams.
|
| But this "other" form on first glance can look very
| different. There are no boxes and lines in C++ or python
| for example.
| yunohn wrote:
| I think you misread my post. Why would C++/Python not fit
| my description? Which languages do you consider?
| ninetyninenine wrote:
| No. I did not misread. I believe you misunderstood.
|
| You're saying c++ and python fit into a category of
| programs that is foundation-ally really just boxes and
| lines.
|
| I'm saying that's not the right way to look at it.
| Because you can make the opposing statement. You can say
| that programs that are boxes and lines are foundation-
| ally just text programs.
|
| So because both contradictory statements can be made both
| aren't really correct. There is no hierarchy. Boxes and
| lines is one concept and textual programming is another
| peer concept standing on equal footing. Both are
| isomorphic and thus translatable to each other.
|
| The point of my post is to suggest that there are other
| concepts that occupy this equivalency space. Imagine a
| category of interchangeable interpretations of
| programming that are all translate-able between each
| other with no hierarchy. It's similar to the space of
| human languages. All human languages don't occupy a
| hierarchy yet all are isomorphic.
| TeMPOraL wrote:
| Of course it is. At a low level, AST is lines and boxes. So
| is a trace of control flow over time.
|
| Code is downstream of thinking, which makes it downstream
| of lines and boxed.
| ninetyninenine wrote:
| The ast is just the result of a technique for parsing one
| style of programming involving text.
|
| Visual programming with say boxes and lines at its core
| should not involve an ast.
|
| You need to think outside the box. Your thinking of
| textual programming as the core and the visual
| programming as an abstraction above it which is how the
| industry has been focused since its inception.
| mejutoco wrote:
| One can also think as an assembly instruction as origin and
| destination (line) and an operation (instruction).
| sen wrote:
| Programming is the lines, and the boxes is the data you're
| working with (generally).
| DonHopkins wrote:
| Right, generally but not necessarily. There are many
| different kinds of visual programming languages, with
| vastly different pure and hybrid models, including boxes
| and lines like flow charts, interlocking puzzle-piece
| blocks without lines like Snap!, Scratch, and Nassi-
| Shneiderman diagrams, and grids of adjacent cells like
| spreadsheets and cellular automata.
|
| https://en.wikipedia.org/wiki/Nassi%E2%80%93Shneiderman_d
| iag...
|
| Some visual programming languages use lines to represent
| control flow, others use them to represent data flow, and
| others use a mixture of both. Additionally, different
| kinds of control and data flow can operate at different
| frequencies within the same system.
|
| Snap!, which is essentially a visual block based version
| of Scheme, allows you to pass functions or closures, by
| wrapping blocks in gray insulating "gaskets" like lambda
| expressions that delay evaluation, and even supports
| macros, special forms, continuations, user defined
| control structures, and threading, just like Scheme.
|
| https://snap.berkeley.edu
|
| Another example is Max/MSP, which primarily uses lines to
| represent control flow and data flow. However, it also
| distinguishes between data flow at "simulation tick
| frequency" and a much higher "signal processing
| frequency". This means thousands of audio samples can
| flow along one line at every simulation tick, while only
| a single piece of data or signal (like a pure data-less
| control flow "bang") may flow along other lines at a
| slower simulation tick frequency.
|
| https://en.wikipedia.org/wiki/Max_(software)
|
| In data flow visual programming languages, a data flow
| box can emit any number of data outputs at once in
| parallel.
|
| A control flow box typically emits only one control flow
| output at a time, unless it acts like a "fork" operator.
| Fork operators, as seen in Petri nets, support concurrent
| processes by allowing multiple control flow outputs.
|
| https://en.wikipedia.org/wiki/Petri_net
|
| A data flow conditional works like a relay with three
| inputs (A, B, and Select) and one output. The Select
| input determines whether A or B is the output.
|
| A control flow conditional, like a traditional flowchart
| "if", has one or more control flow inputs, a Select data
| input or embedded expression, and multiple control flow
| outputs. The Select input or expression chooses which
| control flow output the "program counter" branches to
| next.
|
| Pure data flow networks do not have a single explicit
| "program counter." Instead, they typically evaluate nodes
| in partial dependency order, which may include loops
| (introducing a one-cycle feedback delay). Petri nets have
| multiple concurrent control flow "tokens" that flow
| between boxes along the lines in parallel.
|
| Another example is Body Electric aka Bounce, which is a
| data flow visual programming system with relay-like data
| flow conditionals, but also each box has an implicit
| "enable" input that you can use to switch it on and off
| (like a power supply), so when it's turned off, the last
| calculated outputs are latched and buffered, and can be
| read by downstream dependencies, but the values are not
| recalculated during simulation frames when it's not
| enabled.
|
| https://donhopkins.medium.com/bounce-stuff-8310551a96e3
|
| In Blender geometry nodes, data flows from left to right,
| while functions can be passed and applied in a way that
| evaluates right to left against the data flow. That is,
| functions are passed on the left, but the data is then
| processed through the function, either once or
| iteratively.
|
| https://docs.blender.org/manual/en/latest/modeling/geomet
| ry_...
|
| New Blender 4.0 Loops!
|
| https://www.youtube.com/watch?v=mr_nQBoJPXw
|
| Functions or operations can be encapsulated within nodes
| and passed along the data flow. For example, a
| "Subdivide" node contains a function to subdivide
| geometry, and this function is applied to the geometry
| data passed into the node.
|
| Nodes can pass functions as parameters to other nodes.
| For example, a "Function Input" node can be used to
| define a custom function that can be passed into another
| node, such as a "Map Range" node, which applies the
| function to its input data.
|
| Functions are applied to the data as it flows through the
| nodes. For example, a "Set Position" node can apply a
| function that modifies vertex positions based on certain
| criteria (e.g., noise texture values).
|
| Some nodes, like "Attribute Math" or "Attribute Vector
| Math," apply mathematical operations to attributes of the
| geometry, effectively using these operations as functions
| that transform the data.
|
| Nodes can be configured to apply functions iteratively or
| conditionally. For example, a "Repeat" node can apply a
| function multiple times to achieve iterative processing,
| such as repeated subdivision or transformation.
|
| Conditional nodes, like "Switch" or "Boolean Math," allow
| functions to be applied based on specific conditions,
| enabling selective processing of data based on attributes
| or other criteria.
|
| While data flows left to right, some nodes can evaluate
| data in a right-to-left manner when applying functions.
| For instance, a "Function Output" node can send data back
| up through connected nodes for additional processing
| before final output.
|
| This evaluation allows for more complex operations where
| data may need to be processed in multiple stages or
| cycles.
| kevthecoder wrote:
| Another example of 'control flow' is the new behavior
| graph from Khronos (glTF Interactivity Specification).
| They did a survey of existing visual programming
| langauges and are trying to making a standard. It's just
| been released for public comment:
| https://www.khronos.org/blog/gltf-interactivity-
| specificatio...
| Bost wrote:
| > Because at the end, it's all just graphs with vertices and
| edges.
|
| That's not the end. At the end there are just objects and
| morphisms between them. Graphs (with their edges and
| vertices) are just a special category of... Well, have a look
| at "Category Theory".
| high_5 wrote:
| Because programming is essentially about receiving the input
| data, transforming it through a process and in the end
| outputting it in some other form?
| ninetyninenine wrote:
| But then how come we can represent this concept in a form
| without boxes and lines? See python, C++, javascript, etc.
| theshrike79 wrote:
| The functions and objects are the boxes, them calling each
| other are the lines =)
| scott_w wrote:
| Because the boxes and lines are just an abstraction.
| ninetyninenine wrote:
| No the text is the abstraction. Or perhaps neither is the
| abstraction and they are just equivalent concepts like
| how uno is the same thing as one. Uno is spanish, one is
| English... there is no hierarchy where one concept is an
| abstraction over the other.
|
| The main point of my post is to suggest that there are
| other equivalent concepts between text and box/line.
| taeric wrote:
| I think we are operating on different definitions of
| abstractions/concepts, here. My guess is you are trying
| to invoke a very specific meaning of "abstraction" in
| another context.
|
| To note, most text is a symbolic representation of spoken
| words and words are abstractions of any number of things.
| Both physical things such as "running" and less physical
| things such as "thinking." (Yes, legographic texts exist,
| such that they may not be representing spoken words, but
| I fail to see how that changes things here?)
| scott_w wrote:
| Both are abstractions. They're just different types of
| abstractions. Your question makes no sense.
| ninetyninenine wrote:
| Abstractions go over things. An abstraction is something
| fundamentally layered on top of another thing. Two things
| cannot be abstractions of each other. If you encounter
| such a concept and you still think they are both
| abstractions then it is you who is not making sense.
| tgv wrote:
| As other people have said, boxes and lines is another way
| of representing order, or sequential steps. You can also do
| that in text.
|
| But you're right that there are other ways. Scratch is an
| example. But most programming languages are relatively
| close to math notation, and that appears to be convenient
| for us.
|
| If you want to see a programming language that works
| entirely differently, yet still has a graphical
| representation alternative in boxes and lines, look at
| CSound. It's an (old) language for generating sound, but
| its representation is different.
| SquareWheel wrote:
| Lines of code are essentially doing the same thing as boxes
| and arrows. It's just a different representation of the
| same idea. You can see pretty direct translations in
| programming languages like Scratch, Unreal Blueprints, and
| Godot 3's visual scripting. As well as logical
| implementations in tools like Davinci Resolve Fusion.
|
| I think most programmers find this logic easier to type out
| than to "draw" in a graph, but it's conceptually the same
| thing.
| astrange wrote:
| Sometimes it's about searching a preexisting database or set
| of constraints. In that case there may or may not be input
| but it's not as important.
| onion2k wrote:
| In the Scratch language blocks connect to each other like
| jigsaw pieces for internal function logic.
| TeMPOraL wrote:
| Scratch is Lisp, you're literally assembling the AST :).
| DonHopkins wrote:
| Scratch is much simpler and less powerful than Lisp,
| missing many of Lisp's most important features, like first
| class functions, complex data structures, and other first
| class objects. You can't even pass lists around as
| parameters, or include lists in other lists, and it's
| missing homoiconicity (representing code as data),
| reflection, macros, special forms, or even user defined
| blocks and control structures, etc.
|
| But Snap! goes way beyond Scratch, since it was inspired by
| and visually similar to Scratch, but is just as powerful as
| Scheme, including functions and everything else as first
| class objects, lexical closures, continuations, user
| defined blocks, macros, special forms, etc. It's much more
| powerful than Scratch, but uses the same visual
| interlocking block syntax.
|
| https://snap.berkeley.edu
| EclipseMantis wrote:
| Scratch does support user-created blocks.
| DonHopkins wrote:
| Block based visual programming languages don't use lines
| connecting boxes to each other, which has been widely used for
| a long time with popular visual programming languages like
| Snap!, Scratch, Squeak eToys, Blockly, etc.
|
| Snap!:
|
| https://snap.berkeley.edu
|
| Scratch:
|
| https://scratch.mit.edu/
|
| Squeak eToys:
|
| http://www.squeakland.org/
|
| Blockly:
|
| https://developers.google.com/blockly
|
| Also, spreadsheets are an extremely popular form of visual
| programming that has been around for a long time, which don't
| use lines to connect boxes either.
|
| VisiCalc:
|
| https://en.wikipedia.org/wiki/VisiCalc
|
| Lotus 1-2-3:
|
| https://en.wikipedia.org/wiki/Lotus_1-2-3
|
| Microsoft Excel:
|
| https://en.wikipedia.org/wiki/Microsoft_Excel
|
| Brad Myers' paper answers the age-old argument about whether or
| not spreadsheets are visual programming languages:
|
| https://news.ycombinator.com/item?id=26061576
| micheljansen wrote:
| Apple's Shortcuts seems heavily inspired by the visual
| programming language that Scratch uses:
| https://support.apple.com/guide/shortcuts-mac/intro-to-how-s...
| Tijdreiziger wrote:
| More like Automator:
| https://support.apple.com/guide/automator/welcome/mac
|
| (although [per Wikipedia] the first Scratch prototype does
| seem to predate Automator by 1.5 years, so I suppose it's
| possible there's a connection)
| micheljansen wrote:
| Shortcuts is indeed the (more than spiritual) successor to
| Automator. Scratch was pretty famous in interface design
| circles in the early 2000s (it was a pretty high profile
| MIT Media Lab project). I'm sure the Automator developers
| were aware of it, but it may also have been a case of both
| having a rich neighbour [1]. Agentsheets, for example,
| predates both by over 10 years [2]
|
| [1] https://folklore.org/A_Rich_Neighbor_Named_Xerox.html
| [2] https://en.wikipedia.org/wiki/AgentSheets
| smusamashah wrote:
| Turns out there is another form of visual programming called
| Spatial Programming https://www.youtube.com/watch?v=eQgxFuw8f1U
| HumblyTossed wrote:
| There is already a diagram that would capture it so much better
| - flowcharts.
| hermitcrab wrote:
| It is also what techies naturally draw on whiteboards. It is a
| very concise, expressive and flexible way to model things.
| whartung wrote:
| Galactic Modeling Language:
| https://wiki.c2.com/?GalacticModelingLanguage
| jzemeocala wrote:
| I wonder if we could pull off some wackiness with custom RSS
| feeds. Might have some utility in keeping an LLM constantly up to
| date as well.
| akshayrajp wrote:
| Can Zapier and similar tools be considered a spiritual successor
| to Yahoo Pipes too?
| TeMPOraL wrote:
| In the same sense a zoo is a spiritual success of the wild for
| a lion.
| gramakri2 wrote:
| I have no idea what this analogy means but it made me laugh
| :-)
| TeMPOraL wrote:
| Less features, more shiny, and your reality is limited to
| what contracts between third parties allow.
| nhggfu wrote:
| the twitter example link goes nowhere.
| onli wrote:
| The twitter integration did not survive their api changes and
| price hikes. I removed the block in the editor a while ago. Did
| I miss a link in the docs?
| mariusor wrote:
| I saw some shared pipes that were using twitter.
| onli wrote:
| Oh, right, one of those might have been mine. I marked it
| as private now. Thank you.
| eitland wrote:
| The docs says Twitter integration is discontinued.
| ozten wrote:
| Yahoo pipes was amazing!
|
| My vague memory of how this ended last time was processing cost
| and lack of a business model.
| yalogin wrote:
| I think this has some value potentially now, but they should not
| use "pipes" to describe it. Yahoo pipes is dead and there is no
| point associating one self with that brand anymore. The concept
| is easy enough to explain without using that word.
| scq wrote:
| What's wrong with the word "pipes"?
| ReadCarlBarks wrote:
| Pipes was created back when people were looking up alternatives
| for the Yahoo service. It isn't a new project.
| Towaway69 wrote:
| Unix pipes came before yahoo pipes (in fact they took their
| name from unix pipes).
|
| Unix pipes are alive and well, or are they also an association
| with a dead product?
| jumploops wrote:
| > The default output format of a pipe is RSS.
|
| Do people still use RSS feeds?
|
| I work on a similar product[0], conceptually at least, to Pipes,
| and many of our users have asked for RSS feeds as output.
|
| Until now I've rationalized that as low priority, but maybe I'm
| wrong?
|
| [0] https://magicloops.dev
| rtpg wrote:
| people totally use RSS feeds still. So many systems just make
| it by default, and I (and many other people) just use RSS
| readers for following blogs and whatnot.
|
| It's not a complicated thing to get setup! There's a bunch of
| details you can flub but at the end of the day it's a big thing
| of markup. The "good enough" thing is easy.
| onion2k wrote:
| _many of our users have asked for RSS feeds as output_
|
| If your users are asking for it then it's _locally_ popular
| regardless of how much use it gets on the general internet, and
| therefore it should be important to you. Local popularity (eg
| the market for your app) is significantly more important to
| product design than general popularity.
| JoshTriplett wrote:
| Exactly. The user base for "people who want Pipes" is likely
| to disproportionately want RSS as a way of generating data
| they can consume.
| gumby wrote:
| RSS is my main interface to the web
| InsideOutSanta wrote:
| Is there anyone who doesn't use RSS, whether it is as
| infrastructure behind the scenes (e.g. for a podcast or some
| kind of notification system) or explicitly in a feed reader?
| rcarmo wrote:
| I only read HN via RSS, and it is one of around two hundred
| feeds I filter, summarize and read every morning...
| onli wrote:
| I can't judge the economic impact of that for you. But with a
| service like this, it's just so easy to do that you might want
| to prioritize it :) If your model is pull based, you create a
| second endpoint that puts the data you'd normally output
| directly (otionally minus the JSON dressing) into the content
| field of RSS items. If it's push based, you could even write
| into an XML file on your server somewhere and let users access
| that. In the end, RSS is just some XML dressing around your
| data, and can be as easily implemented as a html template.
| Storing your old data (a RSS feed should not contain only one
| item) might be the biggest hurdle, if you don't do that
| already.
|
| Or you just send your users to Pipes, they could use a download
| and "build feed" block to create a feed on their own ;)
|
| Edit: Though I noticed that the endpoint of your loops can be
| specific actions, like a text message, that wouldn't translate
| of course. More a "put it into the feed" target action then.
| Pufferbo wrote:
| I exclusively consume YouTube via RSS.
| 8organicbits wrote:
| Can you share more, what reader do you use? I think YouTube
| may be the service with the most RSS feeds, as each channel
| has one. I could see RSS working well for YouTube, especially
| if the ads still play.
| shocks wrote:
| I use https://miniflux.app/ to consume YT. Works great.
| Pawka wrote:
| > Do people still use RSS feeds?
|
| Daily. For me RSS feed is the only option to pick what I want
| to read instead of reading what promoter wants me to read.
|
| If there are other ways - I would be happy to know.
| stareatgoats wrote:
| > Do people still use RSS feeds?
|
| Another data point from me: I recently started unsubscribing
| from as many newsletters as possible, and have organized them
| into hierarchical tree structures in my RSS feed reader
| instead, according to priority and topics. With many hundred
| sources by now, some of them extremely prolific posters, I
| rarely make it past the top priority ones, but for the rest,
| there is the occasional browse, and a full-text search that I
| can use on specific topics.
|
| This setup allows me to consume newsletters and blogposts much
| more efficiently, and the mailbox gets mainly used for
| correspondence, not general stuff. Suits me just fine, and as
| an added bonus: I now more seldom have the feeling my every
| move-move is monitored and stored by who knows who, who knows
| where, for who knows what purpose.
| stareatgoats wrote:
| edit:
|
| > move-move
|
| = mouse-move
| theshrike79 wrote:
| n8n is a similar tool and can be self-hosted: https://n8n.io
|
| And NodeRED is big in the home automation space:
| https://nodered.org
| rapnie wrote:
| n8n "AI-native workflow automation". Wtf.
| yakito wrote:
| I totally agree. What are they doing over there? It's funny
| because some of these companies are just jumping into trends,
| in this case, AI, probably to lure investors and users. They
| might be achieving the complete opposite result by doing so.
| If I didn't know about n8n from before and read the new
| headline on their homepage, I would have thought the tool was
| not useful for what I need.
| theshrike79 wrote:
| Dunno what they mean about that :D
|
| Nothing in the system says "AI" to me, and I've self-hosted
| one for a while now
| dbacar wrote:
| You know, there is also Apache Nifi.
| onli wrote:
| This is my project :) Nice to see this here again! If there are
| questions, I'm ready to answer (I have a rather young baby here
| though that wanted attention while writing this, but she sleeps
| now, so I think we are good).
|
| Pipes saw some internal updates recently. It was fighting with
| some instability. My first measure was an internal re-
| architeture, on a small scale. Before, the data was transported
| as text from block to block. (Almost) Every block parsed the
| input, created an RSS feed object and then worked with that on
| the data, only to then output the created RSS feed as string.
| This was changed to just work with the RSS object directly and
| move that from block to block. That had some consequences for
| some existing pipes, mainly because it also changed when the
| input feeds gets normalized.
|
| As that did not help, next step was a server upgrade. I'm not
| sure whether it was directly the additional processing capacity
| or the dependency upgrades, but that one helped so far. Knock on
| wood. Also reconfigured the amount of threads and puma workers,
| that might have solved a bottleneck somewhere. I sadly never
| pinpointed exactly what made the server process stop before.
|
| I had played with a proper split between web frontend and pipes
| processing, but at least my approach was not viable. Worked
| nicely while developing it, immediately crashed with the
| production workload. And Pipes is not all that huge... I'll have
| to try again some time.
| Gys wrote:
| What I am missing is a block to manipulate contents. For
| example, I might want to add an AI generated summary to certain
| feeds. Or if there is no image, add an alternative image. For
| those cases it could be useful to have a block that can post a
| feed item (or one selecteble field) to my custom url and add or
| replace that item with the result. Then I can make my own feed
| item manipulators for certain tasks. Thinking bigger, you might
| offer a (third party?) store of custom manipulators: 'add
| image', 'add summary', etc.
| onli wrote:
| I like that. I think it would have to be all items that get
| POSTed one by one to a url to work with the current flow, but
| that should be equally useful, no?
|
| Pipes does have some building blocks to prepare that
| operation or work with the result, there is an extract block
| and an insert block (that works by xpath). But I was always
| missing a model for the request block that I remember from
| Yahoo Pipes, that's why there is only a feed and a generic
| download block so far. Post somewhere, replace the item (or
| the item's content?) with the answer, that might be it.
|
| Edit: Thinking a bit more about this, don't forget that you
| can already chain pipes. A service you can POST to can
| probably also consume an RSS feed, download it by itself. You
| can then have one pipe that creates the feed up to the point
| where you would want to have it be treated by something else,
| let that thing output its result, and in a second pipe
| download the output and create the final feed. I still think
| the request block to POST items is a good idea, and there are
| probably services that can't store the output for the second
| pipe as they'd just react directly to a POST, but that
| chainability might be worth to keep in mind.
| Gys wrote:
| Ok, I understand the concept of connecting pipes and that
| will work as well. Still, POSTing items is more separation
| of concerns. The remote service only has to process one
| item (no need to read feeds, worry about formatting, etc).
| harinijan wrote:
| Interesting to see the evolution of Visual programming across
| many tools. We are working on something similar[0] for low-
| code/no-code devs focusing on creation of APIs, backend tasks,
| and AI workflows.
|
| [0] https://buildship.com
| freeqaz wrote:
| This is the code for a YC startup I ran in this space. Very
| similar. Build on AWS Lambda: https://github.com/refinery-
| labs/refinery
|
| I need to fix the docs because they are dead:
| https://web.archive.org/web/20200819221527/https://docs.refi...
|
| Yahoo Pipes was an inspiration though. It was similar to Heroku
| in many ways.
| smetj wrote:
| I have no experience with this particular implementation but I do
| have experience with some important business process developed in
| Azure Logic Apps, which is kind of the same in spirit I believe.
|
| From what I have seen, these kind of frameworks facilitates the
| worst possible combination of factors: A solution designed,
| created and implemented by non-programmers, not using a
| programming language ending up in a production environment.
|
| One could argue this is a success in its own but I have only seen
| these kind of things hit their limits almost immediately after
| the initial POC and evolve into terrible tech debt. Just learn to
| program or script already.
|
| That being said, I do not want to disrespect the effort into
| building this and perhaps there is a place for these kind of
| solutions I yet have to experience.
| blowski wrote:
| If the PoC can be built by those with strong domain knowledge,
| providing the basis for a detailed spec, that seems like a win
| to me.
| smetj wrote:
| Yes, I agree with that.
| robertlagrant wrote:
| That would be a win, but then the hideous code becomes the
| spec, and it becomes a project ten times the size to reverse
| engineer the "spec" into an actual spec.
| hermitcrab wrote:
| But if you let them do it in Excel, it would be 10 times
| worse again.
| onli wrote:
| Fair enough I'd say :) The concern is understandable, but I
| honestly think Pipes is different.
|
| The operations Pipes can do are at the same time very generic
| and very targeted. It's stuff like "Download this url" and
| "Filter this feed for that word", and it gives you a selection
| of blocks like extract and insert with which you can manipulate
| the data in the pipe (or the feed, depending how you look at
| it) a bit more freely. There is surprisingly a lot that can be
| done with that, especially with a replace block that supports
| regex, but it's not a deep abstraction layer above things that
| are not mappable to rather small operations.
| smetj wrote:
| Thank you for highlighting that, much appreciated.
| nsonha wrote:
| kinda related but does anyone know anything similar to yql, in
| the sense that you can query everything using the same syntax?
| cranberryturkey wrote:
| Yahoo pipes was awesome. Way ahead of its time. I don't know why
| yahoo shut it down.
| klaussilveira wrote:
| Finally! Yahoo Pipes was so ahead of its time. The whole concept
| of microdata, RSS and wrangling data around in pipes was amazing
| and it is sad that it died. The web was so promising at that
| time.
| BaudouinVH wrote:
| tiny bit of feedback : I tried to sign-in with a non gmail
| address and did not receive the confirmation email. Switched to a
| gmail address and everything was smooth in the sign-in process.
| RyLuke wrote:
| I see a lot of folks here asking about what happened to the
| original Yahoo Pipes.
|
| We had the same question, so we went and talked with a lot of the
| original team and wrote up the story[0]. We also made a fun mini-
| site that contains a lot of easter eggs (e.g. if you click on the
| "Memory Pipes" folder on the desktop, you'll see a bunch of
| candid photos of the original team circa 2007)
|
| [0] https://retool.com/pipes
| jjordan wrote:
| Those shiny gumdrop buttons hit hard with the nostalgia. <3
|
| Everything is so flat and boring now...
| anothername12 wrote:
| Back in the day I had hundreds of pipes, sub-pipes for all kinds
| of stuff from porn to price gathering, comparisons, to banking.
| When Yahoo Pipes died so did my will to port all that to
| something else.
|
| If I had to do it all again in 2024, what's a robust self hosted
| project like this?
| gabigrin wrote:
| Very cool project. A great demonstration that when you're useful
| you don't need a fancy interface!
|
| If anyone is interested in a more generic visual-programming
| language, check out Flyde - https://www.flyde.dev - an open-
| source visual-programming language I'm working on
| lolive wrote:
| At work, we use extensively Palantir Foundry and its [excellent]
| PipelineBuilder. I was reluctant at first with this low-code no-
| code stuff. But after porting custom python code, I must admit it
| covers 99% of the transformation needs we have. Plus it keeps a
| semantic all along, that is much more difficult to
| understand/maintain by code [especially in Python].
___________________________________________________________________
(page generated 2024-07-01 23:01 UTC)