[HN Gopher] The Perils of RSS
___________________________________________________________________
The Perils of RSS
Author : kn100
Score : 50 points
Date : 2022-03-09 23:22 UTC (23 hours ago)
(HTM) web link (kn100.me)
(TXT) w3m dump (kn100.me)
| _aavaa_ wrote:
| The real lesson to me seems to be summarized by the last two
| lines in the post: "One mistake can permanently wreck your RSS
| feed for users of cloudy platforms you [or your readers] have no
| control over."
|
| The issue isn't RSS, it's using something like Feedly.
| hombre_fatal wrote:
| Well, you also wreck the RSS feed of all of your direct RSS
| consumers too. You would have to get them all to individually
| delete and resubscribe which, unless you have all of your
| content in your feed, also deletes historical content that they
| may want.
|
| It's just that at least you can fix it for _future_ direct
| consumers by fixing your feed.
| karmanyaahm wrote:
| Unfortunately, some blogs I've seen only expose the last _n_
| posts through RSS. In that case, the retaining behavior is very
| useful. However, there could be some middle ground with a
| 'reset' hint.
| kkfx wrote:
| IF, and that's a big if, all your feeds do inject full articles
| in the feed RSS2Email is awesome: as long as you have a good
| MUA retaining and eventually sharing, searching etc in your
| historically collected feeds is hyper effective and simple.
| Unfortunately the starting if is almost never met in the
| present world...
| derekzhouzhen wrote:
| A feed is not a block chain; the feed publishers are free to do
| anything on their feeds. So the feed readers must be able to
| protect themselves from serving deleted items in perpetuity. I
| ran a server-side feed collector in roastidio.us for
| airss.roastidio.us users, my algorithm is very simple:
|
| I keep only the top N items in the latest version of the feed;
| everything else are dropped. For a fast moving feed, if the users
| don't read soon enough some items will be missed. This is still
| better (in terms of usability) than the bombardment of items to a
| user just coming back from a long vacation.
| AnonC wrote:
| > I keep only the top N items in the latest version of the
| feed; everything else are dropped. For a fast moving feed, if
| the users don't read soon enough some items will be missed.
|
| I completely dislike this behavior. I saw this on Ars Technica
| some years ago, where it was said that only the most recent 20
| items (?) would be in the feed. Ars can be a fast changing
| site, and since full text feeds are a paid privilege, I thought
| it was a disservice to paying users not to have the ability not
| to miss out on something just because they were busy with
| something else for a day or two.
| kevincox wrote:
| The problem is that RSS doesn't really have a concept of
| "deleted". In general feeds just contain the newest items.
| Items not on the feed are generally considered old, not
| deleted.
|
| You can play with heuristics like "This feed is usually sorted
| by published date and these entries I saw previously should
| still be here but are missing, maybe they were deleted" but not
| even have published or updated dates and they aren't always
| accurate.
|
| This gets even more complicated with WebSub which allows hubs
| go remove entries they have already seen. So now you need to
| not do deletion on those requests, and if you turn down your
| poll rate for feeds with WebSub (which Feedly does IIRC) you
| might not even fetch the full feed to notice that items are
| missing for weeks.
|
| The one thing that would actually be reliable here is probably
| Atom's pagnation and archiving extensions which you can follow
| to get a view of all of the items in the feed and prune missing
| ones. However you probably don't want to do this every fetch
| (otherwise it kinda defeats the point of pagnation). Maybe you
| could combine this with other heuristics to come up with a
| decent system. But I'm not even sure these have ordering
| guarantees so I'm not sure if you are going to fetch
| potentially unlimited pages just to check if an item was
| deleted.
|
| If we did want to solve this problem the best option would
| probably be to add a tombstone element to the feed to
| explicitly remove items.
|
| TL;DR as far as I am aware neither RSS or Atom have the concept
| of deletion. Much like sending someone 100 email newsletters
| unrelated to their subscription will piss them off with no
| recourse try not to send them 100 feed updates that they don't
| want.
|
| Edit: It looks like there are tombstone proposals for both RSS
| and Atom: https://oleb.net/blog/2015/11/rss-feed-item-
| deletions/. Maybe the author could have tried this, but IDK if
| anyone supports them.
| kevincox wrote:
| Different conversion for a different topic.
|
| > For a fast moving feed, if the users don't read soon enough
| some items will be missed.
|
| This may work well for you but definitely isn't what I prefer.
| For example I often subscribe to recording feeds for
| conferences or other events. These feeds don't post for most of
| the year but them might have hundreds of entries. I definitely
| don't want my feed reader to drop all but the last 10 (YouTube
| for example keeps much less than the latest 100 videos in the
| feed). I want it to keep them all so that I can work through
| the list and watch the ones I am interested in.
|
| So while your some solution might work in some cases this
| definitely isn't what I would want personally and in my opinion
| what Feedly did here was reasonable as a general behavior, even
| if it isn't optimal for this particular case. I'd rather have
| it handle the happy common case we'll then to hurt it for the
| rare case when someone makes a mistake, especially when the
| downsides for the mistake aren't that high.
| derekzhouzhen wrote:
| > I want it to keep them all so that I can work through the
| list and watch the ones I am interested in.
|
| Your usage case can be better served by visiting the website
| directly, after you learn that there are some updates from
| the feed. There is no guaranty that a feed aggregator would
| not miss anything anyway; this is the the nature of the poll
| based delivery.
| kevincox wrote:
| Thanks for explaining my use case to me.
|
| - How to I remember where I am when I have watched half of
| the videos?
|
| - What if two events happen around the same time. Now I
| need to check both instead of them being in the same place.
|
| It turns out that leaving items in my feed reader and
| dismissing them as I watch or decide not to watch works
| incredibly well.
| derekzhouzhen wrote:
| You are using the feed to overcome the usability problem
| of a poorly designed website. I have seen my fair share
| of poorly designed websites so I can't argue with that :)
|
| My feed reader's intended use case is to following a
| large number of slow moving feeds like personal blogs,
| which are usually not bursty and not super important if a
| few items fell in the crack.
| kevincox wrote:
| Maybe, but I don't expect most websites to track which
| videos I have watched and I don't want to need to create
| an account or somehow sync my watch list across devices.
| So I just use my feed reader for that and it works
| excellently.
|
| > My feed reader's intended use case is to following a
| large number of slow moving feeds like personal blogs
|
| This is also the majority of my feeds. While I agree that
| a few items falling through the cracks is a big deal I
| would still prefer if they don't.
|
| But I think the morale of the story is that there is a
| huge set of ways that people use RSS Feeds. They are a
| generic system for distributing updates and it is hard to
| make quick fixes that work well for all of the wide
| variety of use cases that people have.
| hombre_fatal wrote:
| You're just picking from different trade-offs, not doing
| something better nor worse (nor right nor wrong).
|
| A major use-case of RSS readers is to have a local,
| eternal copy of a producer's content to read at your own
| leisure. I have content in my RSS reader that I cherish
| from blogs that aren't even online anymore. That I don't
| need to visit poorly designed websites is only a tertiary
| perk--I don't need to visit the origin, ever.
|
| There's not much to argue here, though. It's just a
| different offering. For example, I use https://sumi.news/
| (created by some HNer) and certainly don't use it read
| the entire history of every news website. Nothing wrong
| with that.
| derekzhouzhen wrote:
| > You're just picking from different trade-offs
|
| Of course. I am not feedly and I can't afford to keep
| everything for all the feeds ever subscribed by my users
| in my database. There are feeds that keep thousands of
| items and there are feeds that have broken date fields so
| I cannot prune by date reliably.
| purjolok wrote:
| One mitigation is to start serving the feed at a new URL. This
| should fix things for new subscribers. The old subscribers can
| optionally be encouraged to change to the new URL manually via a
| specially crafted post in the old feed.
| AnonC wrote:
| > however the cloudy nature of Feedly removes an important avenue
| of control from the user. With RSS reader software, it is
| possible to remove and re-add a feed, thereby grabbing a brand
| new copy of it. When a content creator like myself makes an error
| like this, users can fairly easily do this after the mistake had
| been corrected, mark everything below their most recently read
| post as read, and get on with their life.
|
| I don't agree with the approach suggested here with RSS reader
| software. Removing a feed and adding it again means losing out on
| older articles that are no longer present in the feed, losing
| "starred" status of articles (in my understanding this is also a
| way to save an article permanently even if the feed publisher
| doesn't include it anymore in the feed), and losing the unread
| status of some old articles that one wanted to get to later
| (including as recent as a week or two in the definition of
| "old").
|
| I wouldn't want Feedly or someone else to remove articles after
| those have been received by my reader software. In this case the
| reason may be good and acceptable, but it breaks a contract that
| users of Feedly (not the publishers of the feeds) expect from
| such a service. Soon it could devolve into censorship requests.
| Feedly or any such service is better off, and would spend lesser
| time and effort, by avoiding messing with the feeds. "What you
| publish is what we serve the users" should be their guideline.
|
| The peril I see is publishers publishing feeds without adequate
| testing.
| ssivark wrote:
| Seems to me like this is squarely Feedly's responsibility, and
| everybody should be impressing upon then the need for the RSS
| Reader to behave sensibly.
|
| Btw, I wonder if there are any protocol level
| suggestions/guarantees about this kind of a thing
| (edits/deletes/revocations).
| kevincox wrote:
| In contrast I see this squarely as the authors responsibility.
| In general with RSS if you publish and entry it is published.
| Feedly did the right thing and displayed the published entries.
|
| The subsub+resub is a nice hack to clean old entries but I
| think in general it is nice in Feedly that you can subscribe to
| a feed and get history beyond what is currently on the feed.
|
| To me the author is asking Feedly to make the happy, common
| case worse to make the rare case where the publisher makes a
| mistake slightly better. I think the only thing that Feedly
| could have done better is manually removed the items when the
| author reached out to customer support. But if I was in the
| position I don't know if I would have conplied either. If the
| don't have infra for this it is a risky manual database
| operation. Plus feed entries are argubly their subscribers
| data, I would think twice before deleting something from under
| my subscribers even if I thought that most of them would
| appreciate it.
|
| For edits you just edit the entry. Most readers won't actually
| notify the user again but many will update the content.
|
| For deletions there are some proposals for tombstoning. I went
| into detial here: https://news.ycombinator.com/item?id=30625712
| ssivark wrote:
| I don't see why Feedly has append-only semantics instead of
| simply _mirroring_ the feed on the website. In general, as
| both a publisher and a reader, I would want Feedly's behavior
| to be simply pass-through to the extent possible.
|
| What common case does Feedly's default behavior prioritize?
| rendall wrote:
| Feedly appears to cache one feed for all subscribers. I'm
| speculating, but would it prioritize consistency above
| accuracy? Fewer complaints from people who are confused
| because that blog post they read yesterday is gone today.
| deltarholamda wrote:
| >The subsub+resub is a nice hack to clean old entries but I
| think in general it is nice in Feedly that you can subscribe
| to a feed and get history beyond what is currently on the
| feed
|
| I get what you're saying, but for something like Feedly,
| they're an aggregator, not a publisher, and therefore they
| should, where at all possible, conform to the publisher's
| intentions.
|
| If I have an RSS feed of my blog, and somehow my pet toucan
| manages to publish something extolling the virtues of Froot
| Loop National Socialism, I think deleting that from my feed
| should percolate out before I get pogromed by Tony the Tiger.
| [deleted]
| axelthegerman wrote:
| Exactly this. The pros and cons of open protocols. Does RSS
| solve this in any way? If so, the blogger or feedly have not
| implemented or used that part correctly. If not, we identified
| a missing piece in the protocol and getting that added to every
| RSS feed/aggregator is pretty much impossible (though a popular
| one like feedly should be able to adapt)
| xenomachina wrote:
| There are at least a couple of ways of indicating that an
| entry should be deleted in RSS and Atom, but they don't seem
| to be well-supported.
|
| This blog post mentions a couple:
| https://oleb.net/blog/2015/11/rss-feed-item-deletions/
|
| I vaguely recall that Microsoft also had some proposed
| mechanism, but I can no longer find any trace of it. I doubt
| anything supports it.
| hombre_fatal wrote:
| I don't think Feedly is necessarily doing anything wrong.
| Publisher beware kinda comes with the territory like making
| sure you don't accidentally send too many emails. It's kind of
| a quirk of centralization that you would have the recourse of
| "sorry, can I unsend that?" with the pubsub model to begin
| with.
|
| But I agree that it would certainly be nice if they offered a
| way to delete content. For example, an interesting attack on a
| website would be to silently add spam to their RSS feed knowing
| it will be stored in RSS aggregators forever. That doesn't seem
| great.
| kn100 wrote:
| I wonder how much data you could encode in an RSS feed and
| therefore retrieve from services like Feedly?
|
| I remember reading a while ago about movie piracy sites that
| would store chunks of encrypted video inside images and host
| these images on services like Google Drive, and then client
| side download these 'images' in order to reassemble a film. I
| wonder if a similar thing would be possible with and RSS
| feed? Feedly seems more than happy to store the entire
| content of every post I've made!
| sys_64738 wrote:
| Or, don't fiddle with web code without testing it in staging as
| you'll annoy your users.
| kn100 wrote:
| I suppose you're right, but as I explained in the post, I am
| not some big company, I'm just a person who occasionally likes
| to post stuff on the internet. I'm a huge fan of various blogs,
| and want to be a part of that space. It is so nice to browse a
| proper hobbyist blog that isn't loaded up with attempts to sell
| me things, and is just someone who is genuinely passionate
| about whatever it is they're blogging about.
|
| I personally am not a user of RSS, however I received multiple
| requests to add a feed, so I did. I don't _want_ to set up a
| staging environment for my blog, this is a hobbyist thing I do
| in my limited spare time.
|
| The whole reason I wrote this was to hopefully prevent people
| like me from making this same mistake. I wasn't particularly
| familiar with RSS and its idiosyncrasies, and paid the price
| for it. By providing my content in a format I myself have very
| little interest in without fully understanding the
| ramifications, my content has become available on a third party
| service that I have no control over, and when readers discover
| that my feed doesn't look right, they come to me to complain
| about it. It might just be a fundamental disagreement, but I do
| wonder if tools like Feedly should provide a little more
| control either to publishers, or to users.
|
| I'm just rambling at this point, but hopefully this illustrates
| my thinking behind the post better.
| [deleted]
| dvh wrote:
| My perils with rss:
|
| 1. Too many formats, it's been 20 years since something changed
| in RSS formats, we should know by now what we need. Also rdf is
| really ugly. We should make one final rss format and switch to
| it.
|
| 2. There are no rules for guids. No maximal length, I've seen
| guids several kb long in the wild. Integer guids would be perfect
| (but uuid or some hash is fine too, just agree on some maximal
| length ffs).
|
| 3. no history standard, if site owner decides to show you 20
| items and you go on vacation, when you return there will be 20
| new items and the rest will be lost. I'm thinking something like
| ?limit=200 or even better ?since=5257 (some last seen guid)
|
| 4. Sites not putting entire article in, only summary.
|
| 5. Sites putting page navigation ui in articles.
|
| 6. Pages lacking RSS feed entirely.
| breischl wrote:
| Oh man, the perils are endless. Back when I worked on an RSS
| aggregator I wrote a list.
| https://docs.google.com/document/d/1cvq67iQpk2C7ufOsefsfKnGC...
|
| A few comments on your list:
|
| 2. People got suuuuuper confused about this. I agree several kB
| is too long, though I could see a case for allowing a
| (reasonable length) URL, as that's a natural ID in some
| environments.
|
| 3. Yeah, basically if there's a fast rate of publishing
| relative to feed length you have issues. Another way to handle
| this would be to say the feed must have the last X days of
| content, and interested readers have to check back at least
| that often. Not a problem for always-on aggregator sites,
| though it could be an issue for occasionally-connected clients
| (eg, smartphone standalone app)
|
| 4. To be fair that's a feature in a lot of cases. Some sites
| have to monetize content somehow or other.
|
| 5. Or share buttons...
|
| 6. IKR?! For all my complaints I still wish RSS was used more!
| kkfx wrote:
| Personally:
|
| - lack of the entire article in the feed, Miniflux and some
| others try to download it if needed but does not work
| flawlessly;
|
| - lack of RSS/atom support at all for few but not so few
| websites;
|
| - long push against RSS by Big&Powerful that makes too many
| simply not anymore aware of it's existence, to a point that for
| many site that still have RSS finding the feed URL from the
| page source is the sole way to discover it.
|
| The rest is far less dangerous IMVHO :-)
| ekidd wrote:
| > We should make one final rss format and switch to it.
|
| This was the motivation for creating the last several versions
| of RSS and Atom, if I remember correctly! Sometimes I wonder if
| the RSS situation was the original inspiration for
| https://xkcd.com/927/.
|
| Anyway, RSS parsing is a solved problem in most major
| languages. Just look for a prominent open source library that
| handles all the major format variations, and call it a day.
|
| It's not like real-world RSS feeds actually comply with any of
| the standards, anyway. Most parsers need fallbacks and bits of
| autodetection. ("Is this escaped HTML or plain text or what?")
|
| (Full disclosure: I worked on a number of RSS-related projects
| back in the day and worked with at least one person involved in
| one of the standardizion attempts.)
| rascul wrote:
| > 4. Sites not putting entire article in, only summary.
|
| I prefer the summary and a link.
| breischl wrote:
| Try enabling Simple List Extensions[1] on your RSS feed and see
| if that deletes the old items. Basically this tells the feed
| reader that your feed isn't a typical "stream of posts", it's a
| finite list. So it should be treated differently.
|
| Newsgator used to handle that by deleting any posts not in your
| feed at every retrieval. So maybe Feedly would too.
|
| If you think that guessing at the internal implementation of
| Feedly's code based on what a company that's been dead for years
| used to do sounds like a long shot, then bravo. But hey, it's
| worth a try, right?
|
| It's been a minute, but I think the gist of it is adding
| xmlns:cf="https://www.microsoft.com/schemas/rss/core/2005"
|
| to your <rss> root element, and
| <cf:treatAs>list</cf:treatAs>
|
| under the <channel>. But see the link for examples. Good luck!
| [1] - https://docs.microsoft.com/en-us/previous-
| versions/windows/d...
| kn100 wrote:
| I'm going to try this, as well as other suggestions here across
| a range of cloud-ish RSS readers and have a shoot out! It's
| going to take me a little while since it seems Feedly at least
| does not crawl very often, but I think a little experimentation
| here would make for an interesting follow up.
___________________________________________________________________
(page generated 2022-03-10 23:01 UTC)