[HN Gopher] How we hosted FOSDEM 2021 on Matrix
___________________________________________________________________
How we hosted FOSDEM 2021 on Matrix
Author : Arathorn
Score : 182 points
Date : 2021-02-15 13:43 UTC (9 hours ago)
(HTM) web link (matrix.org)
(TXT) w3m dump (matrix.org)
| Arathorn wrote:
| Feel free to pose any questions, thoughts or feedback on how
| FOSDEM went, and how we might be able to do better in future if
| the need arises!
| exyi wrote:
| I did not have much time so I attended only a few talks.
| However, I was very pleasantly surprised how well it worked.
| Not only technically - it just works great having the video
| stream in a chat window where people can freely talk to the
| presenter. Maybe I'm even going to miss that on real-life
| FOSDEM which was inherently less interactive.
|
| Congratulations to the Matrix and FOSDEM teams.
| dsdsdsds wrote:
| I attended FOSDEM'21 as a speaker and the overall experience
| was very pleasant & smooth. No technical issues that occurred
| during the talk or q/a. Very well done to everyone involved!
| detaro wrote:
| I have no idea if that's FOSDEM infra, element.im or my home
| server (which is a fairly standard synapse install provided by
| a local tech group, so it shouldn't be out of the ordinary) to
| blame, but starting Saturday by joining a bunch of the relevant
| channels and getting told multiple times "Too many Requests!"
| and having to spread out joining 10-12 channels over 20 minutes
| wasn't fun and surprising, especially since the rooms weren't
| even set to provide backlog!
|
| I generally use Matrix quite a bit, but I didn't find FOSDEM a
| convincing example. I guess some people like having everything
| squeezed into one tab, but it didn't work well for me, other
| than that it didn't have much of a value proposition over any
| other chat system for me personally, and as initially described
| even the experience as a chat system wasn't great.
| driminicus wrote:
| You ran in to rate limiting, which can be configured in
| synapse.
| Arathorn wrote:
| All the rooms had history visibility enabled, so it sounds
| like something might have been going wrong there on your
| homeserver.
|
| In terms of membership changes being ratelimited - yes, this
| is an anti-abuse mechanism, and the default synapse config is
| overly conservative. We don't have a way of overriding that
| globally though, and the FOSDEM use case of loads of rooms
| flying around meant some people fell over it a bunch. Sorry
| you got bitten by this.
| detaro wrote:
| Regarding the history, seems that maybe it just was very
| slow to sync - I can now indeed scroll back past when I
| joined. _shrug_
| rakoo wrote:
| I didn't expect anything and was pleasantly surprised to be
| able to follow all talks if I wanted to, all with the room for
| questions and such. Congratulations, and thank you very much !
|
| There were a few pain points, given the special nature of the
| rooms. Here's my workflow for joining a room:
|
| - I have my own homeserver and am using Element Desktop
|
| - When I'm interested in a talk, for a room I'm not already in,
| I need to get the room. I can't simply copy-paste it, because
| there's a link to https://matrix.to and I don't want to fill my
| own homeserver every time. I need to click on that first link
| _then only_ I can extract the "#room:matrix.org" I'm interested
| in. I wish there was just a "matrix://" or something I could
| click and that my system could understand.
|
| - I joined many rooms just for FOSDEM and I didn't want to
| "pollute" my usual rooms. So I used the Communities thing from
| Element but it's counter-intuitive: in all other applications
| (Discord, Slack, I assume others) that button is a "namespace"
| at the top of a usual hierarchy (namespace/channel). In Element
| it's more like a tag: when I put the room in the community, I'm
| actually saying that the community is a view that lists all
| rooms with the corresponding tag. I don't know yet if I prefer
| this way of working or not, but maybe the elements should have
| a different appearance than other tools
|
| - Linked to the above, there's no "community" for the default
| view, which is "all of them" if I understood correctly; I have
| to know that I need to click after the list, below the "+"
|
| - Adding a room to a community is not exactly straightforward
| as well. I need to go in the Community, then add the room...
| but it must already be added beforehand, because
| #room:server.org doesn't seem to work.
|
| All in all it's a lot of manual manipulations for being able to
| use Element the way we're expected to be able to use it. Surely
| your backlog already has some items for the points above, so
| I'm going to wait patiently :)
| Arathorn wrote:
| > there's a link to https://matrix.to and I don't want to
| fill my own homeserver every time.
|
| Hm, for Element Desktop matrix.to should deeplink into the
| app, and so prioritise your client's settings there. If
| that's not working, please file a bug at
| https://github.com/matrix-org/matrix.to/issues.
|
| For Element Web, the _theory_ of matrix.to is that it
| remembers the URL of your preferred client (and ideally your
| homeserver) and so fixes up the links to do the right thing.
| In practice we rushed this in for FOSDEM and got it so you
| can provide the client in the URL (e.g.
| https://matrix.to/#/#misc:fosdem.org?web-
| instance[element.io... means "join this via the element web
| at https://chat.fosdem.org") - but we didn't hook it up to
| remember the default. I've just filed this as
| https://github.com/matrix-org/matrix.to/issues/197.
| Meanwhile, specifying a default homeserver (as opposed to
| client) is https://github.com/matrix-org/matrix.to/issues/15.
|
| > I used the Communities thing from Element
|
| Communities are a mess and being replaced by Spaces.
| Ironically Spaces behave the same way fundamentally (they
| filter your room list rather than switch you between
| namespaces), but we hope this will feel okay if executed
| better.
|
| > There's no "community" for the default view
|
| This should be fixed with Spaces (although quite how is still
| up for debate - the current behaviour still makes you 'click
| in the background' to unselect a Space).
|
| So, good news: spaces should solve most of this. Meanwhile
| i've bumped the priority on the matrix.to issue you hit :)
| DenseComet wrote:
| Matrix.to should remember your homeserver, but a "matrix://"
| url scheme is being worked on [1]. Work on that is caught up
| a bit in the backlog however, but there are folks in the
| ecosystem pushing that along in 3rd party clients.
|
| Communities are going away in favor of spaces (relatively?)
| soon. That was the topic of Matthew's talk at FOSDEM [2], and
| the entire experience should be vastly better.
|
| [1] https://github.com/matrix-org/matrix-doc/pull/2312 [2]
| https://www.youtube.com/watch?v=TzUfS08lMek
| ssivark wrote:
| Thanks for sharing the setup! The process seems involved enough
| for non-experts (even with open source tools), and clear enough
| for y'all who've done it, that I wonder whether it might be
| worth productizing by Element -- something to the effect of a
| turnkey solution for conference organizers (who are domain
| experts but would have a hard time figuring out and managing
| this infrastructure setup). Given the current pandemic
| situation, I wouldn't be surprised if organizers across the
| board jump on to a solution of this kind. It would also serve
| to popularize Matrix and greatly expand its user base.
| Arathorn wrote:
| Yeah, we're thinking about it. The problem is that a
| successful conference is much more than just the tech stack -
| helping guide the organisers through how to get the most out
| of it and then being around to support the event is a lot of
| the effort. My hunch is that it might be time better spent
| focusing on making the core of Matrix & Element better.
| belorn wrote:
| Since the presentation part of the videos were prerecorded, why
| were they not imminently uploaded to the archive after the
| conference?
| markvdb wrote:
| The first videos are popping up on
| https://video.fosdem.org/2021 as we speak.
|
| P.S. The FOSDEM 2021 video team was three people. With day
| jobs. Humans. One of us with kids. Not perfect. Volunteering
| our time. During a pandemic. One of us of was at a funeral
| service on Saturday morning. A close friend had died of
| covid-19 . Another had professional obligations on Sunday
| afternoon. It was great fun though, so chances are you'll see
| us again at FOSDEM 2022...
| belorn wrote:
| Sorry, didn't mean to imply that you should have worked
| harder or faster, mostly just wondered why not just release
| the presentation part first since that was already made in
| advance and the Q/A later once the editing was done.
| jeltz wrote:
| My main issue was that the widget stopped playing when
| switching conversations so I had to break out the videos I
| listened to.
| Arathorn wrote:
| Yeah, we hooked up PIP for broadcasters but not viewers -
| given it's not obvious when you're switching between rooms
| whether you want to switch video or not. Instead, you have
| pop-out button to let you view the video widget separately if
| you want, which sounds like what you were using. The UI could
| be clearer.
| guerby wrote:
| I've seen that FOSDEM chat are archived, does element web
| interface allow user to save their chat room to text files on
| their computer ?
|
| https://github.com/vector-im/element-web/issues/2630
|
| This issue has 149 thumbs up, as I noted I don't know of any
| other chat software without the ability to save discussion to a
| local file.
|
| Thanks!
| Arathorn wrote:
| Matrix itself is a conversation store, which means we haven't
| prioritised exporting to text file (or HTML file or whatever)
| in Element. A workaround for now is
| https://github.com/russelldavies/matrix-archive, which works
| really well.
| guerby wrote:
| Thanks for the pointer.
|
| I installed a debian 11 VM, installed then launched the
| matrix-archive.py script --all-rooms with my not-for-profit
| ISP matrix+element credentials. I follow mostly my not-for-
| profit ISP channels (quite small) and may be 4 or 5 rooms
| on matrix.org, fosdem.org and mozilla.org.
|
| After 30 minutes it has written 878 MBytes of archives and
| it's not done yet.
|
| I don't see an option to resume / append to an existing
| archive. So I assume I will have to restart from scratch
| each day?
|
| I wonder: will it scale if everyone uses this script?
| Arathorn wrote:
| So yes, that tool is intended for one-off archives. And
| I'm not surprised that the archives are large, given
| Matrix rooms live forever by default, so if you're in
| some room like #matrix:matrix.org the full history goes
| back to Aug 2014 will be millions of messages by now.
|
| The 'right' solution is presumably to give Element a
| button to maintain ongoing archives on desktop (it
| already archives your encrypted rooms into an encrypted
| sqlite DB for indexing) - with a control to limit how far
| back in time it goes. Separately, having a "save the last
| N days/msgs of this room" button would be useful in
| Element Web. But both simply haven't got to the top of
| the todo list yet.
| znpy wrote:
| I was curious, how much computing infrastructure did this
| required? How many machines were needed (max autoscale peak)?
| How much bandwitdh did you consume at peak ?
| Arathorn wrote:
| We overprovisioned the hardware a lot as we really didn't
| want to hit any hardware limits. In practice, I think this
| meant there were 16 medium sized EC2 instances for the
| various Synapse processes; RDS for DB; 6x largish boxes in
| Upcloud for Jitsis, and ~70 machines in the automatic scaling
| group for the Jibris. I'm guessing peak bandwidth was about
| 100Mbps but obviously all the bulk HTTP (HLS etc) was behind
| CDNs.
| rakoo wrote:
| Do you believe such a model is viable for any future
| conferences ? Does it need a lot of money and/or system
| administrators to run compared to the number of
| participants ?
| Arathorn wrote:
| It's definitely viable. I'm waiting to see what the bill
| was for the hardware costs, but I'm expecting it'll be a
| few $K. We had a lot of sysadmin on hand to try to keep
| it happy over the weekend though (but having learnt from
| the misadventures, subsequent events should be much
| easier).
| znpy wrote:
| Nice! Can you share about how much did it cost to host the
| conference then, even approximately?
|
| I wonder if your stack can become a pattern to host large-
| ish to large conferences on-line!
| robert_foss wrote:
| The Matrix P2P talk at FOSDEM 2021 was really interesting.
|
| https://fosdem.org/2021/schedule/event/matrix_pinecones/
|
| https://www.youtube.com/watch?v=a7rHtQwpuus
| SubGenius wrote:
| Matrix is not only an exciting ecosystem to build on top off, the
| overall community is pretty amazing too. Fossdem went great and I
| liked a lot of the talks.
|
| I'm beginning to build a couple of projects on matrix and am
| really hopeful for its long-term success.
| ognarb wrote:
| Same I think the Matrix community is amazing and very
| welcoming. Element is really trying to build a community behind
| Matrix and I think they doing it with a great success. They
| regularly invite community members to talk in video interview
| (This Week in Matrix). Alternative clients are actively
| encouraged and promoted and small business build around the
| Matrix ecosystem too.
|
| Shameless plug to the Matrix client, I'm contributing too:
| https://invent.kde.org/network/neochat
| SubGenius wrote:
| Unfortunately I'm running a very minimal arch + i3 setup now,
| I wonder if I can install Neochat without pulling in a whole
| lot of KDE dependencies?
|
| Neochat looks great by the way!
| MayeulC wrote:
| Nheko is quite good too, I mostly use gomuks and nheko,
| though I regularly fire up neochat too :)
|
| Not sure which are packaged, though. You usually want the
| latest goodies, so AUR/flatpak it is for me.
| tra3 wrote:
| Apologies if this is offtopic, but why matrix over discord? I'm
| just starting to learn about both (still salty about IRC's
| demise).
| approxim8ion wrote:
| Can't comment much on feature parity beyond the fact that
| Matrix has had most things I need, but Discord is proprietary
| software and is pretty hostile towards any third-party
| clients. Wouldn't be a good fit for FOSDEM, imo.
| eredengrin wrote:
| If you're an IRC fan then you might also be interested to
| know that the matrix team has made a pretty good effort to
| ensure that matrix and irc can play nice together, vs discord
| being hostile to any sort of third party apps as mentioned,
| so if your main criteria is wanting to extend longevity of
| irc, matrix is certainly the better option.
| SubGenius wrote:
| Well, Discord is proprietary.
|
| I'm not super interested in the chat/IM aspects of Matrix.
| What I really like is the protocol and the built-in
| federation and pub/sub mechanism that you can use to build
| cool decentralized apps that pass events around.
| Arathorn wrote:
| Discord is a for-profit unencrypted proprietary centralised
| walled garden. It's forbidden to connect unofficial clients
| to their server, and there's nothing stopping them from
| monetising your data/metadata in future.
|
| Matrix is a non-profit open source project that defines a
| decentralised communication protocol that forms an open
| global end-to-end encrypted communication network. Anyone can
| run a server to participate, and anyone can use whatever
| client they like. You could see Matrix as descended from IRC,
| and Discord from AOL chatrooms.
|
| The only advantage to Discord is that historically it has
| slicker UX than flagship Matrix clients like Element, more
| users, better room management, and exposes voice/video chat
| as dedicated chatrooms. Matrix is steadily catching up
| however.
| jcul wrote:
| And they can run any server they like too, of which there
| are many up and coming options - synapse (official /
| reference python implementation), dendrite (go), conduit
| (rust), construct (C++).
| kitkat_new wrote:
| I wonder if there has ever been an online conference better than
| this one. It was amazing! I really hope there will be an online-
| opt-in for next year
| jononor wrote:
| Chaos Computer Congress on Matrix next?
| Arathorn wrote:
| If CCC were up for it, we'd be game. Could even go and embed
| Work Adventure into Matrix as a widget, for best of both worlds
| (hah). https://apfelkraut.org/2021/02/virtual-conference-
| experience... was an interesting side-by-side comparison of rC3
| and FOSDEM.
| Buetol wrote:
| +1 for Work Adventure !
| vaylian wrote:
| I attended both conferences and both were great.
|
| The traditional tool for digital communication at CCC
| congresses is IRC. I personally still like IRC, but I think
| the matrix web client at FOSDEM made channel discovery and
| navigation a lot easier.
|
| I also think promoting Matrix could help with promoting open
| ecosystems in general. IRC works well, but it doesn't have
| the potential for mass adoption, which we need in order to
| curb WhatsApp's influence.
| exyi wrote:
| IRC is easy to bridge to Matrix, so that could be fairly
| compatible with the more conservative guys at CCC(C) ;)
| markvdb wrote:
| The public matrix rooms at FOSDEM were bridged to irc.
| This was not by accident.
|
| We tried to give visitors the freedom to use their
| preferred tools to join FOSDEM.
|
| Whether you were using matrix with the latest Chromium or
| irssi in a tmux session on OpenBSD, you were able to
| speak to people at the conference.
|
| Same for the streaming. Inside the matrix channel would
| have worked, but mpv on some framebuffer without X or
| wayland too...
| Athas wrote:
| I attended FOSDEM this year, and the platform worked quite well.
| Sure, there were the occasional technical hiccups, but bugs can
| always be fixed if the foundations are solid, and they appeared
| to be. Since FOSDEM's physical environment stopped scaling years
| ago, I hope this will make remote or mixed remote/physical
| participation increasingly viable.
| turbinerneiter wrote:
| > mixed remote/physical
|
| I want this for all conferences :D
| Shish2k wrote:
| > Jibri is essentially a Chromium which views the Jitsi webapp,
| running in a headless X server whose framebuffer and ALSA audio
| is hooked up to an ffmpeg process which livestreams to the
| appropriate destination
|
| So if I'm understanding this right, you have "private video
| conference -> web browser -> desktop capture -> ffmpeg -> public
| video stream"... why not "private conference -> ffmpeg -> public
| stream"? Having a web browser in the _middle_ of your livestream
| setup feels very weird :P
| Arathorn wrote:
| The reason is that the Jitsi Meet conferencing UI exists only
| as a webapp. In other words, you have to render the "private
| conference" somehow - which means compositing together the
| various WebRTC streams, switching to the active talker,
| highlighting them, showing their names etc, rendering
| screenshares etc. Jitsi is an SFU (Stream Forwarding Unit),
| which means that the compositing is done clientside in the
| webapp with a bunch of HTML+JS+CSS and WebRTC video streams -
| so if you want to livestream a conference, you need to have a
| client somewhere to do that rendering. Which means needing
| headless chromium + ffmpeg to capture it.
|
| The alternative would be to use a system where the conference
| is rendered serverside in the first place - a so called MCU
| (Multipoint Conferencing Unit) like FreeSWITCH. But this has a
| different set of tradeoffs, and having integrated both with
| Matrix we currently default to Jitsi.
|
| Totally agreed that it feels unintuitive to spin up a farm of
| 50 headless chromiums in order to livestream a conference
| though!
| kabes wrote:
| It's not uncommon to do some webrtc video compositing (MCU) in
| a headless chrome, since it's usually the easiest solution and
| you can add some decorations with html on top. But it's
| definitely not very efficient. Especially since this is mostly
| ran on linux and chrome has almost no hardware video
| decode/encode capabilities on linux (starting to change).
___________________________________________________________________
(page generated 2021-02-15 23:00 UTC)