https://forum.qubes-os.org/t/create-a-gaming-hvm/19000 Qubes OS Forum Create a Gaming HVM Community Guides version-r41, configuration, hvm, dom0 taradiddles June 2, 2023, 5:19am 1 Hardware To have an 'HVM' for gaming, you must have * A dedicated GPU. By dedicated, it means: it is a secondary GPU, not the GPU used to display dom0. In 2023, 'Nvidia' and 'Amd' GPU work. Not tested with Intel GPUs. External GPU using thunderbolt work (Create a Gaming HVM - #8 by solene) * A lot of patience. GPU passthrough is not trivial, and you will need to spend time debugging. * A screen available for the gaming 'HVM'. (It can be a physical monitor or just to have multiple cables connected to the screen and switching between input source) [Optional] * Dedicated gaming mouse and keyboard [Optional] . IOMMU Group Goal What The goal of this step if to retrieve the default IOMMU Group (VFIO - "Virtual Function I/O" -- The Linux Kernel documentation) of your hardware. Why It can help understanding potential issue with your setup (what devices live in the same IOMMU group as your GPU) / finding potential workaround. If you feel lucky, skip this step. How You can't see your IOMMU Group when you are using Xen (the information is hidden from dom0). * Boot a live linux distribution * In the grub, enable iommu: Add the parameters iommu=1 iommu_amd= on to the linux commandline * Once you logged in your live linux distribution, you need to retrieve the folder structure of /sys/kernel/iommu_group. You can use the following script to do that: #!/bin/bash shopt -s nullglob for g in /sys/kernel/iommu_groups/*; do echo "IOMMU Group ${g##*/}:" for d in $g/devices/*; do echo -e "\t$(lspci -nns ${d##*/})" done done GRUB modification You must hide your secondary GPU from dom0. To do that, you have to modify the GRUB. In a dom0 Terminal, type: qvm-pci Then find the devices id for your secondary GPU. In my case, it is dom0:0a_00.0. Edit /etc/default/grub, and add the PCI hiding. GRUB_CMDLINE_LINUX="... rd.qubes.hide_pci=0a:00.0 " then regenerate the grub grub2-mkconfig -o /boot/grub2/grub.cfg If you are using UEFI and Qubes OS 4.1 or earlier, the file to override with grub2-mkconfig is /boot/efi/EFI/qubes/grub.cfg. Note: if after this step when you reboot the computer you get stuck in the QubesOS startup that means you are trying to use the GPU you just hid. Check your BIOS options. Also check the cables, BIOS have some GPU priority based on the type of cable. For example, DisplayPort can be favoured over HDMI. Once you have rebooted, in dom0, type sudo lspci -vvn, you should see "Kernel driver in use: pciback" for the GPU you just hid. Configuring the parameter "max-ram-below-4g" Since the release of this qubes version of xen: 4.17.2-8 (R4.2, 2024-01-03), no additional configuration is required. Remove any existing "max-ram-below-4g" workaround If you are using an older version Why do we need to do that ? github.com/QubesOS/qubes-issues/issues/4321 Copy-paste of the comment: This is caused by the default TOLUD (Top of Low Usable DRAM) of 3.75G provided by qemu not being large enough to accommodate the larger BARs that a graphics card typically has. The code to pass a custom max-ram-below-4g value to the qemu command line does exist in the libxl_dm.c file of xen, but there is no functionality in libvirt to add this parameter. It is possible to manually add this parameter to the qemu commandline by doing the following in a dom0 terminal. --------------------------------------------------------------------- ( "max-ram-below-4g" is not related to the amount of ram you can pass to the VM. It is not related to VRAM either. It is related to WHAT is available in the memory space for 32 bits system and WHAT is stored in the memory space for 64 bits system (usable ram is only a part of what need to be mapped in the memory space) ) Finding the correct value for this parameter Below, we set the "max-ram-below-4g" parameter to "3.5G". For some GPU this value need to be "2G" (discovered here: Quick howto: GPU passthrough with lots of RAM). It is not currently well understood why the value need to be exactly "2G" or exactly "3.5G", or maybe some other values for other GPU/configuration we never saw yet. ( AppVM with GPU pass-through crashes when more than 3.5 GB (3584MB) of RAM is assigned to it * Issue #4321 * QubesOS/ qubes-issues * GitHub ) More investigation is required to understand what is going on with this parameter. Current best guess is to run this command in dom0: lspci -vvs GPU_IDENTIFIER | grep Region , for example: lspci -vvs 0a:00.0 | grep Region. if the max value of [size=XXXX] is 256MB, try 3.5G for max-ram-below-4g. If the max value is bigger, try 2G for max-ram-below-4g. Update: I think I discovered the reason (AppVM with GPU pass-through crashes when more than 3.5 GB (3584MB) of RAM is assigned to it * Issue #4321 * QubesOS/qubes-issues * GitHub Xen project Mailing List ). If you want and have the skills required to compile the xen package, try to apply this patch Fix guest memory corruption caused by hvmloader by neowutran * Pull Request #172 * QubesOS/qubes-vmm-xen * GitHub and confirm it if work as expected. With this patch, the part about "patching stubdom-linux-rootfs.gz" is not needed. Patching stubdom-linux-rootfs.gz I modified the original code to: * make it works with Qubes R4.1/R4.2 * removed one of the original limitations by restricting the modification to VM with a name starting with "gpu_" * Added a way to modify per vm the value for "max-ram-below-4g". Ex, if you want specifically to use 2G for "max-ram-below-4g", name the vm "gpu_2G_YOURNAME", if you want specifically to use 3.5G for "max-ram-below-4g", name the vm "gpu_3n5G_YOURNAME" mkdir stubroot cp /usr/libexec/xen/boot/qemu-stubdom-linux-rootfs stubroot/qemu-stubdom-linux-rootfs.gz cd stubroot gunzip qemu-stubdom-linux-rootfs.gz cpio -i -d -H newc --no-absolute-filenames < qemu-stubdom-linux-rootfs rm qemu-stubdom-linux-rootfs nano init Before the line # $dm_args and $kernel are separated with \n to allow for spaces in arguments add: vm_name=$(xenstore-read "/local/domain/$domid/name") if [ $(echo "$vm_name" | grep -iEc '^gpu_' ) -eq 1 ]; then max_ram_below_4g=$(echo "$vm_name" | grep -iEo "_([0-9]*(n[0-9]*)?)._" | cut -d '_' -f 2) if [[ -z "$max_ram_below_4g" ]]; then max_ram_below_4g="3.5G" fi max_ram_below_4g=$(echo "$max_ram_below_4g" | sed 's/n/./g') dm_args=$(echo "$dm_args" | sed -n '1h;2,$H;${g;s/\(-machine\nxenfv\)/\1,max-ram-below-4g='"$max_ram_below_4g"'/g;p}') fi Then execute: find . -print0 | cpio --null -ov \ --format=newc | gzip -9 > ../qemu-stubdom-linux-rootfs sudo mv ../qemu-stubdom-linux-rootfs /usr/libexec/xen/boot/ Note that this will apply the change to the HVM with a name starting with "gpu_". So you need to name your gaming HVM "gpu_SOMETHING". Alternatively, the following dom0 script "patch_stubdom.sh" does all the previous steps: #!/bin/bash patch_rootfs(){ filename=${1?Filename is required} cd ~/ sudo rm -R "patched_$filename" mkdir "patched_$filename" cp /usr/libexec/xen/boot/$filename "patched_$filename/$filename.gz" cp /usr/libexec/xen/boot/$filename "$filename.original" cd patched_$filename gunzip $filename.gz cpio -i -d -H newc --no-absolute-filenames < "$filename" sudo rm $filename grep -i "max-ram-below-4g" init && echo "!!ERROR!! The thing is already patched ! EXITING ! " >&2 && exit patch_string=$(cat <<'EOF' vm_name=$(xenstore-read "/local/domain/$domid/name") if [ $(echo "$vm_name" | grep -iEc '^gpu_' ) -eq 1 ]; then max_ram_below_4g=$(echo "$vm_name" | grep -iEo "_([0-9]*(n[0-9]*)?)._" | cut -d '_' -f 2) if [[ -z "$max_ram_below_4g" ]]; then max_ram_below_4g="3.5G" fi max_ram_below_4g=$(echo "$max_ram_below_4g" | sed 's/n/./g') dm_args=$(echo "$dm_args" | sed -n '1h;2,$H;${g;s/\\(-machine\\nxenfv\\)/\\1,max-ram-below-4g='"$max_ram_below_4g"'/g;p}') fi \# $dm_args and $kernel EOF ) awk -v r="$patch_string" '{gsub(/^# \$dm_args and \$kernel/,r)}1' init > init2 cp init /tmp/init_$filename mv init2 init chmod +x init find . -print0 | cpio --null -ov \ --format=newc | gzip -9 > ../$filename.patched sudo cp ../$filename.patched /usr/libexec/xen/boot/$filename cd ~/ } grep -i "max-ram-below-4g" /usr/share/qubes/templates/libvirt/xen.xml && "!!ERROR!! xen.xml is patched ! EXITING ! " && exit patch_rootfs "qemu-stubdom-linux-rootfs" patch_rootfs "qemu-stubdom-linux-full-rootfs" echo "stubdom have been patched." OUTDATED - DO NOT USE - Other method: Patching xen.xml instead of stubdom Instead of patching stubdom-linux-rootfs, you could inject the command directly inside the configuration template. It is the file "core-admin/ "templates/libvirt/xen.xml" in the "qubes-core-admin" repository. In dom0 this file is in "/usr/share/qubes/templates/ libvirt/xen.xml" See below the part that have been modified to add the needed "max-ram-below-4g" option. /root/.pulse/client.conf The sound was buggy/laggy on my computer. So tried to find a workaround by playing with pulseaudio settings. It was more or less random tries, so I can't really explain it: In /etc/pulse/daemon.conf add the following lines: default-fragments = 60 default-fragment-size-msec = 1 high-priority = no realtime-scheduling = no nice-level = 18 In /etc/pulse/qubes-default.pa{.text} change load-module module-udev-detect to load-module module-udev-detect tsched=0 You can launch you favorite Windows Manager like that sudo ./xorgX1.sh /usr/bin/i3 Automating most of that I trying to write a script to automate all the previous steps. It is available here: https://git.sr.ht/~yukikoo/gpu_template Please be carefull and read the code, not much tests have been done Issues and fixes In one case in a setup with Intel IGPU + Nvidia DGPU, dom0 xorg crashed. Solved the case by adding a Xorg configuration to explicitly use Intel: # GPU Passthrough -> lightdm.service won't start User Support I was able to pass through my GPU without even hiding it using boot parameters by just configuring the correct GPU to boot Xorg on. Since I'm using my iGPU for Qubes, the config file is this: vim /etc/X11/xorg.conf.d/20-intel.conf Section "Device" Identifier "Intel Graphics" Driver "Intel" Option "DRI" "3" EndSection I think this might fix your situation. If it doesn't work, you can simply just delete the 20-intel.conf file you created and go back to normal. References * Archlinux: PulseAudio * Archlinux: PulseAudio/Troubleshooting --------------------------------------------------------------------- This document was migrated from the qubes-community project * Page archive * First commit: 18 Jan 2023. Last commit: 18 Jan 2023. * Applicable Qubes OS releases based on commit dates and supported releases: 4.1 * Original author(s) (GitHub usernames): neowutran * Original author(s) (forum usernames): @neowutran * Document license: CC BY 4.0 Contributors * @neowutran * @deeplow * @xvrhthxn 17 Likes Quick howto: GPU passthrough with lots of RAM Possible to Fix 2048MB RAM Limit for NVIDIA GPU Passthrough to HVM? Curious about best practices for using qubesOS while also working on some ML projects VRAM within a qube for first person games, or other graphically "heavy" loads System keeps rebooting after trying to add any hardware Disabling a qube's virtual display? I have no monitor output, and I must set up gpu passthrough Hide cursos in-game External monitor NOT detected (only) when PCI devices hidden (GPU passthough) Problems with HP G2 Z9-mini GPU's AI Laptop for Running Machine Learning Workloads on Qubes OS - Any Recommendations? Problems with HP G2 Z9-mini GPU's Nvidia GPU Passthrough with Prime on Debian 12 (GUIDE) Qubes OS 2.4.1 Crashes at plymouth-quit-wait.service libvirt.libvirtError: internal error: libxenlight failed to attach pci device 0000:04:00.0 libvirt.libvirtError: internal error: libxenlight failed to attach pci device 0000:04:00.0 Best Way to GPU Passthrough For a Beginner Nvidia gpu for passthrough Probleme peripherique USB PCI AMD GPU Passthrough (but not for gaming) Trying to set up a (preferably windows) gaming HVM with an nvidia GPU on a laptop. "No bootable device" when starting How Painful Will Windows & Gaming Be? How Painful Will Windows & Gaming Be? Nvidia gpu for passthrough Cannot increase Qube memory above 1971 MB NVIDIA proprietary driver installation community guide Passthrough GPU causes system to restart Disable Secure Boot on Qube GPU passthrough extremely bad performance, USB controller fails to load driver Poor GPU performance in Qube Passthrough GPU: Connection reset by peer Windows gpu passthough (Guest has not initialized the display yet) qubes 4.2 Qubes and Windows 11 multi-boot GPU Passthrough -> lightdm.service won't start GPU Passthough Issues GPU Passthough Issues Qubes does not boot without 'module_blacklist=amdgpu' kernel parameter HVM no bootable device after gpu passthrough Need help with gpu passtrough Gpu passthrough performance/usage GPU passthrough : Trying to display on DP GPU Acceleration How to edit GRUB options, unable to boot Devices greyed out - qvm-pci gives "failed" Issue with hiding GPU for passthrough Xorg errors while attempting gpu passthrough Xorg errors while attempting gpu passthrough Qubes-os on Lenovo ThinkPad W541 i7-4810MQ Secure GPU Passthrough By Having GPU Only For A Qube, Intergreated Graphics For Rest [resolu]Soucis avec GPU passthrough What is my GPU really here for? Truely understanding VCPUs and fully ultilizing GPU Qubes gpu passthru causes crash and restart Qubes gpu passthru causes crash and restart Is this article still relevant? Can't start any qube - PCI Device doesn't exist gonzalo-bulnes August 30, 2023, 9:47am 2 # taradiddles: In my case, it is: [...] This guide seems to draw heavily on this article by @neowutran, should some credits be added? (Asking you directly @neowutran since you're in the forum.) https://neowutran.ovh/qubes/articles/gaming_linux_hvm.html Unless, I've got this backwards and it's the other way around! @taradiddles if I understand the migration correctly you created the guide in the qubes-community project on GitHub? neowutran August 30, 2023, 9:56am 3 I published it both on my website and in the qubes-community project on Github, you can check in the footer of the original post that I am the author: " * Original author(s) (GitHub usernames): neowutran * Original author(s) (forum usernames): @neowutran " everything is ok :slight_smile: 1 Like gonzalo-bulnes August 30, 2023, 10:03am 4 I didn't realize there was such useful info in the details of the migration! :bulb::bulb::bulb: 2 Likes Is using an outdated, old or unsupported GPU a security risk? How do Hardware vulnerabilities affect Qubes OS? Xrandr HDMI Disconnected solene August 30, 2023, 2:27pm 5 Pretty cool guide! Did someone ever try with a thunderbolt external GPU? deeplow August 30, 2023, 2:49pm 6 # taradiddles: Pulseaudio So you need to configure pulseaudio for Xorg multiseat. The archlinux documentation explain that very well: Xorg multiseat Use the option without system-mode deamon and adapt it to qube: Add the following line to /etc/pulse/qubes-default.pa @neowutran I got this working more easily with pipewire. And since there was recent work by @Demi in adding support, I think it's a good bet. (Disclaimer: this was on manjaro. Not pure arch) Here are the instructions: Getting audio to work In my case I was having issues with audio. It was stuttering, so I removed the qubes-vm-pulseaudio package and and installed pipewire as a replacement and it seemed to do the job: sudo pacman -Rdd qubes-vm-pulaseaudio pulseaudio sudo pacman -S pipewire-{jack,alsa,pulse} After restarting the machine, the audio was working 2 Likes neowutran August 30, 2023, 6:08pm 7 I just tried it but it doesn't seems to work with multiseat Xorg. Do you confirm that for you it work even in the multiseat Xorg ? ( audio work well on my ":0" xorg session, but no sound on my ":1" xorg session. I removed all the configuration/modification related to pulseaudio ) Update: made it working on my side with pipewire. I needed to use this custom script instead of directly i3 in my example to force connect to the audio daemon i3.sh #!/bin/bash /bin/sudo -u user PULSE_SERVER=unix:/run/user/1000/pulse/native /usr/bin/i3 1 Like solene September 10, 2023, 3:35pm 8 I just got an HVM to display GNOME on my external GPU :+1: I did that (I'll try again from a cleaner environment to get a more reproducible guide) * hide the pci devices (there is no IOMMU on the external GPU, except with the AMD card but it's super clear to see the AMD VGA and AMD audio are part of the same group) * pass the devices to an HVM * use the qube provided kernel * use X :1 -configure to generate an xorg.conf file, then remove all extra in it to just keep the screen/monitor/device related to the VGA card itself was good enough, check you are using nvidia for NVIDIA cards and amdgpu for AMD cards * move that file in /etc/X11/xorg.conf.d/99-xorg.conf, and start your display manager, e.g. systemctl start gdm.service Tested on OpenSUSE tumbleweed with an AMD RX 480 and NVIDIA 1060 :ok_hand: Tested on OpenSUSE tumbleweed with an NVIDIA 1060 but in discrete mode (display in dom0 but uses the GPU) :ok_hand: Tested on Debian 12 using the template with an NVIDIA 1060 :ok_hand: I used a Razer Core X external GPU case connected in thunderbolt 3 on a Lenovo T470, unfortunately it's almost useless in this case because the CPU is way too slow to do anything meaningful with a GPU in Qubes OS :smiley: 7 Likes External GPU with passthrough? VRAM within a qube for first person games, or other graphically "heavy" loads Xorg errors while attempting gpu passthrough renehoj September 29, 2023, 8:52am 9 # solene: unfortunately it's almost useless in this case because the CPU is way too slow to do anything meaningful with a GPU in Qubes OS :smiley: I'm sure using QubesOS doesn't improve the CPU performance, and mobile CPU are not great for gaming to start with, but do you not also run into some serious PCIe bandwidth issues? Can you get more than 4x PCIe lanes?, I also think TB3 and 4 only use PCIe gen 3, and it's not the fast CPU lanes, it's the slower chipset lanes. solene September 29, 2023, 9:09am 10 Not really, on Linux I can emulate Switch games using Yuzu or play games like Death Stranding or Control (CPU bound). The laptop has an i5-7300U, it would take a while before being limited by thunderbolt bandwidth. Maybe if you use a 4k screen this would saturate faster? If you use the eGPU as a discrete GPU, the bandwidth is limiting because the data have to go in both ways, if you use it with an external display, you have more bandwidth because the rendering doesn't need to go back through thunderbolt. renehoj September 29, 2023, 10:10am 11 Yes, it works if you just use it for the external display, but you could do the same with a TB dock without the GPU. I thought you wanted to connect the GPU and use it to play games that need accelerated graphics. solene September 29, 2023, 10:17am 12 # renehoj: I thought you wanted to connect the GPU and use it to play games that need accelerated graphics. that's exactly what I've done, and it's working. 1 Like oneuse October 14, 2023, 9:37pm 13 Great guide but I still have some questions. For the Iommu Group do I have to do everything inside the grub cmd of the usb linux live distro (I have to paste the #!/bin/bash part)? And if yes how can I deal with my OS being LUKS encrypted? Also, what's the deal with the max-ram-below-4g? From what I understood this means this will allow up to 2 GB of ram to your gpu passtrough HVM qube, then I don't understand how people in other threads have straight up 4090s working. (And what about the VRAM?) yexo October 21, 2023, 11:34pm 14 bump, I am also interested neowutran October 22, 2023, 6:37am 15 "For the Iommu Group do I have to do everything inside the grub cmd of the usb linux live distro (I have to paste the #!/bin/bash part)?" : No. Boot into any linux live distro, use the standard terminal emulator to create a script and execute it. "And if yes how can I deal with my OS being LUKS encrypted?": Irrelevant, you don't need to access anything on your OS for this step "From what I understood this means this will allow up to 2 GB of ram to your gpu passtrough HVM qube, then I don't understand how people in other threads have straight up 4090s working. (And what about the VRAM?)": No, you have access to all the ram you want and don't have any kind of limitation on VRAM. Also, what's the deal with the max-ram-below-4g?: You can read differents threads here: # Quick howto: GPU passthrough with lots of RAM Community Guides I am confused. If I have HVM without GPU passthrough, but other PCI device passthrough, what should I do to have it running with 8GiB memory and no "No bootable device" error? # Windows 10: no bootable device General Still confused. I have a situation: a qube named Windows10 without any patches or any modifications to stubdom or xen.xml without GPU pass-through, USB Controller being attached via qvm-pci in turned off qube state, memory is set ot 8GiB. This qube does not start with "No bootable device". I want: to start this Window10 qube. What should be done in this simplest scenario to make it work? Is it possible? If I detach persistent USB Controller or if I set memory under 800 MiB - qube starts.... # Possible to Fix 2048MB RAM Limit for NVIDIA GPU Passthrough to HVM? Hardware Issues I don't have a setting for resizable bar, but I am not sure if it is enabled or disabled by default and I couldn't find out from the Lenovo website or a search. Here is the output of lspci: 00:07.0 VGA compatible controller [0300]: NVIDIA Corporation GA106M [GeForce RTX 3060 Mobile / Max-Q] [10de:2560] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Lenovo GA106M [GeForce RTX 3060 Mobile / Max-Q] [17aa:3803] Physical Slot: 7 Control: I/ O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParE... or search on internet. Not everything is understood about what is going on with TOLUD 2 Likes deeplow October 22, 2023, 1:21pm 16 Btw, I did a presentation about my video editing setup on Qubes, which has GPU passthrough on an nvidia 4090, if anyone is interested. Timestamp 5h34m30s [f8118d8c948b5ac0782c2286a97d54e119361270] 11 Likes IOS November 15, 2023, 6:50am 17 The lines for the original post on patching xen.xml has changed because the original patch interfered with audio VM's. Maybe update the lines and link to the thread in case more progress is made? github.com/QubesOS/qubes-issues AppVM with GPU pass-through crashes when more than 3.5 GB (3584MB) of RAM is assigned to it opened 06:48PM - 19 Sep 18 UTC pqyptixa pqyptixa T: bug C: core C: Xen P: default r4.1-dom0-stable diagnosed affects-4.1 affects-4.2 ### Qubes OS version: R4.0 ### Affected component(s): Debian 10 AppVM -...-- ### Steps to reproduce the behavior: So, I finally could pass a GPU to a VM with PCI passthrough some weeks ago, but for some reason, I can't get it to run with more RAM. I asked #qubes_os @ freenode about this issue, and ended up making a guide on how to get GPU-passthrough done. I'm not going to post the complete guide here, so here's a link to it: https://paste.debian.net/1043341/ . I don't mind if someone else posts it here or elsewhere, but hopefully people will discuss it and improve it. My system: Qubes 4.0 running on a AMD Ryzen 1600 CPU + MSI B350 Mortar motherboard (BIOS 1E0) + ASUS RX560 2GB GPU for the VMs + an old NVIDIA 610 GPU for dom0 (as secondary GPU). ### Expected behavior: AppVM runs just fine with > 3.5GB of RAM. ### Actual behavior: AppVM crashes with more than 3.5GB of RAM. The way it crashes depends on how much memory is assigned to it, but I usually see almost-instant kernel panics, systemd-init crashing, fsck failures and/or other random processes crashing, too. ### General notes: Other than removing "nomodeset" from the kernel command line, everything is set as default. PS: link to the guide expired. Here a a couple of similar guides: https://github.com/ Qubes-Community/Contents/blob/master/docs/customization/ windows-gaming-hvm.md and https://neowutran.ovh/qubeos.html neowutran November 17, 2023, 4:16pm 18 I am going to remove the information about xen.xml patching from this guide. * The two method (xen.xml / stubdom-linux-rootfs.gz ) are currently not doing the exact same things * The difference between the two methods seems to be causing confusion / I see lot mistakes in forum posts ; and it make troubleshooting harder * I don't personally use the xen.xml method and I am not willing to spend time updating xen.xml patching method to have the exact same behavior as the other method 4 Likes IOS November 26, 2023, 3:55pm 19 Are you still having to keep stubdom downgraded to apply the stubdom patch or did they resolve that? neowutran November 26, 2023, 4:26pm 20 I am using the latest version of everything available in the official repositories next page - * Home * Categories * Guidelines * Terms of Service * Privacy Policy Powered by Discourse, best viewed with JavaScript enabled