[HN Gopher] Show HN: We're open-sourcing our session replay tool
___________________________________________________________________
Show HN: We're open-sourcing our session replay tool
Hey HN! We're open-sourcing highlight.io
(https://github.com/highlight/highlight), a session replay and
error monitoring tool. Highlight.io gives you a high-precision
video-like replay of what users are doing when an error or
exception occurs in your web app, along with a full-fledged error
monitoring experience (similar to bugsnag, rollbar, etc..). The
main value prop of highlight.io is that we help you understand the
full context surrounding an error and allow you to drill down to
the code path that a user invoked (i.e user clicked button X, sent
network request Y, and backend code Z was executed). Some of our
customers compare this to a "web debugger" of sorts. A picture of
what this looks like in our app is here [1]. For some background,
when we worked at our previous companies as engineers, we
encountered hard-to-reproduce issues spanning across both the
frontend and backend. The main issues were (1) if a customer
complained about a problem, it was hard to reproduce the issue
without asking for a screen-share or jumping on a video call; and
(2) when viewing errors caught by tools like BugSnag or Rollbar,
understanding the triggered code path required stitching together
logs, errors, and trace; all from different sources. Highlight.io
is completely open source and written in Go and Typescript. To
build the replay capability, we use an open source project called
rrweb [2] and have worked closely with their team to add support
for features like canvas recording, shadow dom recording, and more
[3]. Beyond that, we use the OpenTelemetry spec for our SDKs [4],
which has made it pretty straight forward to support several
languages, even with our small 4-person engineering team! Our
product is completely self-serve at app.highlight.io. Installing it
is as easy as a npm/yarn import and installing the backend sdk of
your choosing. In addition, given the privacy-centric nature of
session replay, we also offer the option to self-host [5].
Highlight.io currently makes money off of our hosted offering, and
our self-hosted deployment is completely free. We're also toying
with the idea of an "enterprise" self-hosted deployment, similar to
gitlab's billing model, and thoughts from the community on this
front would be appreciated! And as far as what's next for us: Our
customers are asking to render logs and traces on a highlight.io
session (and vice versa), and we're excited to be going deeper into
a developer's debugging stack. The long term goal is to build a
platform that connects replay, errors, logs and more so that
engineers can "playback" the full state of a web application.
Overall, we're quite new to the open source scene and would love
the HN community to share their feedback on what we're building. If
anyone has opinions on where we're going, or what they'd like to
see in an open source monitoring product, we're all ears. Check us
out at highlight.io and at github.com/highlight/highlight to give
us a shot. [1]: https://www.highlight.io/docs/getting-
started/frontend-backe... [2]: https://github.com/rrweb-io/rrweb
[3]: https://highlight.io/docs/general/product-
features/session-r... [4]: https://opentelemetry.io/docs [5]:
https://www.highlight.io/docs/general/company/open-source/se...
Author : podoman
Score : 148 points
Date : 2023-02-22 16:02 UTC (6 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| a_c wrote:
| Super interesting project! I'm working on a related/similar
| project that I'm curious about the technical aspect. Would you
| mind sharing how are you simulating the reconstructed screen? Do
| you construct the DOM, together with the CSSOM, on every browser
| events, and send the result to the backend for replaying?
|
| And do you monkey patch the fetch API to intercept request?
| Apology for not looking thorough enough on github!
|
| How would you compare yourself with tool like openreplay?
| Disclaimer, I am not related to them.
|
| An HN comment is not completed without a unsolicited feature
| request: a demo site, maybe using highlight.io itself, and let
| user replay their interaction.
|
| Good luck building!
| vadman97 wrote:
| Hey! Sure thing. Our frontend session replay happens in two
| parts:
|
| 1. The HTML DOM Recording using rrweb
| (https://github.com/rrweb-io/rrweb) 2. The monkey patching of
| fetch, xhr, console methods, combined with data from
| `window.performance`
|
| The rrweb repo has a good in-depth explanation of how the DOM
| recording works. In short, it captures the full HTML of a page
| when a user first loads it, then tracks DOM changes via the
| `MutationObserver` api. From our client, these are all shipped
| to our backend as serialized events. We process the events per
| 'user session', storing them temporarily in redis, then
| compressing a permanent payload once the session no longer is
| sending new events in the local filesystem or S3.
|
| Monkey patching network and console methods allows us to
| capture request/response payloads, headers, status code, etc.
| We then combine that data with the `window.performance` api's
| notion of network requests to ensure we capture all requests
| (even ones that happened before our monkey-patch had time to
| apply), as well as to get precision timing data.
|
| Happy to give more detail about parts of our stack if you're
| interested!
| vadman97 wrote:
| Following up regarding the other points I missed. Compared to
| OpenReplay, we have similar functionality in our session
| replay, but we've focused a lot of effort on making a cohesive
| error monitoring (backend and frontend) experience. We closely
| link sessions with errors (stacktraces and associated metadata)
| and vice versa to make it easy to get to the root cause of a
| bug.
|
| To demo your own sessions on highlight, you can easily spin up
| our next.js example app that has highlight installed:
| https://github.com/highlight/nextjs-13-sample. All you need to
| do is make an account on highlight.io and copy your project ID
| over into the sample app!
| a_c wrote:
| Thanks for the follow up. Definitely checking up the demo.
| All the best to your endeavours!
| TommyDANGerous wrote:
| We use the cloud hosted version and love it. Hope the OSS version
| is easy to set up.
| vadman97 wrote:
| We've put a lot of effort already to make the OSS setup a
| single `docker compose` command despite the extensive infra we
| use under the hood. Longer term, we're planning to support
| other deployment strategies such as kubernetes, cloud
| templates, and openshift templates. Appreciate any feedback on
| the self-hosting end here or on our GitHub issues!
| jpatel3 wrote:
| What applications are good candidate for highlight to be used? is
| there any breakdown of existing client/partner base to learn more
| about in which domain it gets used most and how.
| podoman wrote:
| Any web-based application could be a good fit! We also support
| a lot of the gnarly session-replay features that a lot of other
| vendor don't: namely canvas recording, shadow dom etc.. (see
| https://www.highlight.io/docs/general/product-
| features/sessi...).
|
| In terms of audience, most of our customers are startups, and
| the users at those companies are often engineers or technical
| product people!
| seanw265 wrote:
| What do you say to potential customers who are concerned with the
| legality and privacy challenges associated with session replay?
|
| I recognize how tools like this could level up product/marketing-
| level decision-making. But with litigation as recent as October
| 2022, I'm wary of incorporating something like this into any
| software I control.
| podoman wrote:
| In the past, for customers in healthcare and fintech, we've had
| to actually pass on signing more restrictive contracts (BAA's,
| etc..) due to the liability. For such a small company, it was
| hard to be comfortable with situations like that.
|
| Since then, we've done a couple things that will hopefully open
| up more opportunities to work with these sorts of companies.
| First of all, we have a "privacy by default" mode that
| obfuscates all text by default (see
| https://www.highlight.io/docs/getting-started/client-
| sdk/rep...). And secondly, we're hoping that open sourcing the
| tool will help these companies control the data they're
| collecting so they don't run any risks, and long-term, these
| companies pay us a license of sorts.
| zeeg wrote:
| I think this is a great question, especially with the continued
| shift towards data residency and customer protection in recent
| years (GDPR, CCPA).
|
| When we did this at Sentry - which is also built on rrweb (we
| help fund rrweb btw, everyone else should too!) - we realized
| privacy was really hard to get right (let alone data security),
| so we just block all elements that are at risk by default and
| it turns out the experience is still pretty usable. Ben wrote a
| bit more about this here:
| https://blog.sentry.io/2023/02/16/introducing-session-replay...
|
| Not commenting on highlight, but most of the mainstream players
| in this space take an opt-in protection approach, and often its
| "you can block this specific HTML element".. which glhf getting
| right
| podoman wrote:
| Highlight also has a private by default mode. Totally agree
| that this is the right way to do it. Also, congrats on the
| recent launch :)
| mdaniel wrote:
| I wanted to say thank you for picking a business friendly
| license!
|
| Watch out for the (presumably old) GitHub organization in
| .gitmodules:
| https://github.com/highlight/highlight/blob/d818969acddd7c74...
|
| OT1H, GH does a 301 to the new name, but OTOH unknown how long
| that lasts and is one more thing to go off the rails for fresh
| clones. It has been my experience that using the https URL makes
| clones easier for users which may not have set up ssh auth, but
| it depends on the target audience and whether you have some
| internal tooling that depends on that submodule coming down over
| ssh
| vadman97 wrote:
| That's a great point! Just opened a PR to fix this:
| https://github.com/highlight/highlight/pull/4362
|
| Switching this to https too since the our rrweb fork is now
| public (ssh reference was previously necessary for cloning a
| private fork).
| verghese wrote:
| Could you tell me how this compares with other open-source
| session replay projects such as: 1) OpenReplay
| (https://openreplay.com/) 2) PostHog (https://posthog.com/)
| podoman wrote:
| Having used the Posthog product (which is really good btw!),
| its an "analytics" tool, so their replay product is very good
| at mapping user funnels to sessions.
|
| On the OpenReplay end, its seems we have feature parity for
| session replay, but they don't have support for error
| monitoring and therefore linking errors to corresponding
| sessions (as far as I can see).
|
| Nonetheless, happy to see other oss tools in this space!
| lharries wrote:
| PostHog started as a product analytics tool but we now have a
| fully fledged session recordings product including handling
| console logging and network requests for debugging. Mobile
| support is coming shortly too
|
| Disclaimer: I'm Head of Product at PostHog
| podoman wrote:
| Very cool. Didn't know y'all were investing in that. Always
| been a fan of posthog.
| lharries wrote:
| Thanks! And congrats on the move to open source
| podoman wrote:
| Thank you sir!
| tough wrote:
| I didn't think postHog was a good comparison either,
|
| I knew of rrweb https://www.rrweb.io/
|
| Great to see more open source contendants in the space
| podoman wrote:
| Thanks. Yeah, we actually use rrweb under the hood and have
| met with a few of the core maintainers. We're proudly
| powered by that project.
| kc10 wrote:
| Posthog also seems to be using rrweb.
| a_c wrote:
| On debugging side, sentry (recently?) rolled out replay too
| https://docs.sentry.io/product/session-replay/
| [deleted]
| bentlegen wrote:
| We launched this last week! Right now it's only available on
| SaaS (sentry.io), but we've been developing it in public in
| our main server repository (github.com/getsentry/sentry) and
| plan to have proper instructions for self-hosted users soon.
| dnsmichi wrote:
| GitLab team member here.
|
| Thanks for sharing!
|
| > We're also toying with the idea of an "enterprise" self-hosted
| deployment, similar to gitlab's billing model, and thoughts from
| the community on this front would be appreciated!
|
| Transparency and an open stewardship of the project can help:
| https://about.gitlab.com/company/stewardship/ Make it clear what
| features are paid, and define that for example free features are
| never moved to the paid tier.
| podoman wrote:
| That's a good point; we should be clear about what will always
| be free. Right now, literally everything is free, but what we'd
| consider putting behind an enterprise tier that are top of mind
| are: - multi-tenancy - enterprise authentication features (sso,
| saml, etc..)
|
| Re: the gitlab link you sent, we really like the transparency
| around not making a free feature paid, and the fact that all
| tests are OSS.
|
| Overall, we'll work on something like this. Thanks for pointing
| it out!
| latchkey wrote:
| Please don't put SSO/SAML behind a paywall. Security should
| be a free priority.
| rsavage wrote:
| Please don't put basic features behind enterprise licences.
|
| We are a small start up. We use SAML / SSO. We care about
| security. We care about privacy.
|
| None of this is "enterprise".
|
| We stopped using competitors to you because of data privacy
| issues.
|
| We would happily pay for a self host version, where we
| control all the data recorded.
|
| These type of tools are extremely useful, and it hurts not
| having it. But they are just way too much of a privacy issue.
|
| Self hosted and paid - is the future of data privacy.
|
| I would love to see you (and others) off these options, it's
| really a game changer and makes the switch easy from any
| product that is not self hosted and collects our customers
| data.
| zeeg wrote:
| I would encourage you to consider not even trying to cripple
| the freely available self-hosted version. Personally I'm not
| a fan of open core or the GitLab business model, and Sentry
| has shown you can build a business without removing needed
| features from the self-hosted version. While folks may not
| agree Sentry is "open source" these days, our license
| effectively achieves the same from a customer pov as yours
| (which is ideal for customers). More people should take the
| risk IMO.
|
| Ultimately people do not want to run all their own services
| if they can pay someone else to run it for them.
| podoman wrote:
| Makes a lot of sense. We're going to try to take a bit of a
| different approach, but no doubt that a successful business
| (like sentry) can be built w/o the gitlab model!
| jonas-w wrote:
| Keep https://sso.tax in mind
| [deleted]
___________________________________________________________________
(page generated 2023-02-22 23:00 UTC)