[HN Gopher] Ask HN: How relevant is cross-browser testing anymore?
___________________________________________________________________
Ask HN: How relevant is cross-browser testing anymore?
Given that most of the new companies adopt modern frameworks like
React, Angular, Vue which are quite stable across browsers unlike
jQuery, Backbone days when the same code worked in Chrome but not
Firefox for various reasons. Just wanted to pick your brains on
what do you think how relevant is cross-browser testing. Do we
really need to test our code on 50ish Chrome versions, 25is Firefox
versions and so on.
Author : sachinjain
Score : 27 points
Date : 2022-11-16 12:51 UTC (10 hours ago)
| jfk13 wrote:
| > Do we really need to test our code on 50ish Chrome versions,
| 25is Firefox versions and so on.
|
| Probably not, unless you're a really big outfit with a vast user
| base. But you _should_ test on at least a recent Chrome version
| and a recent Firefox version, and probably Safari.
|
| And unless your product is irrelevant to mobile users, on both
| iOS (Safari) and Android (Chrome, Firefox), with both phone and
| tablet form factors.
| TAForObvReasons wrote:
| Painful as it is to admit, even Firefox is not worth testing.
| In our B2B site, more people are using IE11 than FF.
|
| https://gs.statcounter.com/browser-market-share at ~3% of
| overall share / 7% desktop-only share it's hard to justify
| putting extra effort.
| jfk13 wrote:
| It's not just about the number of your users who are directly
| using Firefox; it's also about ensuring your code isn't
| inadvertently depending on non-standard Chrome quirks, and
| promoting standards compliance, portability, robustness, and
| such-like goodness on the web.
|
| (Test IE11 as well, by all means, if it's relevant to your
| users; but it's much less relevant to the future of the web.)
| tinus_hn wrote:
| Possibly Firefox is making it very easy to block your
| statistics gathering. Besides, 7% is still quite a lot to
| throw away.
| patrakov wrote:
| Maybe they are not using Firefox, because your site is broken
| in it, and they have no other way to tell you this?
| l0b0 wrote:
| Yep, it's the classic self-fulfilling prophecy. Even a lot
| of non-techies are aware that there are multiple browsers,
| and will just switch to whichever gets the job done. So if
| you're seeing less % of any specific browser than the
| overall web your site is probably broken on that browser.
| If it's not obvious by now, that means you need to do
| _more_ testing in that browser, not less.
| kfrzcode wrote:
| Survivorship bias!
|
| Yes, this is something I've experienced directly from end-
| user perspective as well as debugging/user-analytics
| perspective.
| falcolas wrote:
| Exactly this happened at the company I work for, and their
| move to only support Chrome was hailed as a revolutionary
| time-and-effort saving affair.
|
| grrr
| helij wrote:
| This. There are some websites that I have to login that are
| completely broken when using Firefox. I login with Brave on
| those and am on a constant lookout for competitors that
| work on Firefox. As soon as I find them I jump ship and
| take my money elsewhere.
| blowski wrote:
| I can imagine how IE11 gets a higher proportion in traffic in
| B2B. In B2C, particularly amongst some niches, I'd guess
| Firefox is still proportionally popular. For example, I'd
| love to see browser stats for HN.
|
| (Disclosure: I am still very much a Firefox user, so this
| might be projection.)
| PaulHoule wrote:
| I develop w/ Firefox and our tester tests with Chrome. I use our
| product quite a bit with an iPad. If problems are reported with
| Safari we fix them.
|
| I've also worked at places where our product was specified to
| work with Chrome, I would try to run it with Firefox and no dice
| and they would not accept any ticket to fix bugs in Firefox.
|
| There's an assumption today that browsers get updated frequently.
| My work computer is a managed Windows machine that I had
| administrator access to, the one complaint I have is that the
| Firefox is a long-term support version which is often long as in
| "in the tooth". Bloomberg would complain about it incessantly
| until I got the new LTS version. Other than that I think browsers
| are usually up to date.
| sachinjain wrote:
| Fixing any issue specific on Safari is a whole challenge on
| another level. Hope you had an easy time debugging issues on
| Safari.
| PaulHoule wrote:
| At work we've never had one that was that bad. In one cases
| it was third-party cookie issues that affected Safari before
| everyone else.
|
| Personally Safari is the reason why some of my image-based
| sites still use JPG in parallel with WEBP. I think they fixed
| it just recently, but if you have an older mac (i do) Apple
| couldn't be bothered to include a WEBP codec with the OS.
| Firefox, Chrome and every other browser for mac supports WEBP
| but not Safari.
|
| I think they resolved this in the last month, but I think for
| better or first Apple wants to roll out new SIMD instructions
| in their libraries much faster than Microsoft does. So at
| some point your CPU is too old. Of course the flip side to
| that in the Windows World is that people are stuck using a 10
| year old instruction set. Intel doesn't seem to get it that
| by not supporting the latest AVX in their latest chips that
| people just "don't use AVX". If the latest SIMD instructions
| were widely supported you'd have a reason to buy a new
| computer, but when they aren't Intel has lackluster sales and
| wonders why.
| sachinjain wrote:
| I have had a very nasty bug that I debugged in Safari.
| Chrome & Firefox supports redirects to http when the site
| is on localhost or 127.x.x.x but Safari doesn't and throws
| a Mixed content warning error.
|
| Another one was similar when redirect chain got broken
| because of an intermediate redirect (https -> https -> http
| -> https)
|
| I guess Chrome & Firefox automatically upgrades the
| protocol now but Safari doesn't and it broke the chain and
| it was also not showing the broken redirected chain in the
| Safari developer tool.
| helsontaveras18 wrote:
| I experienced a DOM issue where a horizontal scrollbar was being
| rendered on some versions of Chrome running on Windows, some
| versions of Firefox on Mac, but I couldn't reproduce it myself
| (Chrome / Safari on Mac). It happens.
|
| I don't QA for these things beforehand, but when they do occur,
| it helps to have a way to perform cross-browser testing to fix
| the bug.
| andrewmcwatters wrote:
| Regardless if you're doing small personal projects or doing
| commercial work, personally I don't find it too time consuming to
| just breakout the latest versions of Safari, Chrome, and Firefox
| and do some local testing.
|
| Any experienced software developer will have some experience
| testing against those. I don't think you need to go out of your
| way to test against specific builds of those browsers, with maybe
| the exception of Safari and Mobile Safari since I have noticed
| standards-compliant differences over time with older iOS devices.
|
| That is to say, there isn't anything wrong with what Safari and
| Mobile Safari do, but the WHATWG living standards have changed
| over time just enough that older Apple devices may render web
| pages differently.
|
| Try not to just limit yourself to Chrome though. And if you're
| using vendor extensions, don't rely on them, just use them to
| pretty up from your baseline designs.
| Macha wrote:
| Safari does require a Mac, so a little more time consuming
| unless you're already a Mac shop
| pacifika wrote:
| I encountered a cross browser mixed implementation bug only
| today, affecting react components.
|
| https://jsfiddle.net/svandragt/236Lhkub/3/
|
| So I'd say yes
| bryanrasmussen wrote:
| > Do we really need to test our code on 50ish Chrome versions,
| 25is Firefox versions and so on.
|
| The only time I've ever seen this intense level of testing was on
| a government project that was producing someone that everyone in
| the country would be required to use. So if that is what your
| situation is yes, you need to do that much testing because if it
| is mandated by law that citizens use it, it better be able to
| handle whatever combination of citizen and tech it encounters.
|
| Otherwise I think you are probably being somewhat hyperbolic
| here.
|
| Aside from that where accessibility is concerned there can be
| significant differences between various combinations of browsers
| and screen readers.
| dutchCourage wrote:
| I don't think it's worth testing on old versions of Chrome and
| Firefox since they're updated automatically.
|
| iOS Safari on the other hand, can't be updated on older phones
| since it's tied to iOS updates. This makes any Safari bug linger
| for a few years. It's also the only rendering engine available on
| iphone so some people are truly stuck using an old version.
|
| So I'd say, test on Chrome because it's ubiquitous and test on
| iOS Safari like it's the new Internet Explorer.
|
| I'm not sure about Firefox. I personally use it and test on it, I
| think it's the right thing to do for the web. And at least with
| that you've covered most rendering engines.
| s1mon wrote:
| I don't disagree that most versions/instances of Chrome are
| updated regularly, but implying that this is universal is
| questionable.
|
| Chrome is only updated if the user lets it happen. I can't
| count the number of times I've watched some co-worker share
| their screen on a conference call and the Chrome update button
| is lit up, but apparently that's not a trigger to these people
| (I update pretty much as soon as I notice the indication since
| Google has done a remarkably good join of updating in place in
| a way that most SaaS things and web sites just reopen without
| trouble).
|
| I'm sure the next time they restart/reboot it will get updated,
| but if there was some hideous Chrome-related bug which is from
| a release a month ago, you might want to think about that. It's
| also entirely possible to turn off updates if you or some
| misguided IT person is really so inclined.
| politelemon wrote:
| It's still relevant - there are enough differences between the
| browsers to render incompatibilities and bugs. It does also
| depend on how serious you are about your product, as some kinds
| of products not working in a browser can leave a very bad or
| frustrating experience.
|
| It is of course painful to test in so many variants, so a decent
| compromise is often to just test in the latest version of all
| browsers. There's enough coverage there to help you find any
| potential problems.
| thepra wrote:
| That's why I use only features that are cross compatible
| between browsers, even though I feel left behind on some new
| features only available on Chrome -_-
| Bilal_io wrote:
| Not much anymore, at least from my experience. When writing
| Typescript, I know what I am targeting and it'll transpire to
| that.
|
| When writing CSS, I rely on caniuse.com when not certain.
| golemiprague wrote:
| I have noticed that there are still vast differences in the
| printing functionality, both in terms of how things render and
| performance. So if you need to do any kind of printing in the app
| and you know your clients will use an array of different browsers
| it is worth checking it. Even within the same browser things tend
| to go back and forth when it comes to printing, Chrome for
| example tends to break things in one version and then they come
| back in the next one. It is a bit of a mess.
| croisillon wrote:
| all your comments are flagged, please write an email to the
| admins to sort this out
| raxxorraxor wrote:
| Not a web developer, but I usually just test with the latest or
| almost latest version of FF and perhaps Chrome.
|
| I personally don't like or use any of the frameworks you
| presented. If I had to pick one I would choose Vue.
|
| jQuery did a lot for inter-browser compatibility, so I don't
| understand the argument. On the contrary, it was much more
| difficult to manage diverging code without it.
|
| 99% of my web programming is displaying some data, a website to
| interface XY. I don't like modern websites at all. Probably more
| a design descision than a framework issue, but I try to minimize
| any browser scripts in the first place and just haven't felt the
| need for somthing like React or Angular.
| sachinjain wrote:
| Hey @raxxorraxor, I am not against using jQuery. What I meant
| was initially back in 2012-2013 days when jQuery used to be a
| default DOM query/manipulation library people used, the same
| code used to break across browsers and we had to use some
| polyfills.
|
| Like you said you usually test with the latest version of
| Chrome/Firefox you are partly agreeing to the statement I made.
|
| I am trying to find someone who's strongly in favour of testing
| their apps across browsers and multiple versions.
| yeputons wrote:
| I have no idea about how relevant it is, but every few months I
| see a website which seem to be tested in Chrome only, because it
| blatantly don't load in Firefox or Safari. Like, at all, blank
| page. Or have some buttons like "Purchase" not working.
| thepra wrote:
| Can't tell you my distress when government websites use Google
| captchas and on Firefox you're stuck and can't go through your
| business...
| WoodenChair wrote:
| When you see this, complain and let them know. I've had some
| success complaining about this to medium sized companies and
| getting updates.
| chunk_waffle wrote:
| My retirement account and healthcare provider both have this
| problem. In one case I had to manually delete a disabled
| attribute on a button to get signed in because something broke
| halfway through a multi-step workflow.
| Mikeb85 wrote:
| Chrome and Firefox auto-update so no.
| tgv wrote:
| Depends on how critical it is for you/your company. I work on
| products that have internal tooling, and if they run on the
| latest version of Chrome, it's good enough (although I personally
| test them on Firefox, too, and someone used an older version of
| Safari the other day, and found a weird bug).
|
| However, some of our tools are used by the general public (I've
| found the user agent for a coupon app as well as some PlayStation
| browser in the logs; might have been spoofed, of course). Those
| are written to the lowest common HTML/CSS/JS standards, so we
| don't try fancy features, and Babel or similar can be helpful.
| IE11 is out of grace, but we still support years old versions of
| most browsers, because it costs money and goodwill when too many
| fail.
| Arubis wrote:
| I consult, frequently jumping between web application codebases.
| Very often, I bring cross-browser testing with me by sheer nature
| of using Firefox when _everyone_ has been using Chrome
| exclusively for years. I end up finding issues; about half the
| time, clients don't care, because all their customers use Chrome.
|
| This direction worries me (we did this with MSIE and it took a
| decade+ to heal those wounds).
| thepra wrote:
| Just tell them Chrome is going to become the next IE if
| everything only works on Chrome
| Macha wrote:
| More and more of them have never had to deal with web
| development in the era of IE
| Izkata wrote:
| Nowadays the biggest issues are with CSS, and those frameworks
| won't help with that.
|
| However, there are other CSS frameworks (Bootstrap/Tailwind/etc)
| that do.
|
| Even using those, I'd still give other browsers a look over to
| see if anything looks odd. One weird one we had was, on
| Chrome/Edge a React dropdown looked fine, but on Firefox it was
| somehow positioned like 80px too high. I don't remember exactly
| what the fix was, just that it was a CSS problem and the original
| developer was relying on a Chrome default they didn't realize was
| there and not in other browsers. Something akin to "position:
| relative; top: 20px;" just happened to look vertically centered,
| but only in Chrome-based browsers.
| polygot wrote:
| It depends on your risk tolerance and how many DAU/WAU/MAU you
| have (and whether you're willing to have an untested experience
| for X users.) For example, I would test really old browser
| versions, and if there is a bad experience, you could let the
| user know to use a different browser or to upgrade.
|
| My intuition says you'd get better value out of testing different
| browsers shallowly rather than 50 old versions of Chrome, though.
| squeegee_scream wrote:
| PeacockTV has problems on Safari but just works on Firefox. This
| is just the most recent example I have of large companies whose
| site doesn't work well on all browsers. So yes I think we should
| test on multiple versions of multiple browsers on multiple OSes
| aww_dang wrote:
| Feels like everything just works on Chrome and Firefox, but
| Safari is the new IE.
| timack wrote:
| Absolutely, feels like almost 100% of browser specific issues
| we deal with are related to Safari.
| yakubin wrote:
| Here is a bug I've recently discovered, which is present only
| in Chromium-based browsers: <https://bugs.chromium.org/p/chromi
| um/issues/detail?id=138298...>.
|
| Doesn't happen on Safari, nor Firefox.
| sagarsoni wrote:
| Most of the cross browser issues arise either from CSS or JS, and
| both of these occur because of older browsers. Since we don't
| have control over people as to which browser they use, devs
| generally tackle this issue by using different stylesheets for
| different browsers conditionally. Modern tools like lambdatest,
| browserstack generate multiple previews easily.
| n8cpdx wrote:
| Please test with Safari on desktop at least. I routinely find
| things that are just completely broken. And not small names, big
| names like Hulu, Clover, green chef, and others have just
| completely broken on Safari. Integration with Google's captcha
| seems to be a common failure point.
|
| Also please consider writing your apps in a way such that they
| don't completely shit the bed if your 600th ad tracker script
| fails to load. I hate having to disable pihole so that Google can
| track me while I pay my utility bill.
| Raed667 wrote:
| Safari is a pain with bugs, unimplemented standards and just
| overall unpleasantness.
|
| I can't see why anyone would still use it today when FireFox
| for Mac has become so good.
| codetrotter wrote:
| Safari is probably better optimised for power consumption on
| macOS than Firefox is. Meaning you'll probably get more hours
| of use with Safari on macOS than with Firefox on macOS, when
| you are running on battery.
| lotsofpulp wrote:
| Battery life, speed, and integration with Keychain is why I
| use Safari.
| lelandfe wrote:
| The simple answer is that Safari ships with the platform and
| few people choose browsers based on standards adherence.
|
| For daily use, it's a pretty normal experience.
| drummer wrote:
| Safari is junk software and should die sooner rather than
| later.
| Vinnl wrote:
| We sometimes hit a bug in Safari which doesn't appear in GNOME
| Web, which is annoying, because that's really the only WebKit-
| based browser that runs on my machine. I'd be happy to better
| support Safari if I wouldn't have to buy new hardware for it :/
| sosodev wrote:
| It is absurd that Apple doesn't distribute Safari to other
| desktop platforms or at least make it source available so the
| community could compile it and fix bugs for themselves.
| jitl wrote:
| There was Safari for Windows back in the iTuned era.
| Macha wrote:
| iTunes would even try install it with iTunes updates at
| one point.
|
| The fact that they ported the Mac font rendering stack to
| windows for it always made it stand out like an ugly
| duckling though
| hebrox wrote:
| We're using Playwright [0] to do e2e testing. Locally I just run
| Chrome, but on CI FF and Safari are used as well. Using the
| official Docker image, this was really easy to setup. Because all
| browsers are automatically updated nowadays, we only test the
| most recent versions.
|
| A recent issue we found, was that Safari was responding different
| to some HTTP headers.
|
| [0]: https://playwright.dev/
| Venezio wrote:
| True
| sachinjain wrote:
| We might be wrong here and missing some valid point. I see
| companies doing automation testing beyond a point but do they
| really test beyond 2 browsers?
|
| One line of thought could be if we have already invested in
| automation testing then why not run in 80 browsers too when we
| can. Its just additional hardware cost or cost of platforms
| like BrowserStack/Lambdatest.
___________________________________________________________________
(page generated 2022-11-16 23:02 UTC)