[HN Gopher] Launch HN: Preflight (YC W19) - Better automated tes...
___________________________________________________________________
Launch HN: Preflight (YC W19) - Better automated testing for web
apps
Author : mesadb
Score : 27 points
Date : 2021-12-14 13:35 UTC (9 hours ago)
(HTM) web link (preflight.com)
(TXT) w3m dump (preflight.com)
| infocollector wrote:
| Can you please add a pricing page to your website? Also, can one
| run these tests with an api key on ci/cd?
| mesadb wrote:
| Hi, the pricing is tricky part in any startup. We have some
| smaller plans and bigger plans. It really depends on what your
| needs are.
|
| Absolutely, integrating with CI/CD is super easy. We have some
| integrations already with major CI/CD pipelines. You can find
| it right here after signing up:
| https://app.preflight.com/account/integrations
| kierenj wrote:
| What are the prices for the various plans? :)
| [deleted]
| mesadb wrote:
| It's here now: https://preflight.com/pricing - thanks
| everybody for the clear feedback.
| dubcanada wrote:
| This is the wrong mindset, it does not depend on what my
| needs are. It depends on what you can offer me. My needs are
| meaningless, maybe I need it to also monitor uptime. Are you
| going to build that un-needed featured because I need it.
|
| It should be about what you can offer, and I can determine if
| that fits my needs.
| neeleshs wrote:
| Can it work with canvas based drag/drop apps? Would be very
| interested, if it can
| mesadb wrote:
| Hi @neeleshs, we are actually about to launch that. We will use
| Computer Vision to find the elements and drag and drop. Would
| love to help you with that or drop you an email when we launch
| it. It should be live next week or so
| neeleshs wrote:
| Thank you! Will sign up
| claytongulick wrote:
| I actually need this too right now, but without a pricing page I
| just move along.
|
| I'm bombarded by vendors every day trying to "recapture" a sale,
| as part of an automated sales pipeline process.
|
| I rarely provide contact details prior to being able to:
|
| 1) understand the product, what it does, and whether it will work
| for me
|
| 2) understand the pricing and sla
|
| 3) understand the business continuity model
|
| I'd rather have a more difficult solution (selenium) that I know
| the above three points about than to get sucked into yet another
| sales pipeline.
| [deleted]
| mesadb wrote:
| It's here now: https://preflight.com/pricing
| endisneigh wrote:
| I don't understand the OPs attitude around publishing basic
| information without signing up.
|
| Here's the pricing for those who are curious:
|
| Starter - 300 tests/month $99 /mo
|
| Startup - 1,000 tests/month $300 /mo
|
| Competitors are cheaper and include more tests.
| mesadb wrote:
| Yeap it's right here: https://preflight.com/pricing (Edit: just
| added.)
|
| Which competitors? Where are they running the tests on? Do they
| adapt to changes and save your valuable time? Or charge small
| and run tests with selectors like: .wudfj289hf .a9dfh82hf
| .sadf98h which breaks each time you deploy?
| endisneigh wrote:
| > Yeap it's right here: https://preflight.com/pricing
|
| pretty misleading for you to act like that page was already
| there. you just added it. your lack of transparency is pretty
| disturbing.
|
| EDIT: wow OP got flagged and deleted their original reply,
| lol.
|
| cheaper options (just YC, not counting non-YC)
|
| https://www.superadmin.so/
|
| https://reflect.run/
|
| https://www.rainforestqa.com/
| [deleted]
| mesadb wrote:
| Sorry - you're right of course - I was just scrambling to
| reply to everyone and didn't mean to leave a misleading
| impression. I've edited my comment to be clearer now.
| easton wrote:
| Why Windows/macOS and not Linux for the test runners?
| Chrome/Firefox runs just fine on Linux, and then you don't have
| to pay for licenses. (I suppose if you get one of those resale
| agreements with MS you wouldn't have to either as long as you
| bill the customer, but as the customer I don't care about what OS
| the browser is running on.)
| mesadb wrote:
| Thank for your comment. I see your point. Linux is fine for
| functional testing. But we just want to test how your users use
| your app. Our end goal is to use keyboard and mouse on OS
| level. And adapt to the changes. Some of the functionality and
| fonts don't work on linux so it's a little tricky.
| depressedpanda wrote:
| That's unfortunate.
|
| Even on GitHub -- owned by Microsoft -- Windows runners are
| twice the cost of Linux runners (macOS is 10x). Given that, the
| extra unnecessary cost is almost certainly pushed onto the
| customer.
| mesadb wrote:
| Yeah I didn't like that news as well. But most of the users
| are on MacOs and Windows. So we really want to test how your
| users use your app. We have plans on adding linux but it will
| take longer.
| beingflo wrote:
| I'm actually in the market for a product like this right now. But
| I genuinely don't understand why you wouldn't publish your
| pricing, that's an immediate no-go for me.
|
| Just a couple of days ago I started trying checkly [1] and have
| been quite happy, although I didn't try their browser testing
| yet. What do you offer over other services like this?
|
| [1] https://www.checklyhq.com/
| tnolet wrote:
| Checkly CTO and founder here. Thanks for trying Checkly. Just
| looking at Preflight, I think the audience might be different.
| Where we -- Checkly -- focus heavily on "code first / developer
| first", Preflight seems to target no/low code.
|
| For us, code-first / dev-first means we directly integrate with
| popular JS frameworks (like Playwright and Puppeteer), allow
| you to code any setup / teardown scripts in NodeJs and
| generally want you to treat your testing and monitoring setup
| "as code".
|
| So, TLDR is that I would 100% give Preflight a try and see if
| it matches your workflow. Always choose the best tool for your
| problem!
|
| P.S. we do help with generating test scripts with our Chrome
| extension https://github.com/checkly/headless-recorder
| mesadb wrote:
| If you sign up, and you should see the prices
| dubcanada wrote:
| Agreed, anything without a pricing page is an immediate no
| for me. Looked for pricing, did not see it. Closed page.
|
| I don't care at all about anything else, as soon as the
| pricing page is hidden means it's not affordable to me.
| mesadb wrote:
| I listened to all of you and put a page on the website:
| https://preflight.com/pricing
| runako wrote:
| Coming here to second this. I am not signing up to find out
| how much it costs, and I am actively in the market for a tool
| like this.
|
| Why won't I sign up to find pricing? Because gaming pricing
| makes a bad impression and makes me think you will try to
| obfuscate other critical information once I become a customer
| and generally be a difficult vendor to work with. Nobody
| needs to add a difficult vendor, especially in a competitive
| category.
|
| Edit: there are 6 top-level comments on this post besides
| yours. 4 of the 6 are looking for pricing.
| mesadb wrote:
| I respect that. But If you don't have a serious pain point
| around testing and you want to miss the best option out
| there. It's fine by me. There are couple of other
| alternatives out there: reflect.run, rainforestqa.com,
| ghostinspector.com, mabl.com, testim.io.
|
| We don't obfuscate, we care deeply about our clients. I
| always say this. If Preflight doesn't add value you don't
| have to pay for it nor use it.
| simlevesque wrote:
| > you want to miss the best option out there
|
| Part of my first impression of your product is the fact
| that you have something to hide. I want to trust the
| products I use.
|
| Also, my time is precious. I can't count the number of
| times I wasted time by looking at a product to find out
| the pricing makes no sens at all.
| mesadb wrote:
| I expressed that I heard you so I put the pricing up in
| about an hour ago: https://preflight.com/pricing
| runako wrote:
| I shouldn't have had to say this, given that this is HN
| and comments tend toward the helpful, but: I meant my
| comment in the sense of helping you to improve your
| offering.
|
| If I didn't care, I wouldn't have taken the time to write
| a comment as if I had a stake in your company beyond just
| wanting entrepreneurs to succeed. If I had seen this on
| Twitter, I would have just kept scrolling. But HN has
| given a lot to me, so I try to help others.
|
| So in that vein, one more constructive criticism: learn
| to take feedback better. The best customer feedback often
| comes packaged in vitriol and is much more unpleasant
| than anything posted here. (Especially from support
| tickets. People are frequently angry when they write
| them, but they also provide invaluable insight into your
| product.) You'll have more success if you gracefully
| accept all critiques, both warranted and unwarranted,
| regardless of how gently they are phrased.
|
| Good luck!
| beingflo wrote:
| I hope you take the critical comments here
| constructively, we just think that you're missing out on
| a lot of potential users by not being transparent with
| pricing. As the other comment points out, we've come to
| expect that no pricing page means sales contact etc.,
| individual customers are typically not interested in
| that.
| mesadb wrote:
| Yes you are completely right. I didn't know everyone
| would expect the pricing on the website. I quickly
| created a page and released it right here:
| https://preflight.com/pricing
| au-arms wrote:
| Love the activity in this space. We are on the lookout for a tool
| like this with built in runtime performance monitoring (frame
| jank, composited layer size, long tasks, component render times,
| etc). Would happily pay for a service that could do so.
| mesadb wrote:
| Hi HN! It's Mustafa. I'm the founder of Preflight
| (https://preflight.com). Preflight is a no-code tool that
| automates manual web app testing. It lets everyone test the
| product by simply recording actions in the browser with a Chrome
| extension. Here's a demo video:
| https://www.loom.com/share/4b09462bf52942e28b0dc71c9c0c319a.
|
| I was the first engineer at ShipBob and ended up hiring over 40
| engineers. We had a lot of UI issues after we grew beyond a team
| of 5 or so. With all the engineers adding new code to the
| existing code, everything started to break.
|
| That's when we started writing Selenium tests. I and several
| other engineers wrote 100 Selenium test cases in 2 months. But we
| were not able to maintain them in 3- 4 weeks because we had other
| responsibilities, and our Selenium test cases soon went to trash.
|
| The problem with automated UI tests with Selenium or Cypress is
| that they depend on the internals of the design (e.g. via CSS and
| XPath selectors), resulting in tests that break frequently
| because of unrelated changes. When that happens, you either have
| to fix all the tests and run them again, or replace the broken
| automated tests with manual testing.
|
| Then I hired 2 QA engineers, they took 5 months to come up with
| 5% test coverage and we needed more testing so we couldn't add
| more to it and maintain those tests as well. These issues pushed
| me towards creating a tool for automated testing that could
| understand the context of the page--the way humans do, not the
| way browser engines do--and can adapt to changes based on each
| steps' context. So if the UI changes completely and the test is
| looking for a Password Input in a sign-up form or an Orders menu
| item in the menu, we will find it without any related selectors.
|
| As humans, we understand the page in terms of buttons, menus,
| headings, borders, and so on. I wanted to mimic that. On top of
| that we have implemented a way to understand navigations through
| these different elements with their context, again, as humans do.
| We call this Context-Awareness. It's our secret sauce to adapt to
| the changes and also calculate what your test coverage is. Since
| we understand the context of each step, it's pretty easy to see
| if, say, a given button is in the tests.
|
| Here's a video that shows the Context Awareness aspect I just
| mentioned:
| https://www.loom.com/share/50426244253943e49906dd881e0fb7d7.
|
| You record your tests using our browser extension, and then we
| run them on a bunch of Windows 10 machines that access your web
| app over the internet and execute the tests you recorded. We're
| working on adding Macs. (Fun fact: Selenium cannot handle more
| than 40 nodes. The first version of Preflight was depending on
| Selenium Grid and it stopped working when we wanted to add more
| than 40 machines to it.)
|
| We charge money based on how many tests you are planning to run.
| We don't want to limit you based on server time because it's not
| easy to predict how much server time your tests will take. So
| everything is based on test results.
|
| Anyone can sign up and start using Preflight in minutes after
| signing up at https://app.preflight.com/get-started.
|
| Thank you for reading. I'd love to hear your thoughts and help
| all of you with any of your QA needs!
| danr4 wrote:
| Pricing: https://i.imgur.com/Hhv747C.png
|
| $99/mo 300 tests / month
|
| $300/mo 1000 tests / month
|
| Aside from hiding prices, the sign up process has another
| annoying dark pattern - you sign up, they prompt you to schedule
| onboarding call (only option, no other button), then if you click
| anywhere or wait a few seconds a "Continue to Dashboard" button
| appears. If you click it you have an "Are you sure?" button
| instead.
|
| Instant turn off.
| mesadb wrote:
| Thank you for making my life easier. It's here now:
| https://preflight.com/pricing
|
| We want to make sure we help you. Testing is not easy even
| though we try everything we can to make it easy.
| wesleyfsmith wrote:
| Seems really promising, but how does it work with browser
| extensions? For instance, I'm looking for something like this
| that will help me test my crypto app that uses metamask at
| various points through the app.
|
| Also, no public pricing page is a bit of a turn off, especially
| when I created an account and still couldn't find any pricing
| info.
| [deleted]
| mesadb wrote:
| Hi Wesley, you should be able to see the price on here now:
| https://app.preflight.com/account/billing. Sorry if we made a
| mistake
|
| It should be no problem with MetaMask.
|
| Edit: Also listened to everyone and put the pricing on the web
| page: https://preflight.com/pricing
| kanwisher wrote:
| How can you interact with another browser plugin from a
| plugin, that doesn't seem to make sense
| mesadb wrote:
| I'm sorry I am a little confused. We use Chrome extension
| to record your Browser tests. (Not browser plugin tests)
| notadev wrote:
| Selenium IDE allows recording/playback of actions via browser
| extensions as well, without having to figure out the XPath, etc.,
| yourself. Preflight looks much nicer than Selenium IDE interface,
| but is it much more different in its function?
|
| Can you run Preflight on-prem or must the AUT be accessible to
| the Internet?
|
| Regarding the Context Awareness, is there any issue with working
| with React apps? If I renamed an item in the navbar, or changed
| the position of an element (thus changing the XPath), would CA be
| able to identify and workaround that?
|
| Questions aside, it's a great idea and the app looks great.
| mesadb wrote:
| Yes, it's almost similar to Selenium IDE. We capture more data
| like target screenshot and context of the element etc. We get
| rid of automatically generated IDs or classes.
|
| We can access to environments behind the VPNs. But it's hassle-
| free if it's accessible through the internet.
|
| Actually we also implemented Context awareness for SPAs with
| CSS Module to adapt to the changes. We are agnostic to HTML
| changes and if we can find the element we will update the
| selectors after the test is successful.
|
| Thank you very much for the kind words!
| sandman83 wrote:
| Any plans on integrating with Github / Github Actions at any
| point?
| mesadb wrote:
| Hi, we have a webhook that can be triggered from Github PR's
| etc. If that's something you need would love to help if it's
| not clear.
| econnors wrote:
| how's this compare to datadog synthetics?
| https://docs.datadoghq.com/synthetics/
| mesadb wrote:
| They work with selectors in general. We understand the page
| structure and even though there is a UI change we adapt to it.
| They have a little bit more functionality at the moment but we
| should catch them soon.
___________________________________________________________________
(page generated 2021-12-14 23:02 UTC)