[HN Gopher] Show HN: We built a tool that automatically generate...
___________________________________________________________________
Show HN: We built a tool that automatically generates API tests
Hi, this is Mish and Sebastian. We are working on Step CI - a fully
automated API testing platform for developers. Step CI works
programming-language independent and for different API paradigms
(REST, GraphQL, XML). Our CLI and test runner are available on
GitHub (https://github.com/stepci) under the MPLv2 license. Since
our last launch, Step CI is now able to generate automated tests
for your API based on your OpenAPI (Swagger) spec. This saves you a
lot of time as you never have to write and maintain your tests
again! We would like to invite you to try our tool and give us
feedback! Please star us on GitHub, if you like what we are working
on! We are very thankful for your attention and any feedback or
suggestions we receive from you :) Mish and Sebastian from Germany
Author : Wissmueller
Score : 88 points
Date : 2022-10-10 13:57 UTC (9 hours ago)
(HTM) web link (stepci.com)
(TXT) w3m dump (stepci.com)
| jzig wrote:
| Yo, I tried uploading my openapi.json file and all I got was a
| big error on the right hand side.
| jzig wrote:
| In the console:
|
| ``` Uncaught YAMLParseError: The : indicator must be at most
| 1024 chars after the start of an implicit block mapping key at
| line 5551, column 3: # $ref:
| '#/components/schemas/String' schemas: ^^^^^^^
|
| ```
|
| Edit: Apparently I have some validation errors when pasting
| into the Swagger Editor. That must be why; will have to fix
| those up and try again.
| jicea wrote:
| Really cool, with already a lot of features!
|
| Given the raise of CI/CD, I really think these kind of tools (CLI
| tests on HTTP requests), based on a simple format, will be really
| important. We've build Hurl (https://github.com/Orange-
| OpenSource/hurl), that shares a lot of similarities with Step CI
| (plain text instead of yaml, captures, jsonpath, xpath etc...). I
| will shamelessly take inspiration for some new features (like
| GraphQL for instance)!
| Sebastian_Wis wrote:
| Oh, nice to meet you!
|
| Our Matcher functionality was actually inspired by Hurl (you
| call them predicates!)! We would take your inspiration from us
| as a huge compliment :)
|
| The main difference between Hurl and Step CI is from our
| persepective an ideological one. What do you think?
|
| Hurl is a CLI tool, while Step CI is a library. This means that
| you can include our Test Runner
| (https://github.com/stepci/runner) and use it with other tools.
| For example, you can combine it with testing frameworks like
| Mocha, Jest, and Ava
| (https://github.com/stepci/stepci/blob/main/docs/using-
| test-t...) or build your own platform on top of it.
|
| That's what's so nice about open source.
| jicea wrote:
| Yes, I can see that! We focus on the CLI and I see how Step
| CI can shine. Happy to be a (small) source of inspiration.
| Good luck with Step CI!
| airocker wrote:
| Hi , I tried to build cli from swagger, it was a disaster: very
| verbose and did not really work. I am wondering what's the
| verbosity of your tool. How do you deal with imperfect specs?
| Wissmueller wrote:
| We're sorry to hear that! Would you mind sharing some context
| with us?
|
| Our OpenAPI (Swagger) integration is currently in Beta and only
| available through our website. We acknowledge, that it is not
| ready for prime-time _just yet_
|
| Everything we do is open-source. You can see the list of issues
| in the repository and contribute if possible:
| https://github.com/stepci/plugin-openapi/issues
|
| Disclaimer: We have tested our OpenAPI integration with the
| Petstore example and some other API definitions we had
| available. In both cases the generated tests would run after
| only some little adjustments
| airocker wrote:
| Swagger has the ability to turn api definition to libraries
| in any language. It is too verbose
| raydiatian wrote:
| openapi is insanely verbose. I personally treat openapi as
| an interchange format: I don't write it by hand. The best
| workflow I've found thus far is:
|
| FastAPI (Python) or NestJS to produce self documented APIs
| via route annotations, handled by framework, and then from
| that, export an openapi file that you use to create the
| client in your necessary language. In so doing, every
| service I make can be commanded with ease by whoever wants
| to use it, at minimal overhead for me as the service
| maintainer.
|
| It was a decent amount of effort to work out the bugs,
| enough so that I think better documentation around getting
| setup would be valuable, but of course the documentation is
| so cross-cutting that it's difficult. But the work was well
| worth it--I'm much happier with this workflow where I don't
| have to roll a service layer lib for my front end, nor use
| vanilla fetch libraries.
| yuvalsteuer wrote:
| BTW I appreciate good UI, your site is incredible, how did you
| build it?
| fernandorojo wrote:
| Cool idea. I'll definitely check it out. We've been thinking of
| making something similar internally.
| ushakov wrote:
| Hello Fernando, this is Mish from Step CI
|
| We're glad you liked the idea! I had this idea in my head for
| about 2 years until I met Sebastian and we decided to give it a
| try!
|
| We already support sending GraphQL queries and validating the
| responses
|
| Soon we will also be able to generate automated API tests based
| on your GraphQL Schema. You can watch our main repository
| (https://github.com/stepci/stepci) or subscribe to our
| newsletter to follow our progress!
|
| Thanks in advance for checking out our tool!
| manishrana wrote:
| This is really a useful tool
| reagan83 wrote:
| This is a great idea & brilliantly executed. I love to see the
| innovation in this space. Nice work Mish and Sebastian!
| inglor wrote:
| How do you deal with state between API calls? (For example, a
| user filling a shopping cart, getting discounts based on their
| geo/products and then checking out)
| ushakov wrote:
| Hello, this is Mish from Step CI
|
| Thanks for your question!
|
| We allow you to capture content like headers, response and
| cookies between steps using captures, check out the "Captures"
| example in our Demo
|
| Here's a documentation with examples for all capture types:
| https://github.com/stepci/stepci/blob/main/docs/workflow-syn...
|
| also: we manage cookies for you automatically
| fernandopj wrote:
| Does capture implement what I'm trying to discuss here?
| https://github.com/stepci/stepci/discussions/17
| mkl95 wrote:
| OpenAPI Generator + This tool + some good UI would be a killer
| combination
| BerislavLopac wrote:
| How does this compare to Schemathesis [0]?
|
| [0] https://schemathesis.readthedocs.io
| raydiatian wrote:
| Way to go guys.
| barefeg wrote:
| I was looking in the site to understand the automatic generation
| of tests from OpenAPI but I couldn't find it. Maybe it's very
| obvious but I missed it. Does it mean that it produces test cases
| from the examples provided? Or how does it figure out the request
| response pairs?
| Wissmueller wrote:
| Thank you for the question!:)
|
| You can generate tests from your OpenAPI definition using the
| "import from OpenAPI" button on our website.
|
| The tests are generated from your request/response examples. If
| there are no examples given/available, we will generate
| examples _for you_ based on the schema provided.
| barefeg wrote:
| Thanks for the quick answer! Are the examples generated based
| on the types? As in if a field is set as an integer, the
| tests will pass arbitrary integers? But still how does it
| know how to produce positive examples? Do you maybe have some
| documentation on this functionality?
| fernandopj wrote:
| Fantastic project. I worked on something similar that had the
| same approach, but the main goal was to test availability in
| different environments (dev, stage, production) against the same
| tests. We also had the feature of using the result of one test as
| input to another, so a complete user case from an API could be
| tested (auth, POST data, GET results)
| LeonidBugaev wrote:
| If you are looking for a more general tool, which can do not only
| API tests, but also interact with Databases and etc, but still
| use declrative syntax, try Venom https://github.com/ovh/venom
| Wissmueller wrote:
| Hi, this is Sebastian from Step CI
|
| Thanks for sharing Venom!
|
| Certainly, there are lots of tools in the space, both SaaS and
| open-source. Mish and I can name you a few!
|
| However, there's one thing these tools seem to be missing - a
| greater goal.
|
| Our vision is to give you a tool that will free you from
| writing and maintaining API tests ever again - in any shape or
| form. What you currently see from us is the first step!
| itake wrote:
| > Venom is a CLI (Command Line Interface) that aim to create,
| manage and run your integration tests with efficiency.
|
| The linked repo seems to have a greater goal.
| airocker wrote:
| Why use venom? Why not just pytest ?
| mrwnmonm wrote:
| love the design
| yuvalsteuer wrote:
| Love it!
|
| > "never have to write and maintain your tests again!"
|
| this feels a bit misleading though.
|
| What are you doing step ci? :D
| Sebastian_Wis wrote:
| We have been watching you and we know testing APIs is your
| hobby and you spend a lot of time on it. We'll have to take it
| away from you! ;)
|
| Just give us a little more time and hopefully your support. :)
| yuvalsteuer wrote:
| Haha! love the spirit! :)
___________________________________________________________________
(page generated 2022-10-10 23:01 UTC)