[HN Gopher] Launch HN: Codeparrot (YC W23) - Automated API testi...
___________________________________________________________________
Launch HN: Codeparrot (YC W23) - Automated API testing using
production traffic
Hi HN, we're Royal and Vedant, co-founders of CodeParrot
(https://www.codeparrot.ai/). CodeParrot automates API testing so
developers can speed up release cycles and increase test coverage.
It captures production traffic and database state to generate test
cases that update with every release. Here's a short video that
shows how it works:
https://www.loom.com/share/dd6c12e23ceb43f587814a2fbc165c1f . As
managers of engineering teams (I was CTO of an ed-tech startup,
Vedant was the founding engineer of a unicorn company) both of us
faced challenges in enforcing high test coverage. We ended up
relying a lot on manual testing but it became hard to scale, and
led to reduced velocity and higher production bugs. This motivated
us to build CodeParrot. How it works: we auto-instrument backend
services to capture production traffic. Requests and responses
coming to your backend service, as well as the downstream calls
made by it like DB calls are stored. As part of your CI pipeline,
we replay the captured requests whenever your service is updated.
The responses are compared with the responses from production env
and regressions are highlighted to the developers. To ensure that
the same codebase gives the same response in CI environment and
production, we mock all downstream calls with the values from
production. Most tools to record and replay production traffic for
the purpose of testing capture traffic on the network layer (as
sidecar or through load balancer), CodeParrot instead relies on an
instrumentation agent (built on top of OpenTelemetry) to capture
traffic, enabling us to capture downstream request/response like
database responses which are otherwise encrypted on network layer.
This helps us mock downstream calls and compare the response from
CI environment vs production environment. Additionally, this helps
us sample requests based on code flow and downstream responses
which provide better test coverage compared to just relying on API
headers & parameters. Our self-serve product will be out in a few
weeks. Meanwhile, we can help you integrate CodeParrot, please
reach out at royal@codeparrot.ai or you can choose a slot here -
https://tidycal.com/royal1/schedule-demo. We'll be selling
CodeParrot via a subscription model but the details are TBD. In
addition, we will be open sourcing the project soon. If you've
already tried or are thinking of using tools in this space, we'd
love to hear your experience and what you care about most. We look
forward to everyone's comments!
Author : royal0203
Score : 116 points
Date : 2023-03-17 18:35 UTC (1 days ago)
| poobags wrote:
| Are the requests being only replayed, or is there some amount of
| mutation going on too, to potentially reach buggy states?
|
| If the latter, I'm wondering how it might compare to tooling with
| a similar intent, e.g. https://www.microsoft.com/en-
| us/research/publication/restler...?
| shireboy wrote:
| I just this week did something similar for a line of business app
| API I'm migrating. I added code to the API to record all unique
| requests to a .http/.rest format file (natively supported in
| vlatest Visual Studio 22 and via extension in VS Code). I can
| then play those back manually or via automated integration tests
| that read the .http files. Yes, I'm hand-waving over
| authentication and downstream database and 3rd party APIs, but
| overall it's working well to quickly test against tons of
| production-like API calls.
| vedant_ag wrote:
| Wow very interesting!
|
| Did you manage to play it back manually and/or via tests? I ran
| into unexpected challenges while doing this.
| pradeep_bishnoi wrote:
| Congratulations on the launch folks. Will be interesting to see
| the open source product and offering. Self hosted version of this
| with ease to change the environment will some great addition.
|
| BTW, good folks at https://www.hypertest.co are doing something
| on same lines here in India.
|
| Cheers!
| royal0203 wrote:
| Thank you! Will keep you posted on open source version.
|
| We have come across Hypertest, seems pretty cool and useful.
| eganist wrote:
| Neat concept. In regulated environments, how would you propose
| implementing this to minimize the spread of live, regulated data
| into non-prod environments? think full PANs (PCI) etc.
|
| I don't know that there's necessarily a wrong answer here (well,
| there probably are, but wrong only in the sense that a given
| solution might be prohibited by the regulation), just want to see
| how y'all have thought through the prompt.
| [deleted]
| lttlrck wrote:
| This seems like the burning question, maybe that's the .ai...
|
| HAR can already be recorded in middleware (e.g. loadmill/har-
| recorder) and replayed in multiple CI compatible ways.
| muhash wrote:
| How would that potentially require AI?
| eganist wrote:
| Unrelated to this but responding to your other question
| that was deleted but still valuable:
|
| > What do you mean by "live, regulated data into non-prod
| environments" exactly? Could you provide some examples?
|
| Credit card numbers, card verification codes, protected
| health data/electronic medical records... list goes on.
|
| Every environment that has live data in it functionally
| increases the valuable attack surface for most adversaries.
| I.e why bother attacking production when they can slurp up
| production data from test environments that are less likely
| to be well protected?
|
| As for the "AI," I think the op was just commenting on the
| TLD used by the startup.
| vaishnavsm wrote:
| Oh wow, we're building something very similar to this. We were
| literally just filling up the YC form for S23 too haha. I may be
| a bit biased, but I think this is an excellent idea, and it can
| change the way testing works for backend services. We were
| inspired by Meticulous, too! Hope we can learn from y'all as well
| :)
| royal0203 wrote:
| Nice! Happy to share our experience if it helps :)
| inerte wrote:
| Does it work the other way too? It would be nice to also get the
| response from downstream services, and test if the app is still
| massaging the data correctly and generating the correct output.
| keeptrying wrote:
| Nothing gives you confidence in your system than testing on high
| variance sample of last 4 months of production traffic.
|
| Especially for rewrites or code refactorings.
|
| I've built this system at many companies myself. Never thought of
| doing this as a service for others.
| vedant_ag wrote:
| So true!
|
| > I've built this system at many companies myself.
|
| Interesting! Would I know any of those companies/products?
| zlatan99 wrote:
| Amazon has similar systems operating since last 10 years.
| roflyear wrote:
| Do you issue new sessions and modify the requests?
| LeonidBugaev wrote:
| I love to see more activity in this area!
|
| I'm maintainer of GoReplay https://github.com/buger/goreplay and
| work in this area for the last 10 years.
|
| It is quite hard problem to solve, because you have to deal with
| state difference between test and production environments. Love
| your approach to mocking dependencies, and leveraging
| OpenTelementry. It potentially can solve some of state issues.
| But still require modifying user code. I wonder if it can be done
| purely using OpenTelementry (e.g. you depend on typical OTel
| setup), and then read the data directly from OTel DB.
|
| Cheers!
| pradeep_bishnoi wrote:
| thank you Leonid for the GoReplay. A great ecosystem of
| products will be built on top of it.
| LeonidBugaev wrote:
| I hope so! But I also hope that I will be also able to
| monetise some of this movent. GoReplay dual licensed under
| AGPL and Commercial license. I also sell special appliance
| licenses.
|
| If anyone in this thread wants to build a product based on
| GoReplay technology (capture network traffic directly, via
| AWS Traffic Mirroring or k8s), sent me message :)
| royal0203 wrote:
| Go replay has been one of the inspiration Leonid, so glad you
| checked out CodeParrot :)
|
| Typical Otel implementation don't capture some request data esp
| parameters and replay part is missing among few other issues,
| so we need to extend it.
| nfw2 wrote:
| I worked on something like this briefly years ago and caching and
| mocking the downstream calls was a beast of a technical problem.
|
| For example, suppose a startup has an Express app that made
| downstream requests to Postgres, Redis, Kafka, other REST APIs,
| etc. How do the outgoing requests, which may follow different
| protocols, with varying serialization formats and handshakes, all
| get intercepted, recorded, and matched to the same outgoing calls
| when the session is replayed? How are the outgoing requests
| indexed and then evaluated for sameness when replayed?
|
| It definitely seems possible to implement something like this as
| a series of high-level one-off integrations, for example, a
| middleware package for Django apps. But if the goal is a
| universal and automatic downstream service mocking system, it's
| not clear to me where this sort of middleware would even sit in
| the application stack. It seems like it would need to be pretty
| low-level, but not too low-level because the data needs to be
| decrypted first.
|
| Anyway, if you guys have figured out a good way to do this, I'm
| definitely interested in hearing how you managed it. I'm in no
| way a great BE engineer, so there could be something elegant or
| obvious that I missed.
| Aeolun wrote:
| I think the fact they use opentelemetry figures into this.
| They've already done the hard work of intercepting calls to
| many libraries.
| royal0203 wrote:
| Yes, we rely a lot on openTelemetry for this. They have
| really good support for most libraries in Java, node and are
| progressing quickly in others. We are also contributing to it
| by extending support for other languages, which we'll be open
| sourcing soon.
| nfw2 wrote:
| Does opentelemetry also support mocking outgoing requests
| values on replay? Or is something else used for that part?
| royal0203 wrote:
| No, it doesn't have by default but can be extended to
| support it.
| Aeolun wrote:
| If you are contributing, isn't it open-source by default?
| Or you mean you have 'proprietary-ish' (since it's all
| client side, kind of impossible) packages that aren't part
| of opentelemetry yet?
| royal0203 wrote:
| Yes, I meant packages not part of opentelemetry - example
| python has lot of DB packages which don't have support
| yet.
| pcthrowaway wrote:
| > For example, suppose a startup has an Express app that made
| downstream requests to Postgres, Redis, Kafka, other REST APIs,
| etc
|
| And besides network calls network-bound requests, what about
| reading from sqlite, files, and interprocess communication.
|
| What about ZeroMQ over UDP? A remote time service. Pulling from
| a git repo.
|
| No really, I'm asking because I'm trying to figure out how to
| deal with all this in my tests right now, and I honestly have
| no idea. But I doubt Codeparrot is going to be able to
| intercept and intelligently mock everything.
| rkhacker wrote:
| Few missing details that are crucial to usage within an
| organization:
|
| 1. what is the type of service instrumentation needed to capture
| the data? Wonder why this is needed when typically the data is
| already captured in an APM log? The instrumentation might add
| performance and security concerns.
|
| 2. what is the sampling logic to capture the traffic? It might
| compromise the fidelity of the test data and give a false sense
| of test accuracy.
|
| 3. what is the duration of data capture? Is it a week's or
| month's or quarterly data? Meeting 90% coverage on a week's
| production sample data will provide a false metric.
|
| 4. can it faithfully handle data privacy and customer
| anonymization? This is critical for API's dealing with PCI and
| other sensitive data.
| MrSqueezles wrote:
| Yeah, 4 is key. Many privacy regulations stipulate that account
| data must be deleted within a certain period of time, usually
| days or less, after a requested account deletion. In this
| system, all recorded requests would have to be discoverable by
| the requestor's ID and production systems would have to
| remember to perform deletions when necessary. Also, this
| database and all related testing systems would have to be held
| to production level standards for data access because anyone
| who can see test data to root cause errors can see people's and
| business' real, private information. Especially for data
| controlled and regulated industries like government and health
| care, this would be a nightmare.
|
| It's a neat idea. These kinds of systems often require lots of
| care and grooming. Since it's used to retroactively test
| features after they're in production, there's a repeating
| process of discovering we're saving data we shouldn't,
| scrubbing, filtering, anonymization, etc. In most cases, I've
| watched them eventually get replaced by fuzzers. Still, having
| a central service used by lots of companies may allow this
| solution to scale up, develop necessary features to solve these
| problems and function well. I hope it works out!
| vedant_ag wrote:
| > 1. what is the type of service instrumentation needed to
| capture the data? Wonder why this is needed when typically the
| data is already captured in an APM log? The instrumentation
| might add performance and security concerns.
|
| Implementation is very similar to an APM log. So the same
| performance and security concerns apply. We are working on
| giving both at the same time (Automated tests, and APM), to
| reduce overhead.
|
| > 2. what is the sampling logic to capture the traffic? It
| might compromise the fidelity of the test data and give a false
| sense of test accuracy.
|
| It is random sampling. I feel, 1M or 10M randomly sampled
| requests should cover all cases.
|
| > 3. what is the duration of data capture? Is it a week's or
| month's or quarterly data? Meeting 90% coverage on a week's
| production sample data will provide a false metric.
|
| I was thinking 1 week should be enough. Maybe we will have to
| add some custom sampling logic for lesser frequency calls (like
| monthly crons).
|
| > 4. can it faithfully handle data privacy and customer
| anonymization? This is critical for API's dealing with PCI and
| other sensitive data.
|
| Yes. Additionally, for compliance, we offer a self-hosted
| solution- Our code runs on your servers and no data ever leaves
| your cloud / on-prem.
| chrsig wrote:
| I've been having to put together a test suite at $work
| similar to what the product tries to offer. If there were a
| sufficiently advanced product to buy, I'd petition for it to
| be adopted at my org.
|
| > It is random sampling. I feel, 1M or 10M randomly sampled
| requests should cover all cases.
|
| 1. I suggest providing alternate approaches to sampling: The
| input itself may have bias towards a single use case. If 70%
| of the input exercises the same code path, there's no benefit
| to having a uniform sample. Ideally it would be stratified
| amongst customers, or perhaps on other dimensions to allow
| for covering the most surface area.
|
| 2. Requests don't happen in a vacuum. They likely have data
| dependencies on prior requests. I recommend some way of
| sampling sessions rather than individual requests. Replaying
| the 3rd request in a series of 6 is likely just going to be
| exercising failure paths.
|
| 3. Behaviors may vary between requests with respect to time.
| If requests were sampled over a number of days but replayed
| within a short time period, there are behaviors that could
| differ from what actually occurs in production.
|
| I didn't see any explanation on how results are determined. I
| think it's important to surface those types of details on the
| website. I'm not going to watch the video on it in hopes of
| learning.
| jonahx wrote:
| "All the downstream services are mocked, you don't need to set up
| a test environment."
|
| Could you elaborate on how this is achieved? Eg, say I have a
| lambda endpoint and the code in it is querying one or more
| databases. Are you somehow automatically hooking into those
| function calls, recording their return values, and then mocking
| the functions in the replays? Or are you doing something else
| entirely?
| debarshri wrote:
| If you are looking at this. You might be also be interested in
| Speedscale [1]. They have been around for a while. Interesting
| thing is that it is a YC company too.
|
| [1] https://speedscale.com/
| royal0203 wrote:
| Yes, and Ken from speedscale is a very helpful person too.
| pixelatedindex wrote:
| Neat idea! Two nitpicks from reading the website:
|
| > How does CodeParrot work
|
| This part without a question mark is making me feel like I'm left
| hanging. If you don't want a question mark, I would much prefer
| this to be "How CodeParrot works".
|
| > CodeParrot ... What's more? ...
|
| There's no question mark where there should be one, and where I
| don't expect one I do see one. Here it can be a simple comma
| instead of a question mark. This flow of language works if you're
| speaking live to an audience but on paper it feels awkward.
|
| It bothers me more than it should, mostly because of my reading
| cadence. Hopefully others can chime in and let me know if I'm off
| base here or not.
| royal0203 wrote:
| Nice catch, thanks for taking the time to review the website!
| Have updated it, should reflect in sometime.
| iLoveOncall wrote:
| I swear I see exactly the same idea at least once a month on HN.
| YPCrumble wrote:
| How is this different from Meticulous, another YC company?
| royal0203 wrote:
| it's similar in the sense that both rely on production traffic
| and user sessions to generate tests. However, we are focusing
| on API testing and I think Meticulous is building for UI
| testing.
| quickthrower2 wrote:
| Nice! How do you deal with:
|
| 1. Statefulness: simple example, api call that returns a query
| from the DB that has a high transaction load
|
| 2. Non determinism: for example returns a random number, guid or
| a time
|
| 3. Privacy requirements for certifications or legislation that
| mean private data cannot be used in a test environment.
|
| 4. GDPR laws
|
| 5. Authentication
|
| I presume this is more microservice friendly and monolith
| unfriendly. But that is probably a reason on the pro side of
| smaller, bounded services.
| kyriakosel wrote:
| Nice one guys! Congrats on the launch!
| bluelightning2k wrote:
| I built something like this years ago. It's just on the edge of
| roll-your-own-solution.
|
| I definitely agree with your approach of auto mocking the
| database and/or third party services too. That's what I did with
| my home rolled solution.
|
| Happy to see more projects in the space. Generating tests by
| snapshot including the DB and service calls always seemed like
| the obvious way to go for me.
|
| As a dev I'm not keen on using services which I could easily
| replicate so this would have to be both substantial and
| cheap/free for smaller teams.
| royal0203 wrote:
| I can relate to this perspective, however, some complexities we
| have come across in building this so far:
|
| - Support for high number of languages, downstream dependencies
| - Intelligent sampling to choose requests with high coverage
| and auto update them over time - Performance, safety and data
| compliance guarantees
| rishsriv wrote:
| Looks promising! Would love to know how this is different from
| creating postman collections, and running tests on those
| vedant_ag wrote:
| - Postman collections have to be created manually by a
| developer (usually).
|
| - Downstream calls (think DB, 3rd party API calls, kafka) are
| not handled.
| [deleted]
| Jarobq18 wrote:
| Isn't your tech fragile? Feels like it'd break/provide false
| positives easily. Integration is also not easy, the matrix of
| possible tech stacks is big. It doesn't test new edge cases, so
| it's useful only for regression, which is important but if your
| testing hygiene is good, which means devs are writing tests, you
| wouldn't have the need for your product, am i wrong? Good luck
| either way. Seems like VCs are gambling at products that save
| costs.
| royal0203 wrote:
| Good observation, it's challenging to solve these problems,
| here's how we are going about it -
|
| To reduce false positives - we run the same request twice to
| eliminate flaky fields in response like current timestamp, mock
| the downstream dependencies as they behaved in prod env and are
| providing options to ignore / modify the sampled requests
|
| To make integration easier - we are building on top of
| opentelemetry which has seen remarkable increase in support
| across languages / frameworks, which makes it easier for us to
| support different tech stacks.
|
| Regression testing - our primary goal is to provide regression
| tests. We have come across two type of teams where this makes
| sense - companies with low test coverage and companies which
| high number of micro-services as they find it hard to cover
| every production scenario in tests
| a_c wrote:
| Congratulations on the launch!
|
| Quick question, how do you (plan to) deal with schema/API change?
| Or the tool is more intended for regression testing?
| vedant_ag wrote:
| New API tests can be generated by enabling the agent locally
| and/or on a staging env.
|
| Its simply a matter of using a baseline of: `production`
| (usually for regression tests), `staging` or `local`.
___________________________________________________________________
(page generated 2023-03-18 23:02 UTC)