[HN Gopher] Show HN: I open-sourced the in-memory PostgreSQL I b...
___________________________________________________________________
Show HN: I open-sourced the in-memory PostgreSQL I built at work
for E2E tests
Author : n2d4
Score : 138 points
Date : 2024-04-07 13:13 UTC (9 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| n2d4 wrote:
| Hey HN! For a few months, I've been building an in-memory version
| of Postgres at work. It has full feature parity with production
| databases.
|
| The cool thing about it is that you don't need any external
| processes or proxies. If your platform can run WASM (Node.js,
| browser, etc.), it can probably run pgmock. Creating a new
| database with mock data is as simple as creating a JavaScript
| object.
|
| It's a bit different from the amazing pglite [1] (which inspired
| me to open-source pgmock in the first place). pgmock runs an x86
| emulator with the original Postgres inside, while pglite compiles
| a Postgres fork to native WASM directly and is hence much faster
| and more lightweight. However, it only supports single-user mode
| and a select few extensions, so you can't connect to it with
| normal Postgres clients (which is quite crucial for E2E testing).
|
| Theoretically, it could be modified to run any Docker image on
| WebAssembly platforms. Anything specific you'd like to see?
|
| Happy hacking!
|
| [1] https://github.com/electric-sql/pglite
| waldrews wrote:
| Ooh! The 'docker image on WASM' thing sounds promising for a
| wide range of problems. Recently I wanted to run a FFMPEG/SoX
| pipeline on the client - too many dependencies to easily
| recompile with Emscripten; could your approach help there?
| n2d4 wrote:
| Yeah, that should be possible! Though, for audio processing,
| the performance will probably be terrible (because it's all
| emulated).
| jasonjmcghee wrote:
| There's already ffmpeg wasm. I've used it in projects. Works
| great.
|
| https://github.com/ffmpegwasm/ffmpeg.wasm
| samwillis wrote:
| This looks really cool, awesome work!
|
| Correct on PGlite only being single user at the moment, and
| that certainly is a problem for using it for integration tests
| in some environments. But I'm hopeful we can bring a multi-
| connection mode to it, I have a few ideas how, but it will be a
| while before we do.
|
| There are a few other limitations with PGlite at the moment
| (related to it being single user mode), such as lacking support
| for pg_notify (have plans to fix this too). Whereas with this
| it should "just work" as it's much closer to a real Postgres.
|
| I think there is a big future for these in-memory Postgres
| projects for testing, it's looks like test run times can be
| brought down to less than a 1/4 with them.
|
| (I work on PGlite)
| justinclift wrote:
| As a data point, the online demo seems broken for queries it
| doesn't like: select foo();
| Error.captureStackTrace is not a function
|
| That's when using Firefox 124.0.2 on Linux.
| rezonant wrote:
| Yes, that's a nonstandard function provided by v8, so it
| wouldn't work on Firefox. [1]
|
| This can be worked around by just constructing an Error and
| taking it's stack property, captureStackTrace is just a
| convenience function, so hopefully they can fix that.
|
| [1] https://developer.mozilla.org/en-
| US/docs/Web/JavaScript/Refe...
| negus wrote:
| That's great. But doesn't the whole concept of E2E test mean
| that you use real environment without mocking the components?
| refulgentis wrote:
| Explicitly mentioned in the comment as a drawback. In
| practice E2E means "E2E as much as humanly possible", and I'm
| glad to see any work that can help.
| rsyring wrote:
| Why not just run Postgres with it's files on a ramdisk?
|
| Update: this can apparently run in a browser/Node environment so
| can be created/updated/destroyed by the tests. I guess I'm too
| much of a backend dev to understand the advantage over a more
| typical dev setup. Can someone elaborate on where/when/how this
| is better?
| orphea wrote:
| I don't get it either. I feel like this is so much unnecessary
| code, an emulator, a network stack...
|
| Why not use something like https://testcontainers.com/? Is a
| container engine as an external dependency _that_ bad?
| bastawhiz wrote:
| It's the same amount of code and on Mac you still run a full
| VM to load containers (with a network stack), so I'm not
| really sure what your point is. If anything it's _less_ code
| because the notion of the container is entirely abstracted
| away, and the whole thing is entirely a wasm dependency that
| you load as a normal import.
| medellin wrote:
| It is annoying is you want to run your teat inside a
| container for ci and now you are running a container in a
| container and all the issues that come with it.
| dorianmariefr wrote:
| which issues?
| c0balt wrote:
| Depending on the setup it can be a pain to get nested
| containers working sometimes. There is, e.g., Docker In
| Docker but this often required a privileged host
| container which is often not provided in CI/CD pipelines.
| j45 wrote:
| Which issues/pains with getting nested containers?
|
| I am aware of only a few settings that make a container
| nestle, or not, whether it is a vm, lxc/lxd type
| container, etc.
| n2d4 wrote:
| That's more or less what happens inside the emulator (the
| emulated disk is an in-memory 9P file system). It's in
| WebAssembly because that makes it more portable (same behaviour
| across platforms, architectures, and even in the browser or
| edge environments), and there are no external dependencies (not
| even Docker).
|
| Because the emulator lets us boot an "already launched" state
| directly, it's also faster to boot up the emulated database
| than spinning up a real one (or Docker container), but this was
| more of a happy accident than a design goal.
| gchamonlive wrote:
| You could also use memory state dump from a microvm manager
| like firecracker and have the state replicated
| zer00eyz wrote:
| The whole purpose of End to End testing is that your testing
| the system in a real state. It's an emulation of your live
| environment. Because of that you can do interesting things like
| find out what happens if you pull the plug or run out of disk
| or ....
|
| The moment that you shove a mock in there, your unit testing.
| Effective but not the same. One of the critical points of E2E
| is that without mocks you know that your tests are accurate.
| Because this isnt Postgres I'm testing it every time and not
| that system.
|
| >> Can someone elaborate on where/when/how this is better?
|
| If your building PG for an embedded, light weight, or under
| powered system then this would make sense for verification
| testing before real E2E testing that would be much slower. (a
| use case I have)
|
| Other than that its just a cool project and if you ever need a
| PG shim it's there.
| andrelaszlo wrote:
| Cool! Which pg version is this based on?
| justinclift wrote:
| The online demo seems to be PG 14.5. Output from "SELECT
| version()" is: "version": "PostgreSQL 14.5 on
| i686-buildroot-linux-musl, compiled by i686-buildroot-linux-
| musl-gcc.br_real (Buildroot 2022.08) 12.1.0, 32-bit"
| rickette wrote:
| I used to run all kinds of (custom) fake in-memory servers in my
| tests. Nowadays I just run the real thing using Testcontainers
| (https://testcontainers.com)
| drzaiusx11 wrote:
| How does this compare to H2 in postgres compatibility mode?
| herpdyderp wrote:
| This might be a stupid question, but do you know how this might
| be used with a Prisma client?
| tumidpandora wrote:
| Off-topic, but the title confused me a bit - "...I built at
| work." Doesn't this imply that the intellectual property for this
| project belongs to your employer, assuming you used resources
| from work? If so, are you technically allowed to open-source it?
| Gracana wrote:
| Stackframe owns the repo and the LICENSE file says "Copyright
| 2024 Stackframe." I think the author works at Stackframe.
___________________________________________________________________
(page generated 2024-04-07 23:00 UTC)