[HN Gopher] Show HN: Goqite, a persistent message queue Go libra...
___________________________________________________________________
Show HN: Goqite, a persistent message queue Go library built on
SQLite
I wanted to build a persistent message queue based on SQLite,
because that's what I'm using for my main state anyway. This gives
me ACID across state and messaging, which is nice! I've been
inspired by the terminology of AWS SQS for this, but it's obviously
much simpler. Maybe you can use it too. :)
Author : markusw
Score : 73 points
Date : 2024-03-11 10:39 UTC (12 hours ago)
(HTM) web link (www.goqite.com)
(TXT) w3m dump (www.goqite.com)
| rco8786 wrote:
| Looks neat, but how on earth is "Goqite" meant to be pronounced?
| markusw wrote:
| Thanks! See the first sentence of the readme. ;)
|
| "goqite (pronounced Go-queue-ite) is..."
| ncruces wrote:
| You should upcase the Q then. ;)
| markusw wrote:
| Hehe. Perhaps.
| jareklupinski wrote:
| maybe GoQuette? (like eti-quette or cro-quette, go-quette)
| markusw wrote:
| Nah, I've already got the domain. :D
| foxbee wrote:
| Annoying but I felt exactly the same. I'd advise you to rename
| the project - it's still early days and the longer it goes on
| the harder it will be to change - I've felt this pain.
| markusw wrote:
| I actually quite like it. :D
| Kerbonut wrote:
| I have strong feelings in the opposite direction.
| markusw wrote:
| Which is very much your right. Agree to disagree? :D
| remram wrote:
| Ignore feedback at your own risk.
| markusw wrote:
| Feedback and opinion are not the same thing.
|
| Also, yes, I will take on that risk. As anyone who has
| ever been at the receiving side of an issue tracker
| knows, not all feedback is equal. I appreciate taking the
| time to provide feedback, though, but I always reserve
| the right to not act on it.
| remram wrote:
| Is it feedback or opinion then? Your two sentences do not
| agree.
|
| If 7 people tell you that "go kite" is not a great name
| for an important database system, I don't know what to
| tell you other than "consider the feedback".
| lifeinthevoid wrote:
| In the example the user has to create the schema and manually
| open the SQLite connection, can't you abstract that away?
| abound wrote:
| Not the author, but I imagine most projects already have a way
| of managing their SQLite schema (e.g. a migration system like
| go-migrate or goose) that they'd integrate the new table(s)
| into.
| markusw wrote:
| This exactly. If you're using SQLite in your Go project, you
| most probably already have your sql.DB and schema migrations
| sorted, so I don't want to get in the way of that. :)
|
| (But I can see how that could be clearer from the example
| code. I think I'll add that as comments. Thank you for the
| feedback!)
| markusw wrote:
| I've updated the code example to reflect this.
| whalesalad wrote:
| Wouldn't you want to use a different db file for this? I
| wouldn't want to put this in the same db as the rest of my
| app, especially considering how simple it is to just open a
| new file.
| markusw wrote:
| If you want to be able to send to the queue in the same
| transaction as other state changes, it has to be the same
| file. That's often a useful property.
| lifeinthevoid wrote:
| SQLite allows transactions involving multiple databases
| (see https://www.sqlite.org/atomiccommit.html section 5)
| markusw wrote:
| Oh! TIL! Thank you for that. :)
| yard2010 wrote:
| SQLite is such a great thing. Amazing value for zero cost
| both shmeckels and maintenance. It's a shame most
| startups go for postgres blindly due to "scalability
| concerns" it feels like a major case of premature
| optimization.
| markusw wrote:
| It really is! I've become a SQLite fanboy. :D Although I
| can definitely see the value in Postgres as well. But
| it's usually one of those two.
| pdimitar wrote:
| My beef with SQLite is the weaker typing + not even
| enforcing foreign keys unless you open the DB in a
| certain mode.
|
| IMO it should become more opinionated and stricter. I'd
| honestly pay to buy a license for SQLite with the legacy
| baggage removed and rewritten to be as strict as
| PostgreSQL or more. Well... basically embedded PostgreSQL
| I guess.
| markusw wrote:
| I also don't like those parts, but for an otherwise great
| project, I'd happily use the parts I do like. Strict
| tables go a long way for me, and you can make foreign key
| checks a compile-time enforcement.
| packetlost wrote:
| I made something extremely similar, but lighter weight awhile
| ago: https://github.com/chiefnoah/goalpost
|
| It uses bbolt/bolt instead of sqlite. It was for use on shared
| servers that had questionable uptime.
|
| I haven't touched it in a long time, but it served its purpose
| well
| markusw wrote:
| Yeah, there are a lot of queue libraries like ours. :) It was
| important to me it was SQLite, since that's what I use as the
| storage layer already.
| packetlost wrote:
| Yeah, not trying to detract from your work! At the time when
| I made mine (like, 6 years ago) there was nothing in the
| space. Now there's plenty! Nice work!
| markusw wrote:
| Didn't take it that way, but thank you for the
| clarification. :) And thanks! Likewise. :D
| slicedbrandy wrote:
| Also did something very similar, but swapped out the storage
| layer with an embed of leveldb.
|
| Also supports both an HTTP and Redis API.
|
| https://github.com/tomarrell/miniqueue
| markusw wrote:
| Interesting that it supports the Redis API as well!
|
| As noted elsewhere here, it was important to me to use SQLite,
| since that's what I use as the storage layer already. But I've
| added HTTP adapters for remote clients.
|
| I hadn't thought of using a different remote API, interesting
| idea. Thank you for the pointer!
| pstuart wrote:
| I love Go _and_ SQLite and enjoy using them together. One bummer
| is that it requires CGO to use and that makes cross compiling a
| pain. It might be nice to use the modernc variant (via build
| tags) to make this more portable --
| https://gitlab.com/cznic/sqlite
| markusw wrote:
| It's bring-your-own-driver, so you can use a CGo-free one. :)
| The one in go.mod is only used in tests.
| willi59549879 wrote:
| Zig allows cross compilation of cgo
| pdimitar wrote:
| There is a pure Golang SQLite implementation btw, I love that
| aspect of the Golang ecosystem. Not even Rust has that and I am
| envious.
| heipei wrote:
| Cool idea, I can definitely see the appeal of having everything
| in one database. I would probably also add an "attempts" column
| (int) so you can discard messages after X attempts.
| markusw wrote:
| That's the "received" column. :) Messages after X attempts
| aren't received by the normal receiver method anymore.
| emadda wrote:
| How is this better than just using SQLite transactions directly
| and a task queue table?
| markusw wrote:
| It has a Go API. ;)
|
| Check the source, that's essentially what it is. But you get
| this particular design, the tests etc. for free, so if you
| value your time, maybe this is worth something to you. Maybe
| not though. Either way, it's fine.
| gigatexal wrote:
| As much as I love SQL I would have thought this would have
| abstracted away the SQL bit of the queue and just given me high
| level queue methods to do stuff with. I don't like this much for
| that reason.
| markusw wrote:
| I don't understand your comment. Abstracting away the SQL is
| exactly what this library does (and pretty much the only
| thing): Send, Receive, Delete methods.
|
| Could you clarify?
| echoangle wrote:
| Well your example on the website contains a Database schema
| and manually opens a DB connection. Why do I as a library
| user need to care about that? Just do that internally when
| creating the queue instance
| markusw wrote:
| See this other thread:
| https://news.ycombinator.com/item?id=39670199
| janderland wrote:
| From the example, it appears you still need to setup schema
| yourself using SQL. This makes the library a leaky
| abstraction because it only partially hides the SQL.
| gigatexal wrote:
| Exactly this.
| markusw wrote:
| There you go! :D
| https://github.com/maragudk/goqite/pull/31
| asa400 wrote:
| This is super cool! I like your method for handling the claiming
| of queue items. I always end up wishing SQLite had something like
| Postgres' LISTEN/NOTIFY/SKIP LOCKED but your UPDATE solution is
| nice and low tech and fitting for SQLite, in a good way!
| markusw wrote:
| Thank you, I'm really glad you like it! :)
| polyrand wrote:
| I also built something similar for Python [0]. I decided to go
| with a separate DB file for the queue, I think it's better for
| performance but also to avoid mixing the data storage with the
| message queue (in my case, I assume the message queue is just an
| implementation detail and not really tied to the rest of the DB
| schema).
|
| [0]: https://github.com/litements/litequeue
| markusw wrote:
| Yeah, that's a good idea! I was today old when I learned that
| SQLite can have transactions across separate db files.
| markusw wrote:
| Well peepz, it's been fun, but I'm off to bed now because
| sometimes I get sleepy. Ta! :D
|
| (Thank you for all the comments!)
| dev-tacular wrote:
| > says they are sleepy
|
| > 30 minutes later they open a new PR to address a comment left
| by [an HN user](https://news.ycombinator.com/item?id=39673795)
|
| I think this person might like coding
| Ingon wrote:
| In a me too way, I also created something similar -
| https://github.com/klev-dev/klevdb
|
| It started on top of bbolt, but over time moved a completely
| custom implementation. The main reason for this was performance,
| but I also wanted to run a lot of queues, so each one had to be
| as light as possible. On top it even provides some indexing
| capabilities, which (when enabled) allow for quick find by key
| and time.
| mattrighetti wrote:
| Interesting, this seems to be something that a lot of people
| tried to implement, it's the first time I see this many "I did
| something similar" under a project!
|
| Cool project! Would love to implement something similar in
| another language and maybe compare performances, even though I
| guess they would be very similar since there's gonna be SQLite
| under the hood.
___________________________________________________________________
(page generated 2024-03-11 23:01 UTC)