[HN Gopher] Fundamentals of Object Storage in Elixir
___________________________________________________________________
Fundamentals of Object Storage in Elixir
Author : lawik
Score : 99 points
Date : 2024-03-23 06:26 UTC (16 hours ago)
(HTM) web link (underjord.io)
(TXT) w3m dump (underjord.io)
| thibaut_barrere wrote:
| Good write-up!
|
| I will add:
|
| - make sure to "stream" your transfers in general, instead of
| keeping whole file blobs in RAM. This to avoid refactoring at the
| least comfortable moment when a file gets too big in production
|
| - it is quite easy to use Mox for isolated testing in general,
| and it works nicely with ExAws, so make sure to integrate those
| tests early on
| sph wrote:
| Seconding the Mox recommendation, it's pretty good and enforces
| good separation of concerns in your application. It was created
| by Jose himself, so you know it's got a well-designed API. You
| just need to get used with using behaviours, and using the
| Application config to tell your code when to use the mock vs
| the real thing.
|
| I use it to separate and test external API calls (i.e. to test
| the payment layer without making any request), to maintain
| separation between my code and an external library I might want
| to swap out at a later date, or even between various components
| of the application. Then writing unit tests becomes a breeze,
| and you don't need many integration tests.
|
| https://hexdocs.pm/mox/Mox.html
| sarchertech wrote:
| I hate Mox with a passion. Because I hate separating out the
| interface and implementation of something that is only ever
| going to have one real implementation.
|
| The extra layer of indirection just for testing bugs me so
| much. And I just can't justify it with "one day I might want
| to swap out the implementation". YAGNI and if you do, deal
| with it then.
|
| So much legacy elixir code is an unreadable mess because of
| all the extra layers of interfaces that only get used for
| Mox. There's nothing I dislike more than diving into an
| unfamiliar section of code and having to keep 15 files open
| to follow the logic for something that should have been in
| one module.
| sitkack wrote:
| Kinda horribly half serious, but can you use Elixir's
| Metaprogramming features to demoxify mox?
|
| https://www.youtube.com/watch?v=2Bjzml_Hpvk
| sph wrote:
| That just means you're putting a boundary when it's not
| necessary. Mox exists to make tests simpler to write and
| reason about, if you keep hitting extra layers of
| indirections then, "you're holding it wrong."
|
| Most things are best used sparingly. That said, the vast
| majority of developers, even senior, have a very limited
| understanding of testing practices, and tend to create
| millions of separated, isolated modules, for no reason
| whatsoever. That's not Mox's fault.
|
| A stupid example in practice: a naive blog app does not
| need Mox at all. If you add SSO sign up, you might want to
| put a boundary there to unit test without needing to set up
| a fake auth server, but 99% of your code won't ever touch
| it, just the authentication tests. Tomorrow you add
| payments, you might want to put a boundary just before your
| _StripeRestAPI_ module to test the entire payment flow
| without contacting Stripe in your unit tests. Again, only
| the payment module tests ever need to touch Mox, the rest
| of your code doesn 't. You place it before modules in the
| outer edge that have real-world side effects, not to
| isolate your functional and mostly pure core.
|
| I recommend the DestroyAllSoftware talks, especially
| https://www.destroyallsoftware.com/talks/boundaries
|
| > And I just can't justify it with "one day I might want to
| swap out the implementation". YAGNI and if you do, deal
| with it then.
|
| I agree with that. Using Mox to swap out implementations is
| a possible use case, but in practice it's better to
| hardcode things rather than trying to make it generic and
| swappable the first time around. Junior devs have a
| tendency to over-engineer code "because we might need to
| change it later." My motto is _" copy and paste it three
| times before adding a layer of abstraction"_
| sarchertech wrote:
| I hope you see the irony in using "you're holding it
| wrong" here?
|
| If real world use of a library tends to push towards a
| certain outcome that I dislike, I'm going to tend to
| dislike that library.
|
| You're also never going to convince me that creating an
| interface for a single implementation is the best way to
| facilitate unit testing.
|
| Mox is nothing more than a kludge to get around the
| limitations of the language and I think there are much
| less invasive kludges available. To be honest when I'm
| working with devs who are new to elixir I'm embarrassed
| when I have to explain how Mox works.
|
| >but in practice it's better to hardcode things rather
| than trying to make it generic and swappable the first
| time around. Junior devs have a tendency to over-engineer
| code "because we might need to change it later." My motto
| is "copy and paste it three times before adding a layer
| of abstraction"
|
| This I agree with 100%
| gashcrab wrote:
| Could you share the less invasive kludges you've found? I
| share your frustration with Mox, although maybe not so
| passionately.
| sarchertech wrote:
| I've had a good experience with Mimic for one.
| sph wrote:
| > I hope you see the irony in using "you're holding it
| wrong" here?
|
| Not really. If you use a hammer on a screw, is it the
| hammer's fault? Sounds like the code base you are working
| on is badly-organised, rather than Mox being the root of
| all evil. There are far too many stories of people
| picking up Elixir and coding in it as if it were Python
| or Ruby. You cannot base your experience with it on a
| codebase written by people new to functional languages.
|
| And no one forces you to use Mox, anyway. It's far from
| being a core Elixir component. It is just _one_
| implementation of the mock object pattern that comes on a
| standalone, optional library.
|
| > Mox is nothing more than a kludge to get around the
| limitations of the language
|
| Mocks are not an Elixir invention, and exist in pretty
| much all languages.
|
| https://en.wikipedia.org/wiki/Mock_object
| sarchertech wrote:
| "You're holding it wrong" was a response from Steve Jobs
| blaming users for a design flaw.
|
| >Mocks are not an Elixir invention
|
| I have no problem with Mocks, I have a problem with Mox.
| Mox is the kludge, not Mocks in general.
| goosejuice wrote:
| > You're also never going to convince me
|
| Such confidence, how do you approach testing your
| clients? Personally I have never used Mox, but I read the
| originating blog post when it came out and have followed
| it's principles ever since.
| thibaut_barrere wrote:
| > Mox is nothing more than a kludge to get around the
| limitations of the language and I think there are much
| less invasive kludges available. To be honest when I'm
| working with devs who are new to elixir I'm embarrassed
| when I have to explain how Mox works.
|
| There are other ways, more similar to Ruby etc (e.g.
| https://github.com/edgurgel/mimic, or with_mock etc).
|
| And also there is the risk to "over-mock", for sure.
|
| But a good middle ground brings the right value here!
| sarchertech wrote:
| I've used mimic an I think it's a much better option than
| Mox.
| sph wrote:
| We spoke about this topic for far too long today, but
| looking at Mimic and its _" nothing is mocked unless I
| tell you to"_ design, it works only iff you are using
| mocks the wrong way and at the wrong boundaries.
|
| If used as intended, you will want mocks to ALWAYS be
| active in tests, which is exactly the idea Mox pushes you
| towards.
| throwawaymaths wrote:
| > Mox is nothing more than a kludge to get around the
| limitations of the language and I think there are much
| less invasive kludges available.
|
| You're definitely using or setting up mox wrong. Mox is
| incredibly good: it lets you do things in elixir that you
| definitely can't do in other languages: concurrently
| running tests with different mocks.
| thibaut_barrere wrote:
| I hear your pain: if boundaries are not correctly
| established, it can be problematic, as with any
| "boundary/interface" system.
|
| Finding the right spot is indeed important here!
| throwawaymaths wrote:
| > I hate Mox with a passion. Because I hate separating out
| the interface and implementation of something that is only
| ever going to have one real implementation
|
| Are you injecting mox or substituting it as a dependency at
| compile time using an attribute?
| depr wrote:
| I agree though I wouldn't say I hate Mox, rather I hate
| working with it (the things it does do it does well). As
| you say it forces you to do stupid stuff, like creating an
| interface for something with one implementation, but also
| the way you have to then add that to your configuration.
| I've also seen people only add "@callback" to modules,
| without any behaviour, because then it works with Mox.
| oxidant wrote:
| You can write a using macro to automatically create a
| behaviour from specs in a module with
| https://hexdocs.pm/elixir/Module.html#spec_to_callback/2
|
| It's been a minute but I've done it.
|
| The behaviour/implementation setup reminds me of Java,
| especially when they're exactly the same.
|
| I do recommend only using mox as far down as possible (only
| around libraries).
|
| Hammox is Mox on steroids, check it out too.
| SSLy wrote:
| what's up with the swedes and their insistence of using : in lieu
| of apostrophe?
| emj wrote:
| It is often used in swedish technical writing to separate and
| foreign word or an acronym from the plural so "VPS:es" in
| english is VPS:erna in swedish. I do not know if this is a
| swedish tradition, it most probably exists in Germany or the
| US. I've seen this use in Sweden for several decades.
| SSLy wrote:
| I've seen it used by Swedes and Swedish speaking Finns, at
| least.
| bobwaycott wrote:
| Even in English, the use case you're referring to would not
| have an apostrophe--apostrophes are used for contractions and
| possession, not plurality.
| penguin_booze wrote:
| Meet our hero: https://www.bbc.co.uk/news/av/uk-39459831.
| atombender wrote:
| It's a fairly old tradition specific to Swedish and Finnish.
| Wikipedia has an explanation. [1]
|
| [1] https://en.wikipedia.org/wiki/Colon_(punctuation)
| losvedir wrote:
| Nice post. One part that raised a red flag for me was:
| defp bucket!, do: System.fetch_env!("BUCKET_NAME")
|
| This will look at that environment variable at build time. For
| many workflows this doesn't work because you may build the app in
| CI and run it with different buckets in production and staging.
|
| I always default to using Application.get_env/2 and setting the
| value in runtime.exs, now that that's a thing in recent Elixir.
|
| For things where performance matters in a hot loop, it might be
| worth compiling in the value, but I wouldn't imagine that's the
| case here where it's a part of a network call to AWS.
| ilikehurdles wrote:
| This will look at it at runtime not build time.
| `System.fetch_env!("BUCKET_NAME")` is still in the body of the
| function and won't get executed until it's called after
| application start.
|
| It is still a good reason to use Application.get_env, and it
| should be preferred over peppering system environment variable
| fetches all over the application.
| losvedir wrote:
| Oh, of course. Brain fart moment. I pattern matched (ha!) to
| code I've seen too much where that's at compile time.
|
| Yeah, runtime here _works_ but I agree is still not as good
| as Application.get_env. Aside from (IMO) code clarity
| reasons, last I checked it is (or can be) much quicker.
| Application is a quick ETS lookup, while System reaches out
| to the system and I forget the details but at one point that
| was much slower.
| thibaut_barrere wrote:
| As pointed out, it's not compile time but runtime here unless
| I'm missing anything. And it would be a good reason to build a
| Credo check exactly for this (I've done a bit of research about
| this).
|
| Any calls to System in regular compile time config (e.g.
| config/prod/dev/test or included) should be forbidden, same for
| attribute-level / meta code.
|
| I hope we will get this one day!
| 29athrowaway wrote:
| All Elixir projects I've seen, when you run "mix xref" you will
| see spaghetti.
| leftsaidtim wrote:
| Oh boy another underjord post. It's always such a joy getting to
| read his writing.
___________________________________________________________________
(page generated 2024-03-23 23:02 UTC)