[HN Gopher] Django 3.2 - News on compressed fixtures and fixture...
___________________________________________________________________
Django 3.2 - News on compressed fixtures and fixtures compression
Author : pauloxnet
Score : 80 points
Date : 2021-04-06 16:02 UTC (6 hours ago)
(HTM) web link (www.paulox.net)
(TXT) w3m dump (www.paulox.net)
| ctur wrote:
| Not including Zstandard is a lost opportunity -- it is far more
| appropriate than XZ or LZMA for real-time applications like
| serving web pages. Should be easy to add, though, and likely
| would be a very nice compromise between space, memory, and speed
| (including tuning various levels of Zstandard compression).
|
| gzip is basically never worth using anymore except for backwards
| compatibility; zstd is both faster and more space efficient.
|
| lz4 similarly is worth adding for utmost speed but I tend to
| think zstd is better in practice (particularly when the control
| plane is Python where perf is somewhat limited anyway).
| pauloxnet wrote:
| > Not including Zstandard is a lost opportunity -- it is far
| more appropriate than XZ or LZMA for real-time applications
| like serving web pages.
|
| The article is about offline creation of compressed data dumps
| and not about real-time application.
|
| The Django code here use only compression algorithms already
| present in the Python standard library to avoid unnecessary
| dependencies.
| theptip wrote:
| Is anyone making extensive use of static fixtures? If so what do
| you like about them vs. using setUpTestData on dynamically
| generated models (optionally using something like FactoryBoy to
| make your fixture code more flexible)? I found static fixtures to
| be brittle and hard-to-maintain, because you need to manually
| edit them, or have code to regenerate them, every time your
| schema changes.
|
| Recently I've been tempted to revisit them and build out a
| manage.py command to run the domain scenarios that generate these
| fixtures (to make it easier to keep them up to date), since when
| testing DB migrations you often actually do want to have
| stale/old-schema DB state, instead of the default of using the
| post-migration code to generate your test fixtures. But I'm still
| not sure I'd prefer them in many tests outside of the migration-
| testing usecase.
| tingletech wrote:
| I've used them in production before inside of AWS beanstalk
| (.ebextensions). Was pretty much a static django site on top of
| a solr index. There was a hidden django where the content team
| would work on the content. When it was ready, we would dumpdata
| on the backend server and upload to S3. New beanstalks would
| grab the file from S3 and loaddata.
| brianwawok wrote:
| Not extensive, it's kind of a last ditch then when I can't
| generate random data. For example, I have some "integration"
| tests that I spin through and generate a huge pile of objects,
| and validate them against XSDs, to make sure the XML I am
| making is valid. I don't really want to factory boy this, I
| just want a smoke screen to make sure things are sane.. and as
| XSD changes, I can re-run the test and catch weird stuff quick.
|
| But this is like... < 1% of my total tests.
| michaelpb wrote:
| I use Django test fixtures in one way!
|
| I have a series of E2E tests with Splinter [0] that go through
| setting up a variety of things from scratch and testing core
| features. Instead of having really long, complicated test that
| goes through the steps, I instead wrote a little helper
| function that "freezes" the DB after each step as a JSON
| fixture (which is then loaded by the next test), to allow me to
| break it down into many smaller tests that "start" from various
| reasonable database states. Works pretty well!
|
| I don't use them however for unit tests or any non-E2E tests. I
| just use helper factory functions with hardcoded test data.
|
| [0] https://splinter.readthedocs.io/en/latest/
| woile wrote:
| In a small project I had, I was using fixtures to populate the
| local dev environment.
|
| ``` make dev/init ## runs compose in dev mode make dev/populate
| ## initializes data ```
|
| would give you an "initialized" data, ready to start working
| on. When the project has some data in it, I consider it a
| closer reflection of prod. If you can (data is not sensible),
| you can update the fixtures with data from prod.
| jlg23 wrote:
| > If you can (data is not sensible), you can update the
| fixtures with data from prod.
|
| Just make sure all devs use the same days from prod, then. I
| once was consulting for a dysfunctional team and everyone
| having different test data drawn from prod made reproducing
| even easy bugs hard and impossible for more complicated bugs.
| CraigJPerry wrote:
| If it wasn't for Haskell's IHP framework then Django, Ruby on
| Rails and CakePHP would be fighting to come dead last in the
| techempower composite benchmark
| https://www.techempower.com/benchmarks/#section=data-r20&hw=...
|
| Django, RoR - they're almost 2 orders of magnitude behind the
| fastest frameworks on the composite test and they're an order of
| magnitude behind the majority of Java / Go - i was going to say
| C# here but i see asp.net is much faster than the surprisingly
| homogenous performing "usual suspects" of enterprise web
| frameworks.
|
| And yet, if i had to launch a site tomorrow, it'd be Django i'd
| reach for - and with zero hesitation. Instagram, Youtube, Github,
| Facebook, Dropbox etc. etc. all launched on these "slowest of the
| slow" stacks and then optimised. Did anyone start out on a
| lithium (c++) or actix (rust) or other incredibly high
| performance stack?
|
| This is all a big diversion anyway, I think these benchmarks
| measure the wrong thing entirely. Instead - show me some measure
| of how easy & cheap it will be for me to change or add a
| behaviour to my site 3 years down the road if i've stuck to the
| prescribed idiomatic approach.
| collinmanderson wrote:
| Looking techempower's benchmark code, specifically the "dbs"
| and "update" functions (which are weighted highly), I think
| there's some optimization that could be done, like using
| .values() and .update() in more places.
|
| https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
|
| Django ends up getting benchmarked against "uvicorn" which just
| uses raw sql, so yeah, of course raw sql is faster. Django has
| a raw sql option too, but they're not using it for the
| benchmark.
|
| https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
| travisjungroth wrote:
| I'm tempted to go FastAPI over Django on my next project, even
| though I've been a big Django user. A little more performance
| and a little less coupling, and maybe you can survive longer as
| a Python monolith. Then convert to Rust/Go/something in the
| unlikely event of wild success.
| maxs wrote:
| Why do you think it reduces coupling? You can use Django apps
| to have decoupled Django services that all run in the same
| gunicorn webserver
| travisjungroth wrote:
| I mean less logical coupling inside the app. It's more in
| the family of a collection of libraries versus a batteries
| included framework. I'm considering that may be the best of
| both worlds. I never really liked the recreating the
| universe feeling of Flask.
| bliteben wrote:
| You might want to check out
| https://github.com/vitalik/django-ninja. I much prefer it
| over restframework.
| andybak wrote:
| Wow. vitalik was one of the first Django freelancers I
| hired when I was getting started. Some of his code is still
| in production for one of my clients.
|
| (Vitaly - if you're watching. Hi!)
| Daishiman wrote:
| Why? The FastAPI library ecosystem is anemic.
|
| Django is fast enough. You'll be spending at most a fraction
| of an FTE salary in hosting even with the best of success,
| and if it comes to more than that, you'll be likely paying
| for a DBA or for an improved caching strategy for
| performance.
| travisjungroth wrote:
| > The FastAPI library ecosystem is anemic.
|
| That's kinda the point.
|
| If your business is making webapps for clients, the fact
| that you can pip install django-half-the-feature is
| awesome. What you trade for that is a big dependency tree
| whose packages sometimes snake their way through your whole
| codebase, since it's all Django anyway.
|
| FastAPI is its own thing, but it's also heavily based on
| some other big libraries that also have their own support.
|
| And anyway, what I was saying about performance and
| decoupling are probably just rationalizations. I focus on
| writing APIs, and that looks nicer to do in FastAPI than
| how I've done it in Django. I also _really_ like type
| hinting, and that 's a big part of things. Right now, I'd
| just like to write more Python and less Django.
| ryanisnan wrote:
| The biggest challenge I encountered when doing this exact
| thing was once you get beyond bootstrapping the project,
| and start implementing your first feature.
|
| Great, you need an ORM, what do you choose? Now you need
| a serializer, what do you choose? What about validation,
| what do you choose? Authentication, what do you choose?
|
| A lot of these questions are boring and are not worth
| spending time thinking about. Almost all choices will
| leave you with some amount of headaches though (as will
| choosing Django).
|
| The question to me boils down to... what questions am I
| interested in solving?
| Nextgrid wrote:
| Django (and pretty much any other framework) is usually "fast
| enough"; typically you'll hit the limits of your datastore
| first, and even if you reach the point where the framework
| becomes an issue it's trivial to scale horizontally due to its
| stateless nature.
| sethhochberg wrote:
| Yup, same story on Ruby/Rails. Yes, its true that our median
| response times could be faster if we spent less time on Ruby
| CPU burn. But its also true that this improvement would still
| be a rounding error for many of our slowest requests, which
| are dominated by calls to databases, external services over
| HTTP, or both.
| brianwawok wrote:
| Some of it may be B2B vs B2C, and volume of usage.
|
| I am B2B. My server costs are like 1% of my revenue using
| Django. Okay, I could shrink this by 95% by using raw C.. but
| who cares?
|
| Now if I were an image hosting CDN and Django bill was 50% of
| my revenue.. it might be a different story.
|
| I think it's the classic right tool for the right job story..
| ericholscher wrote:
| We serve one of the largest sites on the internet (top 2,000
| last time that alexa's top sites worked, 25k requests per
| minute) with Django and a tiny web server budget (~$1k/mo). We
| serve mostly static files, but all requests hit Django at least
| 2-3 times. If it works for us, it will work for you :)
| khrbrt wrote:
| I've never used Django, but is it really that much easier to
| manage than a modern Java framework like SpringBoot or Quarkus?
| It's pretty fast to throw up a REST endpoint, and annotations
| have replaced much of the old boilerplate.
| sidlls wrote:
| Yes, it really is. SpringBoot especially has so much
| unnecessary complexity built in. I'd dread having to debug
| any non-trivial application written using any of the most
| popular Java frameworks.
| Daishiman wrote:
| I've recently had to patch a site built on Spring Boot.
| Coming from Django, it's a disaster.
|
| Spring Boot basically attempts to turn Java from its
| statically compiled origins to an annotation-ladled mess
| full of runtime reflections and compiler pluginitis.
|
| If you're doing that, why not just go the whole hog and use
| an actually dynamic language and a framework that plays up
| the language's strengths instead of working _around_ it?
| bsder wrote:
| I wanted to put together an API to open our front door from
| our phones. I spent way more time putting the phone app
| together than popping open a VPS, installing Django and the
| Django Rest Framework, and coding to handle the requests.
|
| Django is stupidly quick to do "standard" things. And
| Django's "standard" extends a long way. And Django
| understands a lot more about "standard" than you do until
| you've been coding your app for a while.
|
| Yes, Django and Python are a bit "boring" today. For those of
| us with a job to do rather than resumes to build, that's what
| we need.
| zdw wrote:
| Looks like .gz is basically a free lunch in terms of CPU/memory
| impact with a decent size savings.
| jonstewart wrote:
| I was surprised to see xz and bzip2, but not LZ4 or Snappy.
| pauloxnet wrote:
| The Django code use only compression algorithms already
| included in the Python standard library to avoid unnecessary
| dependencies.
| minusf wrote:
| the algorithms chosen are a mix of "most common" and what
| comes out of the box with python's standard library to avoid
| extra dependencies.
| pauloxnet wrote:
| Exactly. I replied also before reading you answer.
| pauloxnet wrote:
| You're totally right. But you can choose also to use xz or bz2
| for smaller files if you have spare cpu/memory.
___________________________________________________________________
(page generated 2021-04-06 23:01 UTC)