[HN Gopher] Django Styleguide
___________________________________________________________________
Django Styleguide
Author : r4victor
Score : 147 points
Date : 2023-01-11 12:31 UTC (10 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| Rastonbury wrote:
| I found this useful being someone who was self taught django and
| hacked together a project with no idea of architecture. To this
| day I don't even know what a selector is referring to lol never
| used it
|
| I'm learning node now for another mini project, is there anything
| similar? I know how it achieve certain tasks but to structure in
| a proper way I know nothing, all tutorials I've done never really
| go that deep
| j4mie wrote:
| I've been heavily inspired by this styleguide over the years, but
| I still think it's a bit too complex. A few random thoughts:
| - I think "services" is too much of a loaded term. I prefer
| "actions", and I always use the function-based style. - I
| hate the naming of "APIs" in this document. They use the term
| "API" when they mean "endpoint" or "view". - "Reuse
| serializers as little as possible" is the single best piece of
| advice when using DRF. The inline InputSerializer thing is
| brilliant. - Having each each endpoint only handle a single
| HTTP verb is brilliant. - URLs and views should be separate
| from all other business logic (models, actions etc). - For
| read endpoints and associated business logic, I'd encourage
| https://www.django-readers.org/ (disclaimer: I'm the author).
| collyw wrote:
| After 20 years in this industry I firmly believe that the code
| reuse thing is way oversold in universities. It seems to be the
| reason for so much crappy over engineered monstrosities. YAGNI
| and KISS are far better things to aim for. Avoid duplicate
| code, but don't start out aiming for reusable code, when most
| of the time your requirements won't be especially clear.
| miketery wrote:
| > "Reuse serializers as little as possible" is the single best
| piece of advice when using DRF. The inline InputSerializer
| thing is brilliant.
|
| Can you expand on this? What is the InputSerializer as opposed
| to custom rest serializers?
| hbrn wrote:
| I think the idea is that instead of thinking "here's the
| object I'm serializing" you should think "here's the view
| (endpoint) I'm serializing for".
|
| Contrary to what people usually think, the shape of the
| serialized object is typically defined by the API endpoint,
| not by the object itself. Different endpoints can (and will)
| serve different shapes of the same object.
|
| Even if two endpoints serve the same shape today, they can
| deviate tomorrow. When this happens, most people are trying
| to resolve it through DRF inheritance, which is wrong.
| drcongo wrote:
| Can't believe they're putting `class Meta` at the bottom of
| models.
| YPCrumble wrote:
| What's wrong with that?
| drcongo wrote:
| It's a pet peeve more than anything - I just hate it when I
| have to scroll around to find if a class is abstract or not,
| our team puts it at the top so that's never an issue. Having
| it anywhere else means it can be any arbitrary number of
| lines below the class definition making it harder to find.
| [deleted]
| KyeRussell wrote:
| I've never worked on a Django codebase that puts Meta at
| the top of a model definition. Not saying that it's the
| wrong hint to do, but this just feels like feigned surprise
| because you surely also know that it's far from common.
| yiiii wrote:
| The official coding style guide lines already state that it
| should come immediately after the fields.
|
| https://docs.djangoproject.com/en/dev/internals/contributing.
| ..
| wartijn_ wrote:
| No, it doesn't. Custom manager attributes go between the
| two:
|
| ```
|
| The order of model inner classes and standard methods
| should be as follows (noting that these are not all
| required): All database fields
| Custom manager attributes class Meta def
| __str__() def save() def get_absolute_url()
| Any custom methods
|
| ```
|
| And that guideline is not really a general guideline of how
| to make Django projects, but contributing to Django itself.
| It might make sense to keep that convention for projects
| build in Django, but that doesn't make this a style guide
| for those projects.
| selectnull wrote:
| Official Django docs do NOT say the `class Meta` comes
| __immediately__ after the fields.
|
| The docs give the example (as linked by you) looking like
| that, but few chapters down you can find this:
|
| The order of model inner classes and standard methods
| should be as follows (noting that these are not all
| required):
|
| All database fields
|
| Custom manager attributes
|
| class Meta
|
| ...
| greenSunglass wrote:
| I've been working on a project with flask + sqlalchemy. I have
| those sql queries returning a bit of rows (up to 20k) and that
| are quite slow. SQLalchemy does not seem to support caching the
| results and I've started to use flask-caching[1] with redis using
| the @cache.memoize() decorator.
|
| Just wondering if I am taking the right path or if there is
| better alternative.
|
| [1]https://flask-caching.readthedocs.io
| biorach wrote:
| > flask-caching[1] with redis using the @cache.memoize()
| decorator.
|
| > Just wondering if I am taking the right path or if there is
| better alternative.
|
| Yes, this can be a fine solution to slow queries and is used
| very often in many kinds of web applications.
|
| However... 20k rows is not a very big number for a modern dB.
| If the db query is really the slow part then you should
| investigate why - ensure the relevant sql queries are written
| properly, and that the relevant tables are indexed properly for
| the queries that you are running on them.
| zhte415 wrote:
| Indexed?
|
| If complex, what does Sqlalchemy output as the SQL? Could you
| optimise the query? If quite complex, optimise the tables by
| redesigning them?
|
| Was the move to allow Redis to cache persistently across
| requests? Does it do this?
|
| Are you timing each function to look for slowness?
| Frotag wrote:
| I've been using nginx for caching when the response isn't user-
| specific. Pretty painless to set up and doesn't bloat the
| codebase.
|
| (But it's mostly for small personal projects, so grain of salt
| and all.)
|
| https://www.nginx.com/blog/nginx-caching-guide/
| cproctor wrote:
| I've found it helpful in several projects to implement the
| "services layer" described here as a state machine, modeling
| state transitions for a central object (e.g. an article can be
| drafted, submitted, reviewed, published). The state machine
| enforces permissible transitions and handles side effects which
| touch other models.
| warinukraine wrote:
| I hate making webpages, but Django makes it as bearable as it can
| be.
|
| I hate python, but love Django.
|
| I think I have some sort of emotional attachment to Django.
| [deleted]
| KyeRussell wrote:
| I have a long-running thread on the Django forums with a bunch of
| opinions about this topic:
| https://forum.djangoproject.com/t/structuring-large-complex-...
| olau wrote:
| I think there are too many concepts in this, but rather than
| being negative, here are some tips:
|
| Always keep your models slim. Don't stuff template related stuff
| in there. You need to look at those models often, so compact is a
| win. course_has_finished(course) is not much longer than
| course.has_finished(), and will allow you to expand the
| functionality as time goes on. Do precomputation if you need the
| information in a template - that keeps your templates simpler and
| allows you to easily expand the complexity of the precomputation.
|
| Don't use class-based views, at least not outside very specific
| niches like the Django admin. Class-based views will transform a
| simple, composeable call stack into a ball of inheritance mud.
| Inheritance is not a good code reuse tool.
|
| Don't make separate apps in the same project, unless the project
| actually consists of several different, completely independent
| projects. You can make subdirectories without making apps, and
| thus avoid dependency hell.
|
| Also be wary of the formerly South, now built-in migrations
| stuff. It's built around a fragile model (perfect history
| representation), so has lots of foot guns.
|
| And be wary of 3rd party libraries with their own models. You can
| spend a lot of time trying to built bridges to those models
| compared to just doing what makes sense for your particular
| project. I think 3rd party libraries are perhaps best implemented
| without concrete models - duck-typing in Python let us do this.
| This includes Django itself, by the way. User profiles didn't
| become good until Django allowed you to define the user model
| yourself.
| rlawson wrote:
| Good points. Agree with slim models and I personally like a
| service layer. FBVs are great, CBVs are ok, GCBVs are of the
| devil
| collyw wrote:
| I disagree with almost everything you have said.
|
| Fat models think controllers is the suggested strategy. It
| works well in my opinion / experience, though I guess it could
| get out of hand on very large projects.
|
| There is something that feels very unnatural and unintuitive
| about your course_has_finished(course) versus
| course.has_finished() example. This was one of the principles
| behind OOP, keeping your data and functions / methods together,
| though I know OOP isn't trendy these days. It's far more
| natural to have it as a method of course than some random
| function, stored who knows where. I worked on a system with
| this type of design, it was pretty bad.
|
| One thing I would like to see for larger projects is the
| ability to easily split models into separate files - a bit more
| like Java does with one class per file. Maybe you can do this
| already.
|
| Class based views mean that a lot of code is written and tested
| for you. I'll agree that your view does need to be somewhat
| "standard" in what it's doing (anything you would see in the
| admin, list, create, edit, detail, login, etc), so if you have
| something more complex multiple forms in one view, then thy
| don't give a great deal of advantage. In that situation I would
| still likely choose a basic View class over functional views,
| but more for consistency.
|
| I am probably in agreement on separate apps.
|
| Migrations are one of the best features of Django. I have just
| spent 4 years working on a a system without them and it was a
| shambles as you would expect. Everyone is scared to make
| database changes, so you get a ton of shitty application code
| to compensate for shitty database modelling. Tech debt in other
| words.
|
| I can't think of any 3rd party apps where I have used the
| models directly, or if I did it was frictionless enough for me
| not to remember. So no real opinion on that one. 3rd party apps
| can be pretty hit and miss, especially if they get abandoned as
| you upgrade Django, so I would say use with caution,
| particularly for more obscure ones (just been revisiting
| django-celery-beat, that's been around for years so I doubt
| it's going away).
| dec0dedab0de wrote:
| _Always keep your models slim. Don 't stuff template
| ...course_has_finished(course) is not much longer than
| course.has_finished()_
|
| I disagree with this. When trouble shooting or expanding code
| it is super convenient to import a model and have all of your
| methods on auto complete. Especially when you need the same
| functionality in a view, a cron job, a celery task, and an DRF
| end point.
|
| If you want to keep it clean you can put all your methods in a
| mixin class and import it from another file.
|
| _Also be wary of the formerly South, now built-in migrations
| stuff._
|
| Things are much better than they were in South. But yes be
| careful, rule of thumb is always move forward.
|
| _Don 't use class-based views_
|
| Please. For the love of God, always use class based views for
| almost everything. Almost everything you need is a variant of
| one of the built in class based views, don't make me read your
| copy/pasted reimplementation of it.
| asalahli wrote:
| In my view (hehe), Django's class based views is a good idea
| implemented poorly. In theory you should be able to use any
| of the built-in class based generic views with minimal
| customizations to suit your needs, except when you want to do
| such customizations you're left dealing with a huge
| inheritance tree of mixins. It's all magic unless you know or
| wanna read the documentation on what each mixin brings to the
| view, that is _if_ you know what mixins are involved exactly,
| of course.
| collyw wrote:
| Are you aware of https://ccbv.co.uk/? After I discovered
| that, class based views became easy.
| cjohnson318 wrote:
| > Please. For the love of God, always use class based views
| for almost everything.
|
| This. 100%. Leverage as much pre-built stuff you can,
| especially with something as important as your HTTP layer.
| Whenever I run out of CRUD verbs for a model and I need to
| add a custom endpoint, I'll implement it in a separate
| APIView sublcass. Convention over configuration; write boring
| code.
| switch007 wrote:
| Apps should be buried deep in the documentation in some "super
| advanced" section and not advocated much at all
| rajasimon wrote:
| > Don't make separate apps in the same project, unless the
| project actually consists of several different, completely
| independent projects. You can make subdirectories without
| making apps, and thus avoid dependency hell.
|
| I wrote an article on this. My goto strategy is to create a
| project/core/views.py,models.py,apps.py,tests.py
| cjohnson318 wrote:
| Are you talking about this: https://rajasimon.io/blog/django-
| project-structure/
| cjohnson318 wrote:
| > I wrote an article on this.
|
| Link or it didn't happen :)
| randlet wrote:
| "Also be wary of the formerly South, now built-in migrations
| stuff. It's built around a fragile model (perfect history
| representation), so has lots of foot guns."
|
| My experience has been opposite so I'd be interested in hearing
| your experiences if you are willing. I have been using the
| built in migrations since day 1 on a medium sized Django
| project with 350+ migrations and migration issues for our
| project have been exceedingly rare. edit: We have a small team
| of developers, so merge migrations are very rare for us, which
| might be a contributing factor.
| rowanseymour wrote:
| Same here. If I was starting a non-Python project tomorrow
| I'd consider using Django to manage the database schema -
| especially now that we can describe custom indexes and
| constraints in migrations. Our project has gone thru over
| 1000 migrations so far tho we squash them down to 50 or so
| about once a year.
| jbellew wrote:
| Thats interesting, if I was using TypeScript to access the
| data, how would I keep the schemas in sync between TS and
| python?
| anileated wrote:
| Can introspect and export Django models to JSON schema or
| similar, then in TS read it and use compiler low-level
| API to generate types. There may be libraries for either
| stage...
| crucialfelix wrote:
| Prisma works very well and has usable migrations. Not as
| solid as Django's.
|
| Tiangolo of FastAPI fame is working on
| https://sqlmodel.tiangolo.com/ Which is pydantic models,
| SQL alchemy. Migrations are coming soon. This will
| probably be an excellent way to build APIs with db. Then
| you can generate a typescript client from the built in
| OpenAPI schema.
| rowanseymour wrote:
| I've never used TypeScript to talk to a database but
| there might be tooling to generate classes from tables.
| Even if there isn't, manually keeping some TypeScript
| classes in sync might still be worth the effort, for
| being able to manage schema migrations easily elsewhere.
| dnadler wrote:
| We do this manually along with a pydantic as a middleman
| between Django and TS. Works pretty well and is not a
| major inconvenience to keep things aligned.
| replygirl wrote:
| if using DRF you export openapi to json and use openapi-
| typescript-codegen
|
| if using graphene or strawberry you export sdl to json
| and use @graphql-codegen/typescript
| theptip wrote:
| Diamond-dependencies are hard to work with, and so we forbade
| them. Specifically, you can't revert an individual migration,
| you just specify your desired "target" to roll back to, and
| therefore you can't unapply a single branch of a diamond
| dependency. This means if your second branch of a diamond
| dependency breaks the DB, but your app depends on the first
| branch, you're SOL and are now manually running SQL to fix
| your production DB. (Can you tell I'm speaking from
| experience here? :) )
|
| Your migration code uses the model classes, but the
| migrations are using a rehydrated version of the model that
| doesn't include any methods; another footgun. Basically you
| need to copy in any logic that you're using into your
| migration file, or else that migration's logic will change as
| your code is refactored. You might naively think that because
| `model.do_the_thing()` works now, the migration is somehow
| pickling a snapshot of your model class. It's not.
|
| Because of the above, you should really squash migrations
| frequently, but it's a big pain to do so -- particularly if
| you have dependency cycles between apps. ("Just Say No to
| Apps" is my default advice for new Django developers. If you
| have a real need for making a bit of library code sharable
| between multiple different Django projects then you know
| enough to break this rule. Within a single monolithic
| project, apps just cause pain.) Squashing migrations quickly
| gets to some extremely gnarly and hard-to-work-with logic in
| the docs.
|
| Moving models between apps isn't supported by the migration
| machinery; it's an involved and dangerous process. One idea
| here that might save Apps is if you manually remove the App
| Prefix from your "owned" / internal apps; if I have Customer
| and Ledger apps, I don't really need to namespace the tables;
| `user`, `information`, `ledger_entries` are fine table names
| instead of `customer_user`, `customer_information`,
| `ledger_ledger_entries`, a normal DB admin would not
| namespace the table names. You neeed the app namespacing to
| make it safe to use someone else's apps, but I think
| namespacing for your own apps inside a single repo is
| harmful.
|
| I find the migration framework to be worth using, but it's
| definitely got some sharp edges.
| randlet wrote:
| Great answer thank you!
| rowanseymour wrote:
| Maybe you need to turn this into an article because over
| the last decade of working with Django, we've learnt the
| same lessons, sometimes the hard way. Learning to never
| import real models into data migrations was a big one.
|
| I recently wanted to move a model between apps and ended up
| going the route of create new table, copy all rows over,
| delete old table. It was annoying, but the only way to make
| it work with regular migrations.
|
| We ended up writing our own script[1] to squash migrations,
| and I'd love to know if there's a better way. We needed
| something that works for clean installs or existing
| installs that already have the current migrations installed
| - so it generates empty migrations which get applied on
| existing installs, and then they get replaced with real
| initial migrations on clean installs starting from a new
| release.
|
| 1. https://github.com/nyaruka/rapidpro/blob/main/tools/squa
| sh_m...
| rglullis wrote:
| > Always keep your models slim.
|
| As simple as possible, but no simpler. Django models are meant
| to deal not just with the data, but also with business logic.
| If `course.has_finished` is a property of the course, why would
| you want to have a separate function outside of the class?
|
| > Do precomputation if you need the information in a template
|
| If the precomputation is only needed in a template, you can
| (should, IMO) use template tags.
|
| > Don't make separate apps in the same project (...) avoid
| dependency hell.
|
| My current pattern here has been to create one "core" project
| where I represent all the internal models of the domain of the
| application, and "adapter" apps if I want to
| interface/integrate with anything from the external world. This
| makes it easier to extend or replace third-party tools.
|
| > (Migrations) It's built around a fragile model.
|
| I wouldn't call it fragile, quite the opposite. There are some
| annoying limitations for sure (I didn't find a reliable way to
| change the primary key of a model, except for creating a whole
| new model and migrating the data to it), but I think they are
| due to a matter of strong safety that the migration can only be
| done if it consistent.
| well1912 wrote:
| At my current company, we've had many teams over the years
| fail to make business logic in model methods work, and I
| think many other people have had similar results. The issues
| usually boil down to some combination of "business logic is
| too coupled to the data model" and "this method lives at an
| intersection of these two models and creates weird dependency
| problems". I now feel that Django puts you down a path for
| failure by naming the DB layer "models" and not giving users
| a decent place to put cross-model domain logic.
|
| My current preference is a functional core-imperative shell-
| style architecture where as much code lives in the functional
| core as possible. It's not very elegant with Django but it
| works fine. Cosmic Python (really accessible and fairly quick
| read if you have the time:
| https://www.cosmicpython.com/book/preface.html) has examples
| that are similar.
| lowercased wrote:
| > I now feel that Django puts you down a path for failure
| by naming the DB layer "models" and not giving users a
| decent place to put cross-model domain logic.
|
| I think a lot of 'MVC-inspired' frameworks fail there, not
| just django. Rails... 'app/helpers' maybe? Laravel 'models'
| is it, and 'services' or a variation is something I see a
| lot of folks adding, but it's not an out of the box
| convention. I can't remember anything specific/explicit in
| the asp.net world either.
| rglullis wrote:
| > The issues usually boil down to some combination of
| "business logic is too coupled to the data model" and "this
| method lives at an intersection of these two models and
| creates weird dependency problems".
|
| _Refactor_ is not a dirty word. The problems you are
| describing seem to be more of the nature of having too many
| things concentrated at specific model classes, and that
| this model should be decomposed, broken down. This is not a
| Django-specific issue.
| matsemann wrote:
| > _If `course.has_finished` is a property of the course, why
| would you want to have a separate function outside of the
| class?_
|
| Because one should avoid passing Django models around. It
| leads to bad design. Have a selector or something that uses
| the ORM, but exposes some dataclass or pydantic model
| instead, and put the logic there.
| traverseda wrote:
| You want to add an ORM to the ORM? Why?
| matsemann wrote:
| How is that adding an ORM to the ORM? I want all django
| orm access to happen at defined places, instead of the
| spaghetti mess it is when people do
| SomeOtherModulesModel.objects.filter(..) and expose
| themselves to the internal workings of that module.
| Access it through a selector instead.
| rglullis wrote:
| If for some strange reason your application has data that
| it was not created with Django, sure.
|
| But aside from that you are just adding another layer of
| abstraction that does not give any benefit when all your
| models are managed by Django already.
| traverseda wrote:
| An orm takes a selector (typically an sql query) and maps
| it onto an object.
|
| What you're describing takes a selector, and maps it onto
| an object. Is it just that you want type hints or
| something?
| VWWHFSfQ wrote:
| Agree with migrations. I've spent a lot of time doing migration
| surgery to get things back into a consistent state. But mostly
| I've found that if you just let Django manage the database how
| it wants to then things will be fine. But get very familiar
| with merge migrations and fake migrations if things start to go
| awry.
|
| I disagree with the perspective on CBVs though. I've been
| programming Django since 0.96 and CBVs have made nearly
| everything easier and better for me.
| Daishiman wrote:
| > Don't use class-based views, at least not outside very
| specific niches like the Django admin. Class-based views will
| transform a simple, composeable call stack into a ball of
| inheritance mud. Inheritance is not a good code reuse tool.
|
| People always say this but well-structured CBVs keep a generic
| interface that you'll be really glad that exists when you have
| 80 views spread across 10 apps.
|
| Composing function-based views is a PITA and when you're
| building an API with a bunch of auth/serialization/cache extras
| being bolted on it's way easier to keep disciplined and
| ordered. It is _trivial_ to mess up the order of callers for
| these things inside function-based views.
| drcongo wrote:
| Agree with all of this, although I do like class based views.
| nicolaslem wrote:
| Me too, I think it's important to acknowledge that no
| approach is perfect. Pick your poison.
| roland35 wrote:
| Is there a better alternative to South (built-in now) for
| migrations? I always had issues but figured it was the de facto
| for a reason.
| rowanseymour wrote:
| > You can make subdirectories without making apps, and thus
| avoid dependency hell.
|
| It's hard to get 100% right and thus our projects always have a
| few lazy foreign key relationships and inline imports to avoid
| cyclical imports... but I think our code is easier to manage
| _because_ we try to model the dependency relationships by
| having things in separate apps.
|
| > Also be wary of the formerly South, now built-in migrations
| stuff
|
| Our experience from South to 4.x has been that the
| models/migrations system has matured significantly and is
| probably now the main selling point for Django for us.
| cjohnson318 wrote:
| I'm currently in circular import hell. My business logic has
| Jobs and Loads, and they both need to update each other under
| certain circumstances. Should these two things/monstrosities
| be lumped into the same app?
| rowanseymour wrote:
| Well you're writing Python so you're never truly stuck when
| you run into dependency issues, but if you feel like you're
| in hell then maybe they do belong in the same app. All
| complex real world apps have complex dependencies between
| entities - all I would argue is that "put them all in the
| same package" generally isn't the most scalable solution.
| hbrn wrote:
| The short answer is yes.
|
| The long answer is if two entities are updating each other
| you might benefit from shifting all update responsibilities
| to one of them. Or even to a third entity that knows about
| both and keeps those two isolated from each other.
| cjohnson318 wrote:
| Woof. Thank you so much. I like the idea of a third
| party, like a mediator.
|
| Would that mediator be another app? Or should it be some
| module sitting in the project directory? (I'm not even
| sure Django would import something like that.)
| hbrn wrote:
| > Would that mediator be another app?
|
| Yes!
|
| It is an app that might not even have any model classes.
| But it will contain business logic. And it will probably
| speak domain language, which is great.
|
| If you're lucky, those two other apps will become
| _pluggable_. You will probably never replace them, but
| separation of concerns is always nice.
|
| The downside of course is that you will have 3 apps
| instead of 1. That's the balance you have to maintain.
| cjohnson318 wrote:
| So, the way I understand it, _job /services.py_ and _load
| /services.py_ depend on _mediator /mediator.py_ which
| depends on _job /models.py_ and _load /models.py_,
| instead of _job /services.py_ ultimately using _load
| /services.py_, and vice versa.
|
| Thanks so much!
| hbrn wrote:
| This is good enough to break circular dependencies
| between individual modules, but keep in mind that
| circular dependencies between the apps remain (e.g. job
| depends on mediator, mediator depends on job).
|
| I usually prefer to resolve those too. If job is small
| enough, all the orchestration of jobs should happen
| through mediator (same for load). If it's not plausible,
| then job can emit signals which mediator subscribes to.
|
| A good place to start is to give a more descriptive name
| to the mediator. Sure, it mediates between the two, but
| it probably does that to implement some business process.
| Can you name it after that process?
| cjohnson318 wrote:
| Holy bovine. This finally worked. I've been working
| around the clock on this for two days now. Thanks again
| stranger!
| cjohnson318 wrote:
| > ...all the orchestration of jobs should happen through
| mediator (same for load)
|
| So, in a smaller app, when a request comes into _job
| /views.py_ or _load /views.py_ then we immediately start
| working with _JobLoadMediator_ to handle business logic
| between the two? I was just going to focus on specific
| tasks between job and load. I 'll probably look into
| signals; I haven't used those in several years and as I
| recall, it felt hacky.
|
| > give a more descriptive name to the mediator
|
| When a Job is deleted, it needs to delete associated
| Loads. And the states of the Loads will affect the state
| of the Job. That's the main cycle I'm looking at right
| now.
| hbrn wrote:
| > when a request comes into job/views.py or load/views.py
| then we immediately start working with JobLoadMediator to
| handle business logic between the two?
|
| In my world, mediator.views and job.views would likely
| have different audiences.
|
| mediator.views is for business domain (e.g. your API).
| Though name would not be mediator, it would be something
| domain-specific.
|
| job.views could be for more low-level internal tooling
| (e.g. analytics/monitoring). Or it could be empty, if job
| is just dumb data object that has a lifecycle but doesn't
| require public API.
|
| > When a Job is deleted, it needs to delete associated
| Loads. And the states of the Loads will affect the state
| of the Job
|
| If you want to keep them apart, signals is the right
| (though not the easiest) answer. job owns (depends on)
| loads, and subscribes to loads signals. load doesn't know
| anything about job (ignore the fact that job is
| referenced in load.models as foreign key, that's just
| limitations of SQL DDL).
|
| The easiest is a subtle dependency: load can access job
| through foreign key, job can access loads through
| related_name. Circular dependency is still there, but it
| is resolved at runtime. This will start to cause pain
| when application grows, but is fine at small scale.
|
| Though again, I'd probably merge those apps: if you can't
| reason about load without job, and can't reason about job
| without load, there's very little benefit in keeping them
| separate. It might make you feel more organized, but the
| if the code is _lying_ about your mental model, this is
| false organization.
| cjohnson318 wrote:
| > mediator.views is for business domain (e.g. your API) >
| job.views could be for more low-level internal tooling
|
| That's very interesting. I'm mostly following the
| architecture from: https://phalt.github.io/django-api-
| domains/styleguide/
|
| I knew this was going to be a large project, about 19
| apps, supporting a trucking and inventory web/mobile app,
| and I wanted a sane/organized way to deal with all of the
| models and relations.
|
| Do you know of some blog posts or books that go into the
| way you normally organize things?
|
| > if you can't reason about load without job, and can't
| reason about job without load
|
| There is a lot of interaction between all of the parts of
| the app, but particularly between Jobs, Loads, Inventory,
| Stages, and Drivers. In the future I might start with one
| app, and then add another if I absolutely have to.
| yiiii wrote:
| > Always keep your models slim.
|
| At least never put your business logic in your views, forms or
| even templates.
| collyw wrote:
| I am in more or less in agreement, except for forms. Surely
| some validation could be considered business logic.
| zenith035 wrote:
| Don't create apps just for the sake of project structure even
| when models from multiple apps are closely related.
|
| Moving models from one app to another is doable but it is a pain.
| It's even worse if you are relying on GFKs.
| chairmanwow1 wrote:
| HN comments are really disparaging, but after reading through I
| really liked the content and am going to pull out their service /
| serializer model to use in my project. Nice opinionated way to
| avoid structuring a program in a bad way.
| theptip wrote:
| On testing, I think "one file & class per thing-to-test" is
| subtly bad advice. It's not harmful in the hands of someone that
| knows what they are doing, but it tends to point engineers
| towards a tightly-coupled test suite, which ends up making
| refactoring more painful and error prone down the road.
|
| If you have one testclass per entity, then if you make any
| changes to the structure of your services/models/entities, you
| must restructure your tests too. This means you can't do the
| "dream refactor" where you don't touch your tests, and
| restructure your code without changing any behavior. If you
| rewrite your tests whenever your structure changes, how can you
| be sure you've not broken your tests?
|
| Instead, I advocate for testing behaviors. In a tightly-
| integrated framework like Django, most of your tests are going to
| be integration tests (i.e. you have a database involved). You
| should bias those tests towards an integration-y approach that
| uses the public interfaces (Service Layer, if you have one) and
| asserts behavior. Ideally the tests should not change if the
| business logic has not changed. (In practice you'll often need to
| add some mapping/helper/setup code to make this true.)
|
| If you have any fat-model type behavior that is explicitly scoped
| to a single model, then you can test that in isolation. Many
| Django projects call these "unit tests" even though they still
| involve reading and writing your model from the DB. I call them
| "focused integration tests". All that matters is that you have
| agreement on terminology inside your project. If you have
| extremely complex domain logic, it can be worthwhile to construct
| "true Unit Tests" that use dummy objects to test logic without
| hitting your DB. I've not found it worthwhile to mock the DB in
| most Django projects though.
|
| To provide an example of where my "test behaviors not classes"
| advice differs from the OP's paradigm, let's say you split out a
| sub-object to provide a pluggable Strategy for a part of your
| Model's behavior -- you don't necessarily need to have detailed
| tests for that Strategy class if it's fully covered by the
| model's tests. Only the edge cases that are awkward to test at
| the higher level need to be tested at the granular level of the
| Strategy. Indeed, the first refactor that just creates a Strategy
| holding your existing behavior need not change any of your
| existing tests at all! Indeed, if you do need to change existing
| tests, that suggests your tests were improperly-coupled to the
| code under test, since a mere structural change like this should
| not affect the behavior of your application. Even after adding
| more Strategy logic, most of your old ModelTests are still good;
| they still test the high-level behavior, and now also test the
| integration between your model and the new Strategy class.
| Basically, test at the most-granular level that gives a clear,
| decoupled test for your behavior; resist testing every entity in
| isolation, because some entities have rich coreographies with
| other entites that make them hard to isolate. Sometimes you have
| to contort and tightly-couple in order to test things at the
| very-lowest-level possible.
|
| Inspiration/further reading: https://blog.cleancoder.com/uncle-
| bob/2017/10/03/TestContrav.... (Grit your teeth through the
| "Socratic dialog" style. The principle being described is
| extremely valuable.)
| hbrn wrote:
| Ah, the typical "where to put business logic in Django".
|
| M in ActiveRecord MVC web frameworks is deeply misunderstood. M
| is not "data model" (it would be called DVC if that was the
| case). M is your _domain_ model. It 's the model of your
| business, model of the real world. It's the core of your
| application.
|
| Another thing that I never understood, why are functions called
| services? Is it a subconscious desire to go back to enterprisey
| kingdom of nouns? (apparently it is [1])
|
| A service is either something that lives on a network (e.g.
| database, payment gateway, microservice). Or a class that has a
| state. Your functions are neither of those, they are just
| functions.
|
| You business logic should live in the "models" namespace. Whether
| you put it on Model classes, or onto custom Managers, or just
| dump them into the module is not important, as long as you keep
| it consistent and keep your apps fairly small and isolated from
| each other.
|
| Django already gives you enough tools to support big "enterprise"
| applications. It is far from perfect, but you'll get much further
| if you embrace the framework instead of fighting it.
|
| If you really are attached to this "services" mindset then Django
| API Domains [2] is your best option.
|
| [1] https://www.b-list.org/weblog/2020/mar/16/no-service/
|
| [2] https://github.com/phalt/django-api-domains
| nprateem wrote:
| Business logic goes in the controller. That's why it's called a
| controller because it controls stuff like access to your data.
| crucialfelix wrote:
| I try to use controllers just to connect incoming events or
| API calls to the business layer.
|
| The control part is more like Traffic Controller . Just
| directing traffic.
| hbrn wrote:
| Oh sweet summer child.
|
| Please read this carefully: https://folk.universitetetioslo.n
| o/trygver/1979/mvc-2/1979-1...
|
| Business logic is supposed to be reused. Controllers in web
| frameworks (views in Django) expose no way to do it.
| nprateem wrote:
| If you want to reuse logic you make another http request to
| the right endpoint.
| hbrn wrote:
| Are you just trolling? Or your cronjobs are really making
| http requests?
| roflyear wrote:
| I never understood why this was so hard or why people
| complicate this so much. You have a segment of your application
| that "does stuff" - some mix of classes and functions. This
| stuff has its own API. Then you have your web views call that
| code through that API (which is probably just calling
| functions...).
|
| No, instead, it is that "does stuff" hast to be its own
| library, or god forbid, its own service that lives somewhere
| else, with its own communication layer, its own auth...
|
| Why are we making this so hard on ourselves?
| hbrn wrote:
| Typically it goes like this:
|
| 1. You found one case where complexity is essential.
|
| 2. That one case is not consistent with the rest of your app,
| and you were taught that inconsistency is bad.
|
| 3. Since you can't remove complexity from that case, for the
| sake of consistency you add complexity to all other cases.
|
| Class-based views is a typical example. You found a place
| where CBVs are useful. Now some parts of your app use
| functions, some use classes, that's inconsistent. Edit your
| style guide to enforce CBV everywhere. Now a simple
| healthcheck endpoint that returns "OK" has to be a class.
|
| As some folks used to say, you can write Java in any
| language.
|
| The right approach, of course, is to say "I'd rather have
| inconsistency than complexity". The challenge is that
| perception of complexity is subjective, but inconsistency is
| objective. So the right approach eventually loses, and every
| organization turns into a bureaucratic hell.
| andybak wrote:
| > We use Celery for the following general cases:
|
| >
|
| > Communicating with 3rd party services (sending emails,
| notifications, etc.)
|
| > Offloading heavier computational tasks outside the HTTP cycle.
|
| > Periodic tasks (using Celery beat)
|
| Sigh. No mention of the trade-offs. There's simpler ways to do
| all these things. Celery is a big complex beast and it always
| pains me to see it as the default suggestion for simple tasks.
| hdjjhhvvhga wrote:
| What would be your first choices for each of the above?
| true_religion wrote:
| My choice for periodic jobs is cronjobs.
| andybak wrote:
| Yeah. Or a simple cron wrapper like django-cron to get the
| best of both worlds.
|
| For background tasks - you can just spawn a background
| process and keep a simple status table in the db so the
| main app can check if it's completed (assuming you even
| need that)
|
| And for task queues that can handle the traffic most sites
| will need there's things like django-huey.
| VWWHFSfQ wrote:
| > For background tasks - you can just spawn a background
| process and keep a simple status table in the db so the
| main app can check if it's completed (assuming you even
| need that)
|
| I don't know if making my own bespoke queue system is a
| great idea. It seems simple enough, but it gets so much
| more complicated once you start seeing issues with it.
| Orphaned task processes sticking around on the server
| forever, concurrency control, error handling, etc. I'll
| pretty much always just use celery and not have to worry
| about it.
| andybak wrote:
| I'd never argue for building your own - just for using
| something simpler than Celery
|
| It's not so bad now as CI/CD, Docker etc have made
| complex deployments easier to handle. But back when I was
| wrestling with Django simply deploying Celery on a new
| host could easily waste an afternoon and all those
| dependencies made me very nervous about the overall
| complexity.
|
| I still weigh carefully anything that adds another long-
| running process or non-Python dependency to my sites.
| VWWHFSfQ wrote:
| Do you mean a cronjob that calls a Django manager command
| to do the work? Or invokes an API method? From my
| experience cronjobs have a lot of downsides as well.
| They're great for doing tasks local to the server the job
| is running on. Not so great for the kinds of tasks
| (periodic or transactional) that Celery/a real queue is
| designed for.
| danjac wrote:
| Crons are fine for running Django commands that are just
| talking to the same services as your web views
| (databases, cache, email services etc). It should be fine
| to let them run in their dedicated VM/containers, using
| Ansible etc to keep the crontab configurations in the
| repo.
|
| Celerybeat has the advantage of better visibility e.g.
| you can configure them in the Django admin and check when
| they are running. If you are not using Celery though and
| your needs are simple, it's easier to just use plain
| crons.
| true_religion wrote:
| I use them for invoking django commands on the same
| server. I do use celery for transactional jobs though.
| It's only periodic jobs that get called with cron. For
| the context, I do this on small web apps with less than
| 20 dedicated servers (real servers not VPS), so there is
| a "manager" server that does nothing but run periodic
| tasks, management interventions, and cleanup operations.
| roflyear wrote:
| This is good for really simple scheduling of stuff:
| https://schedule.readthedocs.io/en/stable/
| notdarkyet wrote:
| Do you have additional examples of those simpler ways? I
| totally understand how Celery can be a hammer and everything is
| a nail type situation.
| j4mie wrote:
| https://github.com/dabapps/django-db-queue
|
| (I am the author)
|
| EDIT: oh hi, Andy :)
| nerdbaggy wrote:
| I am a big fan of Huey. And they have a Django module
| https://huey.readthedocs.io/en/latest/
| nicolaslem wrote:
| Since everyone suggests alternatives to Celery, may I plug my
| own healthier celery?
|
| https://github.com/NicolasLM/spinach
| giancarlostoro wrote:
| I would just use serverless functions to achieve the same
| thing personally, if you're in the cloud already, chances are
| high you can trigger functions based on new records in a
| database or new file uploaded, or what have you. Then you
| don't need to import much outside of the serverless SDK which
| should typically be pretty minimal.
|
| That's how I did a timed function for a Django project we
| were hosting in Azure anyway.
| roflyear wrote:
| Serverless functions are good for some stuff: clear this
| cache on this schedule, run this task when XYZ happens,
| etc..
|
| But that is not the popular usecase for celery. Often you
| want "some code" (that you likely already have written) to
| be executed async. Sure, you can create a public interface
| for "some code" then write a record, that the serverless
| function is looking for, that then calls back to that
| interface (but is it a web interface???? then you have a
| problem where if the job takes too long to complete, what
| about http timeouts ... ) and now you're really creating a
| big circle for something that should be simple: execute
| some code outside of the request (send an email, hit an
| api, whatever).
|
| Serverless functions really shouldn't contain much logic
| either, because it's too complicated to test.
| giancarlostoro wrote:
| Ive used serverless in this way as well when it was a
| long running process, basically the end-user needed to
| upload a Shapefile, and some of them can be quite large,
| so I kicked off an Azure Function once the file was
| uploaded to parse the file in the background. If it's
| something that will halt the browser when it needs to run
| in the background instead, that's where I'll use
| Serverless if it makes sense. I'm not fond of having my
| web server running things in the background it ruins my
| mental model of the web.
| roflyear wrote:
| > I'm not fond of having my web server running things in
| the background it ruins my mental model of the web.
|
| Well yes, and you run into other problems as well (now if
| your process dies or you deploy or something you have to
| be careful to not kill running jobs).
|
| How much of your logic is in the azure function?
| Spivak wrote:
| Because it's mature, well integrated with Django and is a path
| so well-trodden there's a McDonald's on the way. Any possible
| use-case you can imagine for a job queue has been done in
| Celery and documented.
|
| Celery being complicated is also entirely on the operational
| side, once you actually have Celery using it from within your
| app is simple enough.
|
| Cron is _awful_ for this use-case. You end up just inventing
| Celery but worse when you decide how your app and the cron
| scripts communicate. If you wanted just scheduled tasks but
| simpler use something like APScheduler.
| andybak wrote:
| > Celery being complicated is also entirely on the
| operational side, once you actually have Celery using it from
| within your app is simple enough.
|
| Yeah and to some degree improvements in devops quality of
| life in the last few years has softened my view. (I used to
| mainly use Webfaction without access to apt-get and my own
| hand-rolled scripted deployment. Ugh...)
|
| But I'd still usually prefer a pure-Python solution without
| additional persistent processes - assuming there is one and
| it's fairly well-documented. Huey is pretty good from
| recollection.
|
| > Cron is awful for this use-case. You end up just inventing
| Celery but worse when you decide how your app and the cron
| scripts communicate. If you wanted just scheduled tasks but
| simpler use something like APScheduler.
|
| I was advocating for something like this. There's django-cron
| etc which solves issues around communicating with scripts.
|
| I do see both sides of the debate between "complexity" and
| "solves problems out of the box". I'm generally on the Django
| side when the flask vs Django discussion happens. There's
| always trade-offs.
| hbrn wrote:
| > Because it's mature, well integrated with Django and is a
| path so well-trodden
|
| > Cron is awful for this use-case. You end up just inventing
| Celery
|
| Isn't it the other way around?
|
| Crons are way more mature, well integrated (mgmt commands
| don't require 3rd party modules), and extremely well trodden.
| Crons are super predictable, have sensible defaults and
| _plenty_ of tooling. Which you will have to reinvent with
| Celery.
|
| There are some benefits to programmatic crons, but the
| downsides are huge.
| collyw wrote:
| Django subreddit. Someone asks for a way to run a simple cron
| job once a week. Everyone jumps in and shouts "Celery". I chip
| in and say, nah, just set up a cron job, it will be simpler.
| Get downvoted by the hivemind.
| nickjj wrote:
| A long time ago I wrote a blog post about Celery use cases at
| https://nickjanetakis.com/blog/4-use-cases-for-when-to-
| use-c....
|
| It applies to Django, Flask or any Python system using it. All
| of it still applies today.
|
| It covers a few use cases on the before vs after of using
| Celery and touches base on why I'd consider using Celery over
| other solutions such as async / await. The TL;DR is Celery
| brings a lot to the table around tracking and retrying jobs.
| It's also nice to separate your web and worker workloads into
| different processes since they have much different requirements
| usually.
| emptysea wrote:
| I think as a concept a task queue with some workers makes a
| lot of sense but having used Celery in production, it leaves
| a lot to be desired
|
| We've run into various bugs and weird performance gotchas
| (like the workers prefetch jobs which is terrible if they
| aren't all the same size)
| roflyear wrote:
| Agreed. I have had more luck writing my own "jobs" engine
| that does stuff that I need that celery doesn't have anyway
| (retries, some record of execution, rate limiting).
|
| I'm sure if I really tried, I can get celery to be very
| reliable... but I never really got there.
|
| Also for whatever reason I have NEVER been able to get
| celery to be reliable for its scheduling/cron stuff. It
| just starts to fail. I use this library for that, which I
| have never had problems with:
| https://schedule.readthedocs.io/en/stable/
| nickjj wrote:
| What type of situations did you run into while using it?
|
| I've been using Celery for a long time in production now.
| Nothing crazy and it's a fairly basic set up of "execute
| job, thanks!" along with a beat server. Over the last 6-7
| years an off the top of my head guess would be that
| there's been at least 5 million jobs processed. It's not
| huge volume when measured over years but it's been
| stable.
|
| One server was running for 6 months without being
| updated. That's a Celery worker process running for ~180
| days uninterrupted. It served hundreds of thousands of
| jobs without maintenance. A lot of these were pretty
| beefy tasks too like performing HTTP requests that got
| 400mb XML responses and then parsed them. I didn't even
| have things like `worker_max_tasks_per_child` set either.
| roflyear wrote:
| Celery has a lot of gotchas, but primarily the issue I
| have had is with resource consumption. Maybe it is a
| problem with my config, but I haven't had this issue with
| other solutions.
| 69Represente wrote:
| I have a bug with celery i can't solve.
|
| when I send an async job that get data from various APIs and
| write all in a DB, in the case of lot there is lot of data, the
| celery task finish properly to but my flask app becomes
| unresponsive. I have to restart flask to get back to a normal
| state.
|
| Anyone would know where I should check?
| roflyear wrote:
| sounds like a resource issue, maybe you are opening
| application contexts and not handling them properly?
| andrewingram wrote:
| I've been using the beta of Temporal's Python SDK with Django,
| and aside from some minor teething issues, it's _very_ nice.
| aequitas wrote:
| A lot of people use Django with uWSGI, which also comes with
| queues, cron, workers, cache and lots more. I've been stuck
| with Celery on previous projects for reasons. But I've been
| dying to try out uWSGI's built in features for this. Hearing
| great things about it.
|
| https://uwsgi-docs.readthedocs.io/en/latest/Spooler.html
|
| https://uwsgi-docs.readthedocs.io/en/latest/Cron.html
|
| https://uwsgi-docs.readthedocs.io/en/latest/Mules.html
|
| https://uwsgi-docs.readthedocs.io/en/latest/Caching.html
| asalahli wrote:
| Fellow uWSGI fan here. Unfortunately uWSGI is now in
| maintenance mode, not because is complete which would've been
| fine, but because the maintainers are not able to dedicate
| time to it[0]
| OgAstorga wrote:
| Somewhat off-topic but django is fundraising over here
| https://www.djangoproject.com/fundraising/
| spapas82 wrote:
| In a similar fashion, for anybody interested I've written some of
| my guidelines on implementing Django apps:
| https://www.spapas.net/2022/09/28/django-guidelines/
___________________________________________________________________
(page generated 2023-01-11 23:01 UTC)