[HN Gopher] A guide to organizing settings in Django
___________________________________________________________________
A guide to organizing settings in Django
Author : senko
Score : 93 points
Date : 2021-11-12 11:48 UTC (11 hours ago)
(HTM) web link (apibakery.com)
(TXT) w3m dump (apibakery.com)
| mattupstate wrote:
| While this might be decent advice around the internal details of
| a Django application's settings, what's missing is a mention of
| who this approach is designed for. It appears geared for
| developers who work on the project because the internal details
| (module names like settings.local, settings.production) leak to
| the command line interface. This might be fine in some cases, but
| this advice is not universal. All apps have a configuration API
| and you might consider how that looks depending on who/what is
| running the application, even if it's people who work on the
| code.
| rmnclmnt wrote:
| Dynamic configuration is the first step, second step is the
| validation of configuration. To that aim, one can use Pydantic
| Settings management with extensive validation to avoid footguns
| down the road. If validation is configured properly, if the
| application starts without crashing, it is already on good
| tracks.
| VWWHFSfQ wrote:
| We just have a single settings.py and use django-environ to
| populate it. # settings.py from
| environ import Env ENV = Env() DEBUG =
| ENV.bool("DEBUG", True) SECRET_KEY =
| ENV.str("SECRET_KEY", "debug-secret-key") ALLOWED_HOSTS =
| ENV.list("ALLOWED_HOSTS", default=["*"]) ...
| # Running with local .env $ foreman run -- python3
| manage.py runserver # Running with prod.env
| $ foreman run -e prod.env -- python3 manage.py runserver
|
| It's easy and scales well with a lot of
| settings/environments/developers.
| scwoodal wrote:
| With a single settings file how do you handle things that
| should only ever be configured in a specific environment?
|
| Django debug toolbar for example for a development environment?
| It needs to be listed as an installed app.
|
| Or something specific to tests when the CI/CD pipeline is
| running the test suite.
| VWWHFSfQ wrote:
| The Django Debug Toolbar has a SHOW_TOOLBAR_CALLBACK setting
| that you can give a dotted import path to a function that
| returns True/False whether it should be enabled or not. By
| default it just looks at settings.DEBUG.
|
| Other configurations work the same way.
| danpalmer wrote:
| I realise that this is probably just an example for HN so this
| isn't criticism of your comment! But in case anyone reading
| doesn't know already, `runserver` is not appropriate for
| running in production.
|
| It doesn't have any security hardening, is single threaded and
| blocking, will reload code as it changes, and is generally very
| slow.
|
| It's generally best to use something like gunicorn, uwsgi,
| uvicorn, etc, and also to then host behind something like nginx
| which is designed to be exposed to potentially malicious users.
| justusw wrote:
| Furthermore I would like to add that having DEBUG set to True
| by default could lead you to accidentally enabling Django
| debug mode in production which could lead to accidentally
| leaking secrets.
| VWWHFSfQ wrote:
| We've gone back and forth on this a number of times
| internally and decided for the most part to just keep
| Django's defaults. There are a number of different ways
| that having "secure by default" can break your local
| development experience. For instance, if you accidentally
| run your local Django server with DEBUG=False,
| SECURE_SSL_REDIRECT=True, SESSION_COOKIE_SECURE=True, and
| any of the SECURE_HSTS_* settings your browser will
| basically remember that for all of eternity and you will no
| longer be able to just use http://localhost:8000/ without
| having to go figure out how to get it to "forget" all that
| stuff.
|
| So for local development we follow the principle of least
| surprise, and we have a fairly rigorous policy/procedure
| for configuring and auditing production environment
| settings.
| amirkdv wrote:
| Having experimented with a bunch of different ways to organize
| complex django settings, I think what OP suggests is a local
| optimum that's hard to scale and doesn't solve a bunch of
| important problems (eg ok, credentials are now in local.py which
| is not committed to git. Now how do I get it to people's dev
| machines / CI?)
|
| My key observation is this: If you're actually deploying and
| maintaining a django app you can't escape having quite a few
| environment variables that control different pieces of config.
| Once I accept that I can't shake these off, I'd rather minimize
| all other complexity, including having to reason about multiple
| settings modules that import/override each other in creative
| ways.
|
| What I've settled on is:
|
| 1. Every thing controlled by env vars, not choice of settings
| module
|
| 2. the one and only settings.py toggles config based on env vars,
| say something like this: STORAGE_MODE =
| os.environ['STORAGE_MODE'] STATICFILES_STORAGE =
| { 'local': '...StaticFileStorage', 's3':
| '...S3Boto3Storage' }[STORAGE_MODE]
|
| EDIT: wording and typos
| Alir3z4 wrote:
| Don't do this.
|
| Keep everything in settings.py and use environment variables
| (django-environ) to configure it.
|
| In a project with tens and tens of apps and so many modules
| requiring configurations and easily reaching more than thousand
| settings key, keeping 1 settings.py has not failed me.
|
| With the approach mentioned in the article, you'll get to a point
| where you'll be having so different setting module for each env.
|
| 01. local
|
| 02. testing
|
| 03. CI/CI
|
| 04. E2E
|
| 05. Docker
|
| 06. Kubernetes
|
| 07. staging
|
| 08. release
|
| 09. production
|
| 10. k8s_win.py
|
| 11. k8s_linux.py
|
| 12. k8s_mac.py
|
| 13. heroku.py
|
| 14. pie.py
|
| 15. home.py
|
| 16. vacation.py
|
| 17. ...
|
| Soon you'll get to a place where you need to kind of merge 2 of
| them (settings/docker.py and settings/kubernetes.py) or worse,
| several of them and then hell is unleashed. You get confused,
| your team gets confused, the new team member is confused, no one
| has any idea where to look at.
|
| Just keep it in 1 place, let your brain stay healthier and
| prevent early cancer.
| jmath wrote:
| anything to prevent early cancer
| marcosdumay wrote:
| Well, now you have the exactly equivalent problem of managing
| your environment variables.
|
| What is better depends on your tooling.
| scwoodal wrote:
| Can you share an example of what a single settings file looks
| like that configures the 1 through 17 environments you have
| listed?
| senko wrote:
| OP here. I agree with you. Quoting from the article:
|
| > We can now go back to only having settings.py that imports
| dotenv, fetches the configuration, and initializes Django
| settings as needed.
| mattupstate wrote:
| I'd go so far as to say that configuration details for specific
| runtime environments should never be the concern of the
| application/service/program itself. It's almost always a smell
| to me when I see "dev" "prod" "stage" "local" in code unless
| it's an abstraction over the configuration for some sort of
| logical runtime profile.
| buixuanquy wrote:
| For newbie, I think the best way to learn how to organize a
| Django project is using cookiecutter-django. They follow most
| best practices, and you have already worked docker-compose config
| to quickly up and running without problems.
| silvester23 wrote:
| So, they figured out they can can use os.environ within the
| settings.py? Am I missing something?
| senko wrote:
| OP here.
|
| You can use whatever you want in settings.py. It's "why" that
| matters.
|
| Yeah, you can boil it down to "use env", but do people know why
| it's usually the best approach? In my experience, many don't,
| hence the blog post.
|
| I mean, if it were obvious and accepted truth, Django would be
| doing this by default already.
| leetrout wrote:
| The community writ large is already using django-environ to
| solve the problems you faced.
|
| https://github.com/joke2k/django-environ
| leetrout wrote:
| This isn't wrong advice but their tone of authority seems a bit
| misplaced.
|
| 1. Make settings.py a package
|
| 2. Please do not put anything more than imports in __init__.py
| since you are adding logic downstream.
|
| 3. Use django environ https://github.com/joke2k/django-environ
| which gives you a defined schema of your environment needs with
| casting and default values when needed.
|
| Now you keep your .env files, keep your setting overrides
| (settings/base.py, settings/local.py, etc) when needed AND you
| have defined a schema with non-string types for your environment
| variables.
|
| From the readme you can see how you make a schema with a casted
| type and default value: env = environ.Env(
| DEBUG=(bool, False) )
| senko wrote:
| OP here. There are different styles you could go about, all of
| them having in common that you wan't to pull things from .env
| nicely.
|
| I personally don't prefer django-environ because it's a bit too
| verbose/magical for my tastes, so use python-dotenv. I would
| recommend against using `local.py`. It's easy to overcomplicate
| things here and one of the things I worry about is how easy it
| is to have a new person set the project up and start working on
| it.
|
| So, in general, absolutely agree with you. Details (which
| package to use and how exactly to set up the layout) are always
| going to be preference-based.
|
| Or at least until the nice folks from Black alleviate us from
| that pain as well :)
| leetrout wrote:
| Sure. Claiming "too verbose/magical" for a library that adds
| structure to generally unstructured, smattered config data
| seems a bit misplaced.
|
| You are entitled to your opinion but I don't think trying to
| defend your stance with homemade env parsing functions while
| writing an authoritative sounding blog post helps the
| community as a whole when there are sound libraries and
| patterns that solve this exact problem.
| yourcelf wrote:
| While we started off using `django-environ` to help manage
| environment-based/12-factor settings, we've moved away from it
| in favor of django-classy-settings.
|
| The biggest knock against django-environ is that it does not
| treat the `.env` syntax the same as Docker or bash -- meaning
| that the same environment file can't be reliably used to
| provide variables for both the container and Django.
|
| django-classy-settings has been a joy to use, and its code is
| really simple and readable (~150 lines).
|
| [0] https://github.com/funkybob/django-classy-settings/
| amanzi wrote:
| Have you got any examples of where the same .env file can't
| be used for both Docker and Bash? This is exactly how I do it
| in my django projects and haven't yet run into any issues.
| whalesalad wrote:
| That's how I do it. .env files are terrible if you're using
| them with a language specific loader. The entire point of env
| is that it's an abstraction you can manipulate externally. You
| can setup the env, then launch the program. It allows for
| runtime changes.
|
| https://whalesalad.com/blog/doing-python-configuration-right
|
| You can do it right without packages or external dependencies.
| pphysch wrote:
| Scrolling through the thread, this is my favorite so far.
|
| I like that it avoids counterproductive dependencies for
| loading a dotfile _within the app_. This is an operation that
| should be done at the same level as launches the Python
| interpreter, and Python should then complain if required envs
| are not set.
| abishekg wrote:
| I use python-decouple with django and it's solved this pretty
| elegantly. A dotenv file for local and env variables for
| production. Isn't that the 12 factor recommendation?
|
| https://github.com/henriquebastos/python-decouple
| wombatpm wrote:
| Seconded. I switched from django-environ to python-decouple.
| Now that python-decouple defers to environment variables over
| .env files I find that I have our DevOps people spin up
| containers with secrets configured using their tool de jure
| while I just have a local .env file. We just make sure our
| env.sample is up to date with respect to our .env. DevOps folk
| use that to define environment variables for the container.
| ensignavenger wrote:
| This seems overly complex to me. They have their settings in a
| .env file, they have a separate program/script that loads those
| settings into the environment variables, then they have another
| program that reads them out of the environment and parses them
| into python objects in settings.py... so why not just put them in
| settings.py to begin with?
|
| I have also never agreed with the common suggestion of putting
| secrets into env vars- I believe secrets belong in something like
| Hashicorp's Vault and stored on the running system using proper
| OS level access control.
| [deleted]
| danpalmer wrote:
| The next step from this is the django-configurations package.
| While it's not perfect, it does help when settings get very
| complex.
|
| We've got a Django site that we run in production for different
| partners. We've got 560 settings, and then dev/test/prod configs
| for each partner, plus we vary the list of INSTALLED_APPS per
| partner. django-configurations is pretty nice for making this all
| manageable, and mypy helps to make sure everything is consistent.
| mands wrote:
| +1 for django-configurations, it's been invaluable for us in
| helping to keep settings managable as we build / test / deploy
| across multiple environments and modes
| nettleseyeball wrote:
| Interesting read!
|
| Just wanted to say though I don't think the critique of the
| "settings/__init__.py" approach is accurate.
|
| You can easily access environment variables inside these files
| using os.environ, and it doesn't require a file locally that
| isn't present in the repo. As you say you just specify which
| settings file to use with the DJANGO_SETTINGS_MODULE env var. Am
| I missing something?
| senko wrote:
| Thanks!
|
| Once you switch to using environment variables, having
| additional specialized settings files is a complexity you don't
| need, in my experience. So I prefer to simplify without losing
| much in terms of convenience.
___________________________________________________________________
(page generated 2021-11-12 23:03 UTC)