Certbot generated certificates and Elasticsearch integration
reddit / selfhosted

----------------------------------------------------------------------
ro55mo · 2023-12-28 10:01

So the other day I finally got around to deploying a Caddy server as a
reverse proxy. The purpose of this (at this point) is to facilitate
Certbot functionality in terms of obtaining signed certificates to use
in my Elasticsearch cluster.

I got all this working. Yay for me!

One issue remains.

When Certbot downloads certificates they are placed in

**/etc/letsencrypt/archive/servername**

with symbolic links in

**/etc/letsencrypt/live/servername**

If I point my elasticsearch config at these symbolic links (as the
documentation indicates) I get permissions errors and Elasticsearch
will not start.

If I copy the certs to

**/etc/elasticsearch/certs**

then adjust by elasticsearch config file and apply suitable
permissions everything works.

I understand certbot can be directed to place the certs in different
locations with **--cert-path**

Can certbot be directed to adjust the permissions of the certs? Or
would it just be easier to use something like Ansible to check for new
certs and then copy them then apply suitable permissions?

----------------------------------------------------------------------
ElevenNotes · 2023-12-28 10:58

What are the permission issues you are having? Yes, copy is the
easiest solution. You don't need Ansible for that to work. A simple cp
or rsync is enough. You could also use [my
image](https://hub.docker.com/r/11notes/certbot) that generates all
certs in a path of your choosing on your local file system.

----------------------------------------------------------------------
ro55mo · 2023-12-28 11:32

The Elasticsearch log essentially said it couldn't read the
certificates.

```
org.elasticsearch.ElasticsearchSecurityException: failed to load SSL
configuration [xpack.security.http.ssl] - cannot read configured PEM
certificate_authorities
[/etc/letsencrypt/live/els01.DOMAIN.net/fullchain.pem] because access
to read the file is blocked; SSL resources should be placed in the
[/etc/elasticsearch] directory
Caused by: org.elasticsearch.common.ssl.SslConfigException: cannot
read configured PEM certificate_authorities
[/etc/letsencrypt/live/els01.DOMAIN.net/fullchain.pem] because access
to read the file is blocked; SSL resources should be placed in the
[/etc/elasticsearch] directory
```

----------------------------------------------------------------------
ElevenNotes · 2023-12-28 11:40

Could it be that you run caddy as root and the cert's are all owned by
root and you run elasticsearch as it's own user? You should always
make sure permissions are correct.

----------------------------------------------------------------------
ro55mo · 2023-12-28 12:25

Caddy is installed on a separate server so I don't think it is getting
involved.

But yes, that is essentially it. Elasticsearch processes are run by
elasticsearch user but certbot wants to be installed with sudo so it
must be running as root. As such the certs in
**/etc/letsencrypt/live/els01.DOMAIN.net** are owned by root.

----------------------------------------------------------------------
ElevenNotes · 2023-12-28 12:41

You should not run any app as root. Caddy and Certbot should both not
run as root, neither should Elasticsearch. Check my
[image](https://hub.docker.com/r/11notes/certbot) again that runs
Cerbot as 1000:1000.

----------------------------------------------------------------------
ro55mo · 2023-12-28 12:54

I checked Caddy and it is running as its own caddy user. I will have a
further play about with certbot.

[https://certbot.eff.org/faq#does-certbot-require-root-administrator-
privileges](https://certbot.eff.org/faq#does-certbot-require-root-
administrator-privileges)

As for your image, I am not a Docker user as the last time looked into
it their was a requirement for iptables and all my servers use
nftables.

----------------------------------------------------------------------
threedaysatsea · 2023-12-28 15:00

Your best bet here is to use certbot's built-in renewal-hooks
functionality. Scripts placed in /etc/letsencrypt/renewal-
hooks/deploy/ will run after cert deployment.

Here's a script I use for kibana:

    #!/bin/bash

domain=myelasticserver.domain.com # using your example name
node_dir=/etc/kibana/
node_user=kibana

cp /etc/letsencrypt/live/$domain/{fullchain,privkey}.pem "$node_dir"/
chown $node_user "$node_dir"/*.pem
chmod g+r "$node_dir"/*.pem

This script copies the certs and modifies the ownership so that they
can be read by the kibana user.

Make sure to +x the script after creation so it's marked as
executable.

----------------------------------------------------------------------
ElevenNotes · 2023-12-28 15:31

By not using containers you are missing out on **a lot**.