Archivebox timing out reddit / selfhosted ---------------------------------------------------------------------- DisasterousLamps · 2023-12-15 10:49 So i've been trying to get Archivebox running for the past few hours and i'm at a loss The logs make it seem everything is working properly but when I go to [192.168.1.32:31200](https://192.168.1.32:31200) it just times out. I've tried using Portainer and running the docker compose setup guide through the shell with no luck. Here are the Portainer logs: chown: cannot access '/browsers/*': No such file or directory [i] [2023-12-15 10:29:49] ArchiveBox v0.7.1+editable: archivebox server --quick-init 0.0.0.0:31200 > /data [+] Initializing a new ArchiveBox v0.7.1+editable collection... ---------------------------------------------------------------------- [+] Building archive folder structure... + ./archive, ./sources, ./logs... + ./ArchiveBox.conf... [+] Building main SQL index and running initial migrations... Operations to perform: Apply all migrations: admin, auth, contenttypes, core, sessions Running migrations: Applying contenttypes.0001_initial... OK Applying auth.0001_initial... OK Applying admin.0001_initial... OK Applying admin.0002_logentry_remove_auto_add... OK Applying admin.0003_logentry_add_action_flag_choices... OK Applying contenttypes.0002_remove_content_type_name... OK Applying auth.0002_alter_permission_name_max_length... OK Applying auth.0003_alter_user_email_max_length... OK Applying auth.0004_alter_user_username_opts... OK Applying auth.0005_alter_user_last_login_null... OK Applying auth.0006_require_contenttypes_0002... OK Applying auth.0007_alter_validators_add_error_messages... OK Applying auth.0008_alter_user_username_max_length... OK Applying auth.0009_alter_user_last_name_max_length... OK Applying auth.0010_alter_group_name_max_length... OK Applying auth.0011_update_proxy_permissions... OK Applying auth.0012_alter_user_first_name_max_length... OK Applying core.0001_initial... OK Applying core.0002_auto_20200625_1521... OK Applying core.0003_auto_20200630_1034... OK Applying core.0004_auto_20200713_1552... OK Applying core.0005_auto_20200728_0326... OK Applying core.0006_auto_20201012_1520... OK Applying core.0007_archiveresult... OK Applying core.0008_auto_20210105_1421... OK Applying core.0009_auto_20210216_1038... OK Applying core.0010_auto_20210216_1055... OK Applying core.0011_auto_20210216_1331... OK Applying core.0012_auto_20210216_1425... OK Applying core.0013_auto_20210218_0729... OK Applying core.0014_auto_20210218_0729... OK Applying core.0015_auto_20210218_0730... OK Applying core.0016_auto_20210218_1204... OK Applying core.0017_auto_20210219_0211... OK Applying core.0018_auto_20210327_0952... OK Applying core.0019_auto_20210401_0654... OK Applying core.0020_auto_20210410_1031... OK Applying core.0021_auto_20220914_0934... OK Applying core.0022_auto_20231023_2008... OK Applying sessions.0001_initial... OK √ ./index.sqlite3 [*] Checking links from indexes and archive folders (safe to Ctrl+C)... > Skipping full snapshot directory check (quick mode) ---------------------------------------------------------------------- [√] Done. A new ArchiveBox collection was initialized (0 links). Hint: To view your archive index, run: archivebox server # then visit http://127.0.0.1:8000 To add new links, you can run: archivebox add Logging errors to ./logs/errors.log [!] No admin users exist yet, you will not be able to edit links in the UI. To create an admin user, run: archivebox manage createsuperuser Performing system checks... System check identified no issues (0 silenced). December 15, 2023 - 10:29:59 Django version 3.1.14, using settings 'core.settings' Starting development server at http://0.0.0.0:31200/ Quit the server with CONTROL-C. And here are the Proxmox shell logs: chown: cannot access '/browsers/*': No such file or directory [i] [2023-12-15 09:20:08] ArchiveBox v0.7.1+editable: archivebox init --setup > /data [+] Initializing a new ArchiveBox v0.7.1+editable collection... ---------------------------------------------------------------------- [+] Building archive folder structure... + ./archive, ./sources, ./logs... + ./ArchiveBox.conf... [+] Building main SQL index and running initial migrations... Operations to perform: Apply all migrations: admin, auth, contenttypes, core, sessions Running migrations: Applying contenttypes.0001_initial... OK Applying auth.0001_initial... OK Applying admin.0001_initial... OK Applying admin.0002_logentry_remove_auto_add... OK Applying admin.0003_logentry_add_action_flag_choices... OK Applying contenttypes.0002_remove_content_type_name... OK Applying auth.0002_alter_permission_name_max_length... OK Applying auth.0003_alter_user_email_max_length... OK Applying auth.0004_alter_user_username_opts... OK Applying auth.0005_alter_user_last_login_null... OK Applying auth.0006_require_contenttypes_0002... OK Applying auth.0007_alter_validators_add_error_messages... OK Applying auth.0008_alter_user_username_max_length... OK Applying auth.0009_alter_user_last_name_max_length... OK Applying auth.0010_alter_group_name_max_length... OK Applying auth.0011_update_proxy_permissions... OK Applying auth.0012_alter_user_first_name_max_length... OK Applying core.0001_initial... OK Applying core.0002_auto_20200625_1521... OK Applying core.0003_auto_20200630_1034... OK Applying core.0004_auto_20200713_1552... OK Applying core.0005_auto_20200728_0326... OK Applying core.0006_auto_20201012_1520... OK Applying core.0007_archiveresult... OK Applying core.0008_auto_20210105_1421... OK Applying core.0009_auto_20210216_1038... OK Applying core.0010_auto_20210216_1055... OK Applying core.0011_auto_20210216_1331... OK Applying core.0012_auto_20210216_1425... OK Applying core.0013_auto_20210218_0729... OK Applying core.0014_auto_20210218_0729... OK Applying core.0015_auto_20210218_0730... OK Applying core.0016_auto_20210218_1204... OK Applying core.0017_auto_20210219_0211... OK Applying core.0018_auto_20210327_0952... OK Applying core.0019_auto_20210401_0654... OK Applying core.0020_auto_20210410_1031... OK Applying core.0021_auto_20220914_0934... OK Applying core.0022_auto_20231023_2008... OK Applying sessions.0001_initial... OK √ ./index.sqlite3 [*] Checking links from indexes and archive folders (safe to Ctrl+C)... [*] [2023-12-15 09:20:19] Writing 0 links to main index... √ ./index.sqlite3 ---------------------------------------------------------------------- [√] Done. A new ArchiveBox collection was initialized (0 links). [+] Creating new admin user for the Web UI... Username (leave blank to use 'archivebox'): admin Email address: ********** Password: Password (again): The password is too similar to the username. This password is too short. It must contain at least 8 characters. This password is too common. Bypass password validation and create user anyway? [y/N]: y Superuser created successfully. [+] Installing enabled ArchiveBox dependencies automatically... Installing YOUTUBEDL_BINARY automatically using pip... 2023.10.13 is already installed yt-dlp Installing CHROME_BINARY automatically using playwright... Chromium 119.0.6045.9 is already installed chromium-browser Installing SINGLEFILE_BINARY, READABILITY_BINARY, MERCURY_BINARY automatically using npm... SINGLEFILE_BINARY, READABILITY_BINARY, and MERCURURY_BINARY are already installed [√] Set up ArchiveBox and its dependencies successfully. 0.7.1+editable ArchiveBox v0.7.1+editable Cpython Linux Linux-6.5.11-4-pve-x86_64-with-glibc2.36 x86_64 DEBUG=False IN_DOCKER=True IN_QEMU=False IS_TTY=True TZ=UTC FS_ATOMIC=True FS_REMOTE=True FS_USER=911:911 FS_PERMS=644 SEARCH_BACKEND=ripgrep [i] Dependency versions: √ PYTHON_BINARY v3.11.6 valid /usr/local/bin/python3.11 √ SQLITE_BINARY v2.6.0 valid /usr/local/lib/python3.11/sqlite3/dbapi2.py √ DJANGO_BINARY v3.1.14 valid /usr/local/lib/python3.11/site-packages/django/__init__.py √ ARCHIVEBOX_BINARY v0.7.1 valid /usr/local/bin/archivebox √ CURL_BINARY v8.4.0 valid /usr/bin/curl √ WGET_BINARY v1.21.3 valid /usr/bin/wget √ NODE_BINARY v21.1.0 valid /usr/bin/node √ SINGLEFILE_BINARY v1.1.18 valid /app/node_modules/single-file-cli/single-file √ READABILITY_BINARY v0.0.9 valid /app/node_modules/readability-extractor/readability-extractor √ MERCURY_BINARY v1.0.0 valid /app/node_modules/@postlight/parser/cli.js √ GIT_BINARY v2.39.2 valid /usr/bin/git √ YOUTUBEDL_BINARY v2023.10.13 valid /usr/local/bin/yt-dlp √ CHROME_BINARY v119.0.6045.9 valid /usr/bin/chromium-browser √ RIPGREP_BINARY v13.0.0 valid /usr/bin/rg [i] Source-code locations: √ PACKAGE_DIR 24 files valid /app/archivebox √ TEMPLATES_DIR 4 files valid /app/archivebox/templates - CUSTOM_TEMPLATES_DIR - disabled None [i] Secrets locations: - CHROME_USER_DATA_DIR - disabled None - COOKIES_FILE - disabled None [i] Data locations: √ OUTPUT_DIR 5 files @ valid /data √ SOURCES_DIR 0 files valid ./sources √ LOGS_DIR 1 files valid ./logs √ ARCHIVE_DIR 0 files valid ./archive √ CONFIG_FILE 81.0 Bytes valid ./ArchiveBox.conf √ SQL_INDEX 204.0 KB valid ./index.sqlite3 Hint: To view your archive index, run: archivebox server # then visit http://127.0.0.1:8000 To add new links, you can run: archivebox add < ~/some/path/to/list_of_links.txt For more usage and examples, run: archivebox help ---------------------------------------------------------------------- ______-_-_________ · 2023-12-15 14:11 I'll offer my experience. I had a similar issue. Sometimes Archivebox would timeout, other times take a very long time to load. After having issues with Plex, I learned that if the database is located on my NAS, the container has difficulty accessing it quickly. So for Archivebox I pointed the folder for the database to a location physically on the box it was running and didn't have a single issue after that. I limped along dealing with the shoddy performance for months before I figured out what was wrong. ---------------------------------------------------------------------- OppositeFisherman89 · 2023-12-15 17:20 It typically runs on port 8000. It would be helpful if you posted your docker compose. ---------------------------------------------------------------------- DisasterousLamps · 2023-12-15 21:49 I've tried on port 8000 and other ports Here's the docker-compose: # Usage: # docker compose run archivebox init --setup # docker compose up # echo "https://example.com" | docker compose run archivebox archivebox add # docker compose run archivebox add --depth=1 https://example.com/some/feed.rss # docker compose run archivebox config --set MEDIA_MAX_SIZE=750m # docker compose run archivebox help # Documentation: # https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#docker- compose version: '3.9' services: archivebox: image: ${DOCKER_IMAGE:-archivebox/archivebox:dev} command: server --quick-init 0.0.0.0:31200 ports: - 30070:30070 volumes: - /archivebox:/data # - ./etc/crontabs:/var/spool/cron/crontabs # uncomment this and archivebox_scheduler below to set up automatic recurring archive jobs # - ./archivebox:/app/archivebox # uncomment this to mount the ArchiveBox source code at runtime (for developers working on archivebox) # build: . # uncomment this to build the image from source code at buildtime (for developers working on archivebox) environment: - ALLOWED_HOSTS=* # restrict this to only accept incoming traffic via specific domain name # - PUBLIC_INDEX=True # set to False to prevent anonymous users from viewing snapshot list # - PUBLIC_SNAPSHOTS=True # set to False to prevent anonymous users from viewing snapshot content # - PUBLIC_ADD_VIEW=False # set to True to allow anonymous users to submit new URLs to archive # - ADMIN_USERNAME=admin # create an admin user on first run with the given user/pass combo # - ADMIN_PASSWORD=SomeSecretPassword # - PUID=911 # set to your host user's UID & GID if you encounter permissions issues # - PGID=911 # - SEARCH_BACKEND_ENGINE=sonic # uncomment these and sonic container below for better full-text search # - SEARCH_BACKEND_HOST_NAME=sonic # - SEARCH_BACKEND_PASSWORD=SomeSecretPassword # - MEDIA_MAX_SIZE=750m # increase this filesize limit to allow archiving larger audio/video files # - TIMEOUT=60 # increase this number to 120+ seconds if you see many slow downloads timing out # - CHECK_SSL_VALIDITY=True # set to False to disable strict SSL checking (allows saving URLs w/ broken certs) # - SAVE_ARCHIVE_DOT_ORG=True # set to False to disable submitting all URLs to Archive.org when archiving # ... # add further configuration options from archivebox/config.py as needed (to apply them only to this container) # or set using `docker compose run archivebox config --set SOME_KEY=someval` (to persist config across all containers) # For ad-blocking during archiving, uncomment this section and pihole service section below # networks: # - dns # dns: # - 172.20.0.53 ######## Optional Addons: tweak examples below as needed for your specific use case ######## ### Example: To run the Sonic full-text search backend, first download the config file to sonic.cfg # $ curl -O https://raw.githubusercontent.com/ArchiveBox/Archive Box/master/etc/sonic.cfg # After starting, backfill any existing Snapshots into the full- text index: # $ docker-compose run archivebox update --index-only # sonic: # image: valeriansaliou/sonic:latest # expose: # - 1491 # environment: # - SEARCH_BACKEND_PASSWORD=SomeSecretPassword # volumes: # - ./sonic.cfg:/etc/sonic.cfg:ro # - ./data/sonic:/var/lib/sonic/store ### Example: To run pihole in order to block ad/tracker requests during archiving, # uncomment this block and set up pihole using its admin interface # pihole: # image: pihole/pihole:latest # ports: # - 127.0.0.1:8090:80 # uncomment to access the admin HTTP interface on http://localhost:8090 # environment: # - WEBPASSWORD=SET_THIS_TO_SOME_SECRET_PASSWORD_FOR_ADMIN_DASHBOARD # - DNSMASQ_LISTENING=all # dns: # - 127.0.0.1 # - 1.1.1.1 # networks: # dns: # ipv4_address: 172.20.0.53 # volumes: # - ./etc/pihole:/etc/pihole # - ./etc/dnsmasq:/etc/dnsmasq.d ### Example: Enable ability to run regularly scheduled archiving tasks by uncommenting this container # $ docker compose run archivebox schedule --every=day --depth=1 'https://example.com/some/rss/feed.xml' # then restart the scheduler container to apply the changes to the schedule # $ docker compose restart archivebox_scheduler # archivebox_scheduler: # image: ${DOCKER_IMAGE:-archivebox/archivebox:dev} # command: schedule --foreground # environment: # - MEDIA_MAX_SIZE=750m # increase this number to allow archiving larger audio/video files # # - TIMEOUT=60 # increase if you see timeouts often during archiving / on slow networks # # - ONLY_NEW=True # set to False to retry previously failed URLs when re-adding instead of skipping them # # - CHECK_SSL_VALIDITY=True # set to False to allow saving URLs w/ broken SSL certs # # - SAVE_ARCHIVE_DOT_ORG=True # set to False to disable submitting URLs to Archive.org when archiving # # - PUID=502 # set to your host user's UID & GID if you encounter permissions issues # # - PGID=20 # volumes: # - ./data:/data # - ./etc/crontabs:/var/spool/cron/crontabs # # cpus: 2 # uncomment / edit these values to limit container resource consumption # # mem_limit: 2048m # # shm_size: 1024m ### Example: Put Nginx in front of the ArchiveBox server for SSL termination # nginx: # image: nginx:alpine # ports: # - 443:443 # - 80:80 # volumes: # - ./etc/nginx.conf:/etc/nginx/nginx.conf # - ./data:/var/www ### Example: run all your ArchiveBox traffic through a WireGuard VPN tunnel # wireguard: # image: linuxserver/wireguard:latest # network_mode: 'service:archivebox' # cap_add: # - NET_ADMIN # - SYS_MODULE # sysctls: # - net.ipv4.conf.all.rp_filter=2 # - net.ipv4.conf.all.src_valid_mark=1 # volumes: # - /lib/modules:/lib/modules # - ./wireguard.conf:/config/wg0.conf:ro ### Example: Run PYWB in parallel and auto-import WARCs from ArchiveBox # pywb: # image: webrecorder/pywb:latest # entrypoint: /bin/sh -c '(wb-manager init default || test $$? -eq 2) && wb-manager add default /archivebox/archive/*/warc/*.warc.gz; wayback;' # environment: # - INIT_COLLECTION=archivebox # ports: # - 8080:8080 # volumes: # - ./data:/archivebox # - ./data/wayback:/webarchive networks: # network needed for pihole container to offer :53 dns resolving on fixed ip for archivebox container dns: ipam: driver: default config: - subnet: 172.20.0.0/24 ---------------------------------------------------------------------- OppositeFisherman89 · 2023-12-16 18:45 I would start by uncommenting and changing the PUID and PGID. ``` # set to your host user's UID & GID if you encounter permissions issue s # - PUID=911                        # - PGID=911 ``` Here's what my docker-compose.yml looks like ``` services: archivebox: image: archivebox/archivebox:dev container_name: archivebox command: server --quick-init 0.0.0.0:8000 restart: unless-stopped ports: - 8000:8000 environment: - ALLOWED_HOSTS=* - PUID=1000 - PGID=1000 - SAVE_ARCHIVE_DOT_ORG=False - PUBLIC_INDEX=False - PUBLIC_SNAPSHOTS=False - PUBLIC_ADD_VIEW=False - CHROME_BINARY=chromium volumes: - ./archivebox/data:/data ``` ---------------------------------------------------------------------- DisasterousLamps · 2023-12-17 00:17 Thank you very much, I played around with different PUID/PGID, different users, all sorts of different settings but it would consistently give me "Permission denied" when uncommenting the PUID/PGID fields. In the end I copy & pasted your docker-compose and changed: ports: - 8000:8000 to ports: - 8001:8000 and everything came to life. Thank you very much, i've been racking my brains over this for the past few days. :) ---------------------------------------------------------------------- OppositeFisherman89 · 2023-12-17 00:25 Glad you got it working! ports: - 8001:8000 Changing that means you already have another service running on port 8000. ---------------------------------------------------------------------- dontworryimnotacop · 2023-12-17 21:43 This is mentioned in the README but to be fair it's a very long README, so people don't always see it. https://github.com/ArchiveBox/ArchiveBox#storage-requirements ---------------------------------------------------------------------- ______-_-_________ · 2023-12-17 22:12 Yeah, I found this after the fact. Was kicking myself for not reading it before. ---------------------------------------------------------------------- dontworryimnotacop · 2023-12-17 22:22 It's completely understandable. I'm trying to move more of our important edge-case "gotchas" into automatic CLI hints so it can warn people of problems proactively and prevent this in the future. I don't want to force users to read our docs cover-to-cover to have a good experience.