Mastodon docker portainer deployment

2024.08.25

As this page may end up being the Mastodon "landing" page some additional background is called for.

Firstly the "verified" backlink is a requirement for Mastodon: Mastodon

  • Mastodon here is self hosted on the cluster of 16+ computers.

  • Everything at ElectricBrain is based on Linux and Dockerized with the help of Portainer
  • Overall stack architecture

Some nitty gritty - while it's still fresh

Postgre SQL setup: Adjust the security on the postgres db container to allow anyone to connect. This is OK here since nobody can actually get to the DB except containers attached to the appropriate network. Edit the pg_hba.conf file by adding "host all all 10.0.0.0/8 trust" to the IPv4 section. This means the web container can attach without providing a user ID and password.

version: '3.7'
services:
  db:
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: 
           - "node.platform.arch!=x86_64"
           - "node.labels.public==true"
      resources:
        limits:
          cpus: '2.00'
          memory: 1024M
        reservations:
          cpus: '0.25'
          memory: 256M
    healthcheck:
      test: ['CMD', 'pg_isready', '-U', 'postgres']
    image: registry:5000/postgres:14-alpine
    networks:
      - mastodon-network
    volumes:
      - postgress/var/lib/postgresql/data:/var/lib/postgresql/data
    environment:
      - 'POSTGRES_HOST_AUTH_METHOD=trust'

  redis:
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: 
           - "node.platform.arch!=x86_64"
           - "node.labels.public==true"
      resources:
        limits:
          cpus: '1.00'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 128M
    image: registry:5000/redis:7-alpine
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
    networks:
      - mastodon-network
    volumes:
      - redis/data:/data

  mastodon:
#   command: bundle exec puma -C config/puma.rb
    command: /root/startup-mastodon.sh
    depends_on:
      - db
      - redis
      # - es
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: 
          - "node.platform.arch!=x86_64"
          - "node.labels.public==true"
      resources:
        limits:
          cpus: '2.00'
          memory: 2G
        reservations:
          cpus: '0.50'
          memory: 512M
    environment:
      # env_file: .env.production
      # This is a sample configuration file. You can generate your configuration
      # with the `bundle exec rails mastodon:setup` interactive setup wizard, but to customize
      # your setup even further, you'll need to edit it manually. This sample does
      # not demonstrate all available configuration options. Please look at
      # https://docs.joinmastodon.org/admin/config/ for the full documentation.
      #
      # Note that this file accepts slightly different syntax depending on whether
      # you are using `docker-compose` or not. In particular, if you use
      # `docker-compose`, the value of each declared variable will be taken verbatim,
      # including surrounding quotes.
      # See: https://github.com/mastodon/mastodon/issues/16895
      #
      # Federation
      # ----------
      # This identifies your server and cannot be changed safely later
      # ----------
      - LOCAL_DOMAIN=mastodon.electricbrain.au
      #
      # Redis
      # -----
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      #
      # PostgreSQL
      # ----------
      - DB_HOST=db
      - DB_USER=mastodon
      - DB_NAME=mastodon_production
      - DB_PASS=
      - DB_PORT=5432
      #
      # OpenSearch
      # Elasticsearch (optional)
      # ------------------------
      - ES_ENABLED=true
      - ES_HOST=https://node-0.example.com
      - ES_PORT=9200
      # Authentication for ES (optional)
      - ES_USER=mastodon_system
      - ES_PASS=
      # Cluster level settings
      - ES_PRESET=small_cluster
      - ES_PREFIX=mastodon
      #
      # Secrets
      # -------
      # Make sure to use `bundle exec rails secret` to generate secrets
      # -------
      - SECRET_KEY_BASE=
      - OTP_SECRET=
      #
      # Web Push
      # --------
      # Generate with `bundle exec rails mastodon:webpush:generate_vapid_key`
      # --------
      - VAPID_PRIVATE_KEY=
      - VAPID_PUBLIC_KEY=BBW2Iyh-09LHHGG0GLVgByZJBa73at9akp8aTVkSQOWiS20ZADR4ruBeRlK6ah5k17YY2LzXN_UuQavpf79vy0w=
      #
      # Sending mail
      # ------------
      - SMTP_SERVER=mail.electricbrain.au
      - SMTP_PORT=587
      - SMTP_LOGIN=mastodon
      - SMTP_PASSWORD=
      - SMTP_FROM_ADDRESS=mastodon@electricbrain.au
      #
      # File storage (optional)
      # -----------------------
      - S3_ENABLED=false
      - S3_BUCKET=files.example.com
      - AWS_ACCESS_KEY_ID=
      - AWS_SECRET_ACCESS_KEY=
      - S3_ALIAS_HOST=files.example.com
      #
      # IP and session retention
      # -----------------------
      # Make sure to modify the scheduling of ip_cleanup_scheduler in config/sidekiq.yml
      # to be less than daily if you lower IP_RETENTION_PERIOD below two days (172800).
      # -----------------------
      - IP_RETENTION_PERIOD=31556952
      - SESSION_RETENTION_PERIOD=31556952
      #
      # Make it work behind the electricbrain reverse proxy (which doesn't talk https to back-ends)
      #
      - FORCE_HTTPS=false
    healthcheck:
      # prettier-ignore
      test: ['CMD-SHELL', 'wget -q --spider --proxy=off localhost:3000/health || exit 1']
    image: registry:5000/tootsuite/mastodon:v4.2.10
    networks:
      - ebrain_rproxy_v2_network
      - external_network
      - mastodon-network
      - opensearch_network
    ports:
      - 3000:3000
    secrets:
      - MASTODON_DB_PASS
      - MASTODON_ES_PASS
      - MASTODON_SECRET_KEY_BASE
      - MASTODON_OTP_SECRET
      - MASTODON_VAPID_PRIVATE_KEY
      - MASTODON_SMTP_PASSWORD
    user: "root:root"
    volumes:
      - mastodon/root/startup-mastodon.sh:/root/startup-mastodon.sh
      - mastodon/mastodon/public/system:/mastodon/public/system
      - mastodon/opt/mastodon/config/environments:/opt/mastodon/config/environments

  sidekiq:
#   command: bundle exec sidekiq
    command: /root/startup-mastodon.sh
    depends_on:
      - db
      - redis
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: 
          - "node.platform.arch!=x86_64"
          - "node.labels.public==true"
      resources:
        limits:
          cpus: '2.00'
          memory: 2G
        reservations:
          cpus: '0.50'
          memory: 512M
    environment:
      # env_file: .env.production
      # This is a sample configuration file. You can generate your configuration
      # with the `bundle exec rails mastodon:setup` interactive setup wizard, but to customize
      # your setup even further, you'll need to edit it manually. This sample does
      # not demonstrate all available configuration options. Please look at
      # https://docs.joinmastodon.org/admin/config/ for the full documentation.
      #
      # Note that this file accepts slightly different syntax depending on whether
      # you are using `docker-compose` or not. In particular, if you use
      # `docker-compose`, the value of each declared variable will be taken verbatim,
      # including surrounding quotes.
      # See: https://github.com/mastodon/mastodon/issues/16895
      #
      # Federation
      # ----------
      # This identifies your server and cannot be changed safely later
      # ----------
      - LOCAL_DOMAIN=mastodon.electricbrain.au
      #
      # Redis
      # -----
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      #
      # PostgreSQL
      # ----------
      # DB_HOST=/var/run/postgresql
      - DB_HOST=db
      - DB_USER=mastodon
      - DB_NAME=mastodon_production
      - DB_PASS=
      - DB_PORT=5432
      #
      # OpenSearch
      # Elasticsearch (optional)
      # ------------------------
      - ES_ENABLED=true
      - ES_HOST=https://node-0.example.com
      - ES_PORT=9200
      # Authentication for ES (optional)
      - ES_USER=mastodon_system
      - ES_PASS=
      # Cluster level settings
      - ES_PRESET=small_cluster
      - ES_PREFIX=mastodon
      #
      # Secrets
      # -------
      # Make sure to use `bundle exec rails secret` to generate secrets
      # -------
      - SECRET_KEY_BASE=
      - OTP_SECRET=
      #
      # Web Push
      # --------
      # Generate with `bundle exec rails mastodon:webpush:generate_vapid_key`
      # --------
      - VAPID_PRIVATE_KEY=
      - VAPID_PUBLIC_KEY=BBW2Iyh-09LHHGG0GLVgByZJBa73at9akp8aTVkSQOWiS20ZADR4ruBeRlK6ah5k17YY2LzXN_UuQavpf79vy0w=
      #
      # Sending mail
      # ------------
      - SMTP_SERVER=mail.electricbrain.au
      - SMTP_PORT=587
      - SMTP_LOGIN=mastodon
      - SMTP_PASSWORD=
      - SMTP_FROM_ADDRESS=mastodon@electricbrain.au
      #
      # File storage (optional)
      # -----------------------
      - S3_ENABLED=false
      - S3_BUCKET=files.example.com
      - AWS_ACCESS_KEY_ID=
      - AWS_SECRET_ACCESS_KEY=
      - S3_ALIAS_HOST=files.example.com
      #
      # IP and session retention
      # -----------------------
      # Make sure to modify the scheduling of ip_cleanup_scheduler in config/sidekiq.yml
      # to be less than daily if you lower IP_RETENTION_PERIOD below two days (172800).
      # -----------------------
      - IP_RETENTION_PERIOD=31556952
      - SESSION_RETENTION_PERIOD=31556952
      #
      # Make it work behind the electricbrain reverse proxy (which doesn't talk https to back-ends)
      #
      - FORCE_HTTPS=false
    healthcheck:
      test: ['CMD-SHELL', "ps aux | grep '[s]idekiq\ 6' || false"]
    image: registry:5000/tootsuite/mastodon:v4.2.10
    networks:
      - mastodon-network
      - external_network
    secrets:
      - MASTODON_DB_PASS
      - MASTODON_ES_PASS
      - MASTODON_SECRET_KEY_BASE
      - MASTODON_OTP_SECRET
      - MASTODON_VAPID_PRIVATE_KEY
      - MASTODON_SMTP_PASSWORD
    user: "root:root"
    volumes:
      - sidekiq/root/startup-mastodon.sh:/root/startup-mastodon.sh
      - mastodon/mastodon/public/system:/mastodon/public/system
      - mastodon/opt/mastodon/config/environments:/opt/mastodon/config/environments

networks:
  mastodon-network:
    driver:     overlay
    attachable: true
    internal:   true
    name:       mastodon-network
  opensearch_network:
    external: true
    name:       dmz-opensearch-network
  ebrain_rproxy_v2_network:
    external: true
    name:       dmz-rproxy-v2-network
  external_network:
    external:   true
    name:       dmz-external-network

secrets:
  MASTODON_DB_PASS:
    external: true
  MASTODON_ES_PASS:
    external: true
  MASTODON_SECRET_KEY_BASE:
    external: true
  MASTODON_OTP_SECRET:
    external: true
  MASTODON_VAPID_PRIVATE_KEY:
    external: true
  MASTODON_SMTP_PASSWORD:
    external: true

In order to bootstrap the mastodon stack with these 4 container various commands have to be run to set things up.

  1. Disable the web container from running anything. To do this change out its "command" to essentially make it just sit there and wait.
    command: /bin/sleep 6000s
    This will cause it to simply spin doing nothing for 100 minutes.
  2. Ensure to comment out the health check for this container.
  3. Run a command prompt in the db container (docker knowledge assumed). su to the postgres user and create the named user/password and database.
  4. Run a command prompt in the main web container and initialize the system (docker knowledge needed).
    bundle exec rake mastodon:setup
    The above command fails at a certain point complaining it is unable to talk to Redis - eventhough the correct address/name/password is setup (this is a known thing and doesn't seem to be a problem).
    At that point one simply restarts the container with the original bundle exec puma -C config/puma.rb command and everything seems to work.
  5. Run a command prompt in the web container and fiddle about with the tootctl program. This program is able to perform all sorts of administrative functions on the system. Most notable is the accounts command which can reset passwords and email addresses even on the admin account. The only thing to be careful of is deleting the admin account itself (which I did) - I had to blow away the DB and restart the whole process.

Currently accounts cannot be created on the local system. It's unknown why at this point. Fixed. There's a setting to enable account signups.


Initially testing the Mastodon installation using a local /etc/hosts entry pointing to node1 on port 3000 revealed the install wants to redirect everything to https. Here at ElectricBrain the reverse proxy handles all https and forwards the decrypted request over http to the mastodon server. The https redirect issue was noted and resolved in this issue:
https://github.com/mastodon/mastodon/issues/21139
This fix is implemented here and seems to work.

Enabling the OpenSearch cluster to provide search functionality

Essentially the OpenSearch network needs to be attached to the two web facing containers.

Point the search engine host settings to https://opensearch-node-coord.dmz-opensearch-network and port 9200.

Next the CA certificate has to be added to Ruby in the Web and Sidekiq containers in the /etc/ssl/certs directory.

Once connectivity is esablished the fun part begins. OpenSearch's builtin security is waaay more advanced compared to Elasticsearch's which needs addons to achieve the same result. I'll include the security settings that worked here. The systems here use a centralized LDAP server for pretty much all account information. OpenSearch is configured to also talk to LDAP. This means a service account needs to be added to the LDAP server so that a masterdon service account can get access to OpenSearch.

Once the service account has been provisioned it needs to be added to an LDAP group which is mapped to an OpenSearch Role. Once the role has been provisioned it's security access settings need to be configured. This part took quite some time to get right. The screenshot below show the result.

The relatively tight security specification is due to the OpenSearch cluster doing all sorts of other jobs here at ElectricBrain. Without tight security a programming error in Mastodon's search functionality has the scope to cause a lot of trouble. It is fortunate that the ES_PREPEND setting is available to limit which indices can be touched - that's quite a good feature.

Deploy search

The documentation instructs the administrator to run the below command, with mixed results.

$ bin/tootctl search deploy
/opt/mastodon/vendor/bundle/ruby/3.2.0/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/transport/base.rb:218:in `__raise_transport_error': [403] {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [indices:data/read/scroll/clear] and User [name=mastodon_system, backend_roles=[MastodonFullAccess], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [indices:data/read/scroll/clear] and User [name=mastodon_system, backend_roles=[MastodonFullAccess], requestedTenant=null]"},"status":403} (Elasticsearch::Transport::Transport::Errors::Forbidden)
        from /opt/mastodon/vendor/bundle/ruby/3.2.0/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/transport/base.rb:347:in `perform_request'
        from /opt/mastodon/vendor/bundle/ruby/3.2.0/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/transport/http/faraday.rb:37:in `perform_request'
        from /opt/mastodon/vendor/bundle/ruby/3.2.0/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/client.rb:192:in `perform_request'
        from /opt/mastodon/vendor/bundle/ruby/3.2.0/gems/elasticsearch-api-7.13.3/lib/elasticsearch/api/actions/clear_scroll.rb:50:in `clear_scroll'
        from /opt/mastodon/vendor/bundle/ruby/3.2.0/gems/chewy-7.3.4/lib/chewy/search/scrolling.rb:47:in `scroll_batches'
        from /opt/mastodon/vendor/bundle/ruby/3.2.0/gems/chewy-7.3.4/lib/chewy/search.rb:61:in `method_missing'
        from /opt/mastodon/app/lib/importer/base_importer.rb:49:in `clean_up!'
        from /opt/mastodon/lib/mastodon/cli/search.rb:87:in `block in deploy'
        from /opt/mastodon/lib/mastodon/cli/search.rb:65:in `each'
        from /opt/mastodon/lib/mastodon/cli/search.rb:65:in `deploy'
        from /opt/mastodon/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor/command.rb:28:in `run'
        from /opt/mastodon/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor/invocation.rb:127:in `invoke_command'
        from /opt/mastodon/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor.rb:527:in `dispatch'
        from /opt/mastodon/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor/invocation.rb:116:in `invoke'
        from /opt/mastodon/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor.rb:338:in `block in subcommand'
        from /opt/mastodon/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor/command.rb:28:in `run'
        from /opt/mastodon/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor/invocation.rb:127:in `invoke_command'
        from /opt/mastodon/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor.rb:527:in `dispatch'
        from /opt/mastodon/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor/base.rb:584:in `start'
        from bin/tootctl:9:in `block in main'
        from /opt/mastodon/vendor/bundle/ruby/3.2.0/gems/chewy-7.3.4/lib/chewy/strategy.rb:60:in `wrap'
        from /opt/mastodon/vendor/bundle/ruby/3.2.0/gems/chewy-7.3.4/lib/chewy.rb:154:in `strategy'
        from bin/tootctl:8:in `main'

It appears that most of the above seems to have worked, judging by results below, confirming that the containers can indeed talk to the search engine cluster. The permissions are almost correct. Oddly the containers do have permission to perform the type of operation complained about on indices named mastodon_*. Could it be that Mastodon wants to talk to the .kibana index for some reason?

There are a couple of references to this issue out there in the nature. Essentially it stems from the permission needed being based on an index handle and not an index name. From what I can understand that means it can't be applied at index level and must be applied at cluster level since, presumably, the code at that point doesn't know the index's name and only has access to a handle with which to reference it.

https://stackoverflow.com/questions/76779713/clear-scroll-in-opensearch-elasticsearch-with-read-only-permission
https://forum.search-guard.com/t/query-regarding-scroll-and-clear-permission/2026

And Fixed:

With the addition of the cluster level permission

One further issue which arose due to security is: Can't clear "Could not connect to Elasticsearch. Please check that it is running, or disable full-text search" #31625

The details are discussed in the issue. The fix was to add a further permission allowing the role to get mappings for all indices.

indices:admin/mappings/get