Fediverse/Installation documentation

part of the Fediverse explorations and installation of the XPUB Mastodon instance (April-June 2025)

Install Mastodon as a software

Following this Installation guide
This guide assumes a bunch of things are already installed, but if you don't have them on your system you will have to install them. We provide here additional guidance because we don't have anything installed. In order to use the installation commands from the guide, we need to log into the root with su and then run the commands

System packages > Yarn

After enabling corepack, the documentation doesn't say it but you should install yarn by just writing yarn in the terminal

Creating the mastodon user

adduser doesn't work? here's how to resolve it:
Unhide the adduser command using the following command:

sudo update-alternatives --install /bin/adduser adduser /usr/sbin/adduser 20

Then run again

adduser --disabled-password mastodon

Setting up Mastodon

To login into the mastodon account, use:

su mastodon

Installing Ruby

All the commands should be ran one by one!!! - make sure to be in the mastodon user as indicated by the tutorial
To resolve the problem about installing rbenv with jemalloc, we do:

exit # out of the mastodon user to go back to root
apt-get install libjemalloc-dev
su - mastodon
RUBY_CONFIGURE_OPTS='--with jemalloc' rbenv install 3.4.4
rbenv global 3.4.4

Installing the last dependencies

Install bundler by running (in root)

apt install bundler
apt install gem

Then run the installation commands (in mastodon user):

bundle config deployment 'true'
bundle config without 'deployment test'
bundle install -j$(getconf _NPROCESSORS_ONLN)
yarn install

When this returns a "no such file or directory" error:

RAILS_ENV=production rails mastodon:setup

Navigate to the live folder and run:

bundle install
RAILS_ENV=production bin/rails mastodon:setup

SETTING UP PostGreSQL host: /var/run/postgresql
PostGreSQL port: 5432
Name of PostGreSQL database: mastodon_production
Name of PostGreSQL user: mastodon
Password for PostGreSQL user:
Redis host: localhost
Redis ports: 6379
Redis password: [empty]
Store on the cloud? No
Send e-mails from localhost? No
SMTP server: smtp.mailgun.org (to reconfigure later, leave the following SMTP prompts empty by hitting enter)
Do you want Mastodon to periodically check for updates and notify you? Yes
Save configuration? Yes
Prepare database now? Yes
Compile the assets now? Yes (we will go back to that later cause it logged errors)
Create admin right away? Yes
Username: admin
E-mail: m.berends@hr.nl
Temp password:

Switch back to root!

systemctl restart mastodon*

Acquiring an SSL certificate

Use the example.com commands, they will be editable later, but you should already have a (sub)domain name registered.
If encountering "An unexpected error" when trying to generate certificate, run

apt install python3-certbot-nginx

Setting up nginx

If you get an error while running systemctl restart nginx to apply changes, ask Manetta for help

Setting up systemd services

$EDITOR might not work, if that is the case, just replace it with nano or your preferred text editor

Finally, out of root:

sudo reboot now

Additional info

The git repo of Mastodon is in /home/mastodon/live
To remove the "dubious ownership", we ran git config --global --add safe.directory /home/mastodon/live/.git

We wanted to make sure that Mastodon services are well running:

cd /etc/systemd/system
systemctl status mastodon-web

We saw that Mastodon is not running, so we went into the log file:

journalctl | grep mastodon-web

With that, we discovered that there is a problem around the ruby installation so we went back to the Mastodon installation guide, to the ruby installation part and re-ran the commands one by one. But that didn't solve it... What we did to fix that is actually remove the .rbenv files and start again from the ruby installation.

Configuring environment

The guide

>:((((

The date is 06-06-2025. We are just now learning that LOCAL_DOMAIN and WEB_DOMAIN are supposed to be written very carefully for they cannot be changed safely. The thing is, we did change them (recklessly) and it is probably what is causing the "After logging in to the admin account" errors :)))

Troubleshooting

mastodon-web is still not running: the xvm VPN IP address is blocked so we followed these steps:

cd mastodon/live
nano .env.production

at the end of the file, add the following lines:

RAILS_LOG_LEVEL=debug
LOG_LEVEL=silly

in the beginning:

LOCAL_DOMAIN=404.xpub.nl
WEB_DOMAIN=404.xpub.nl
systemctl restart mastodon*

Error logs from /var/log/nginx/:

2025/06/02 16:40:42 [error] 19943#19943: *1355 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.1, server: 404.xpub.nl, request: "GET /favicon.ico HTTP/1.0", upstream: "http://127.0.0.1:3000/favicon.ico", host: "10.0.0.19", referrer: "https://404.xpub.nl/"
2025/06/02 16:40:42 [error] 19943#19943: *1355 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.1, server: 404.xpub.nl, request: "GET /favicon.ico HTTP/1.0", upstream: "http://127.0.0.1:3000/500.html", host: "10.0.0.19", referrer: "https://404.xpub.nl/"

Error logs from journalctl -u mastodon-web.service -f:

Jun 02 16:46:46 debby bundle[22928]: E, [2025-06-02T16:46:46.036606 #22928] ERROR -- : [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked hosts: 10.0.0.19

How we fixed the issue

Following this reddit post, we added the ip address 10.0.0.19 to "allow hosts" in the ruby config file, which we found with this command:

find . -iname production.rb

and then opened:

 nano /home/mastodon/live/config/environments/production.rb

This caused another error: it started to redirect http://10.0.0.19 to https://10.0.0.19.

To solve this... we figured out that we need to LIE to mastodon... Michael found this great blog post that explained it: https://blog.vyvojari.dev/mastodon-behind-a-reverse-proxy-without-ssl-https/

We changed the nginx config:

nano /etc/nginx/sites-enabled/mastodon

and changed this line in location @proxy:

#proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Proto https; # the lie is right here (!!!! we love it, sometimes you need to lie)

This resolved the redirect error!!

Then, we restarted nginx and mastodon... and:

NEXT ERROR

An error popped up in the journalctl:

journalctl -u mastodon-web -f

Which listed the error about the assets package not being able to download.

So we went back to the step that failed in the interactive setup wizard... and ran:

RAILS_ENV=production bin/rails assets:precompile

which gave the same errors again...

Michael suggests to run yarn install again, which was successful.

After that, we ran the command above again (assets:precompile)... (which seems to work now!!)

woohoo, different error!!! an elephant-mashing-the-keyboard-error (500), but this is good news, the server is responding.

Since now the errors were pointing at some missing assets, Michael had a hunch about the yarn install part and found this post where someone suggested to remove the yarn.lock file.

Afterwards yarn install threw an error, but we simply re ran RAILS_ENV=production bin/rails assets:precompile and then everything installed correctly.

Afterwards we opened the journal in follow mode to check what happens when someone connects to the landing page, and everything worked fine with no errors.

To avoid having a cluttered output, we commented out the debug lines we added previously at the bottom of the /live/.env.production file

After logging in to the admin account

On June 6th, we logged in to the admin account, but we started encountering various errors:

• "Connection time out" when trying to access the Settings page, because the url redirects to the IP instead of staying on the domain.
• The cookies error appears when we try to update stuff (such as the profile picture)

• Some other errors:

• 
• 

We then looked at the journal of mastodon-streaming

journalctl -u mastodon-streaming@4000 -f

and discovered a big amount of errors regarding the head of the url, which returns two things, 10.0.0.19 and 10.0.0.1, not 404.xpub.nl.

It then tries to get to the api section, failing in loading it because the head is wrong.

We continued reading the next section of the installation (Configuring your environment), after the section we just cleared (install from source), and found out a nasty message.

Pretty dark. We did modify both local domain and web domain for sure, because when we had issues with "blocked hosts" we found a reddit post that was talking about these two variables in the live/.env.production file. We tried getting the rails console working to check ENV[WEB_DOMAIN], but we got an error saying that RAILS_ENV is not set so the console does not open.

Notes Manetta

6 June, 15:00

Diving into this error...

journalctl -u mastodon-streaming@4000 -f says:

Jun 06 14:27:46 debby node[52026]: {"level":"error","time":1749212866199,"pid":52026,"hostname":"debby","name":"streaming","req":{"id":"874c2d6d-4e89-41cc-ae80-e239872e1e27","method":"GET","url":"/api/v1/streaming/?","query":{},"params":{},"headers":{"host":"10.0.0.19","x-real-ip":"10.0.0.1","x-forwarded-for":"10.0.0.1","x-forwarded-proto":"http","connection":"close","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:139.0) Gecko/20100101 Firefox/139.0","accept":"*/*","accept-language":"en-GB,en;q=0.5","accept-encoding":"gzip, deflate, br, zstd","sec-websocket-version":"13","origin":"https://404.xpub.nl","sec-websocket-protocol":"[Redacted]","sec-websocket-extensions":"permessage-deflate","sec-websocket-key":"[Redacted]","sec-gpc":"1","cookie":"[Redacted]","sec-fetch-dest":"empty","sec-fetch-mode":"websocket","sec-fetch-site":"same-origin","pragma":"no-cache","cache-control":"no-cache"}},"err":{"type":"RequestError","message":"Unknown channel requested","stack":"RequestError: Unknown channel requested\n    at authenticationMiddleware (file:///home/mastodon/live/streaming/index.js:540:12)\n    at Layer.handle [as handle_request] (/home/mastodon/live/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/home/mastodon/live/node_modules/express/lib/router/index.js:328:13)\n    at /home/mastodon/live/node_modules/express/lib/router/index.js:286:9\n    at Function.process_params (/home/mastodon/live/node_modules/express/lib/router/index.js:346:12)\n    at next (/home/mastodon/live/node_modules/express/lib/router/index.js:280:10)\n    at Function.handle (/home/mastodon/live/node_modules/express/lib/router/index.js:175:3)\n    at router (/home/mastodon/live/node_modules/express/lib/router/index.js:47:12)\n    at Layer.handle [as handle_request] (/home/mastodon/live/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/home/mastodon/live/node_modules/express/lib/router/index.js:328:13)","name":"RequestError","status":400},"msg":"RequestError: Unknown channel requested"}

So... RequestError: Unknown channel requested

Rewinding for a second... where are all the config files again??

• nano /home/mastodon/live/.env.production: rbenv config, log levels can be set here
• nano /home/mastodon/live/config/environments/production.rb: ruby production config
• nano /etc/nginx/sites-enabled/mastodon: nginx config

And the logs:

• journalctl -u mastodon-web -f
• journalctl -u mastodon-streaming -f
• journalctl -u mastodon-streaming@4000 -f
• tail -f /var/log/nginx/access.log
• tail -f /var/log/nginx/error.log
• nano /home/mastodon/live/log/ - but this folder stays empty for some reason...

And the tmp admin password is: 4d8f2b592bcf974140cd85d59ad78a56

Oke, what is the situation?

Let's restart the mastodon services!

systemctl restart mastodon*

Hmm, actually, after this restart, there are no errors.

I will now try to login as admin.

Oke, i get this:

• on the screen: "Security verification failed. Are you blocking cookies?"
• nginx access.log: 10.0.0.1 - - [06/Jun/2025:15:41:02 +0200] "POST /auth/sign_in HTTP/1.0" 422 1528 "https://404.xpub.nl/auth/sign_in" "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0", a 422 error
• journalctl mastodon-streaming@4000: nothing
• journalctl mastodon-web: no error, but these DEBUG + INFO logs, referring as well to error code 422

Jun 06 15:41:02 debby bundle[59371]: I, [2025-06-06T15:41:02.746978 #59371]  INFO -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] method=POST path=/auth/sign_in format=html controller=Auth::SessionsController action=create status=422 allocations=40742 duration=24.58 view=19.38 db=1.07

And.. this:

Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.719433 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache read: cache:blocked_ips ({expires_in: 10 minutes, namespace: "cache", compress: true, compress_threshold: 1024})
Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.719740 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache fetch_hit: blocked_ips ({expires_in: 10 minutes, namespace: "cache", compress: true, compress_threshold: 1024})
Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.719928 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache read: cache:rack::attack:5830724:throttle_login_attempts/ip:10.0.0.1 ({expires_in: 10 minutes, namespace: "cache", compress: true, compress_threshold: 1024, raw: true})
Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.720028 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache increment: cache:rack::attack:5830724:throttle_login_attempts/ip:10.0.0.1 ({amount: 1})
Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.721507 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache read: cache:rack::attack:485893:throttle_login_attempts/email:admin ({expires_in: 10 minutes, namespace: "cache", compress: true, compress_threshold: 1024, raw: true})
Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.721669 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache increment: cache:rack::attack:485893:throttle_login_attempts/email:admin ({amount: 1})

which makes me wonder if 10.0.0.1 gets blocked by the streaming part of Mastodon... and thus, if the HUB proxy_pass settings are again the issue..

Do we need to "allow hosts" for the streaming service as well perhaps?? Like we did with the mastodon-web service??

(btw.. the "Unknown channel requested" error does not pop up anymore..)

Added ALLOWED_PRIVATE_ADDRESSES=10.0.0.19,10.0.0.1 to /home/mastodon/live/.env.production... found it in the official docs: https://docs.joinmastodon.org/admin/config/#allowed_private_addresses, but it seems to not do anything.. so i disabled it again.

Added ALTERNATE_DOMAINS=10.0.0.1 to nano /home/mastodon/live/config/environments/production.rb... but no.

Added the same https nginx lie trick as we did with the mastodon-web part...: proxy_set_header X-Forwarded-Proto https; # another lie !!! in the nginx config... but no (i kept it there btw.. just good to keep in mind)

Hmmz... so the question remains: how to get around the "Security verification failed. Are you blocking cookies?" error...

Tried one more thing: to visit https://404.xpub.nl/web after getting the error above, and this redirected me (after a long wait) to https://10.0.0.19 ........ (???)

Trying one more thing: does postgresql work?? Following this: https://www.sql-easy.com/learn/how-to-login-to-postgresql/:

# switch to the postgres user
su - postgres

# start psql
psql

# list all databases
\l

# connect to the database: mastodon_production
\c mastodon_production

# show all tables in this database
\dt

# run this to have a better output of the info below
# enabling the expanded view
\x

# read all values in one table
TABLE table_name;
TABLE ip_blocks;

Postgresql seems to work oke, but hmm, i did not find any useful info here...

More notes of Manetta after a night of sleep

7 June 11:30

I reverted the 2nd nginx https lie in the @streaming block (to stay with the original example of the blog post Michael found)...

I found this: https://docs.joinmastodon.org/admin/config/#trusted_proxy_ip.. IF the reverse proxy is still THE issue... it should fix it, but our ip (10.0.0.1) is already in the default settings, so it did not do anything.. and this error still shows up in the mastodon-web logs: Jun 07 11:45:54 debby bundle[69620]: D, [2025-06-07T11:45:54.923732 #69620] DEBUG -- : [2658241c-85ca-408f-8228-a4847c77be74] Cache increment: cache:rack::attack:5830965:throttle_login_attempts/ip:10.0.0.1 ({amount: 1})

To do still

• no SSH password login: https://docs.joinmastodon.org/admin/prerequisites/#do-not-allow-password-based-ssh-login-keys-only

Add Glitch-Soc to the Mastodon installation

From this installation guide