Fediverse/Installation documentation: Difference between revisions

From XPUB & Lens-Based wiki
No edit summary
 
(21 intermediate revisions by 3 users not shown)
Line 1: Line 1:
''part of the [[Fediverse]] explorations and installation of the XPUB Mastodon instance (April-June 2025)''
''part of the [[Fediverse]] explorations and installation of the XPUB Mastodon instance (April-June 2025)''
[[File:76391d8a44b12a8eea46af9be686eae2.jpg|thumb]]
==config files + logs==
where are all the config files??
* <code>nano /home/mastodon/live/.env.production</code>: rbenv config, log levels can be set here
* <code>nano /home/mastodon/live/config/environments/production.rb</code>: ruby production config
* <code>nano /etc/nginx/sites-enabled/mastodon</code>: nginx config
And the logs?
* <code>journalctl -u mastodon-web -f</code>
* <code>journalctl -u mastodon-streaming -f</code>
* <code>journalctl -u mastodon-streaming@4000 -f</code>
* <code>tail -f /var/log/nginx/access.log</code>
* <code>tail -f /var/log/nginx/error.log</code>
* <code>nano /home/mastodon/live/log/</code> - but this folder stays empty for some reason...


== Install Mastodon as a software ==
== Install Mastodon as a software ==
Line 526: Line 544:
but still not into the mastodon account :(
but still not into the mastodon account :(


==== 422 CRLF error ====
==== 422 CSRF error ====
Trying to solve the 422 CRLF error by adding <code>skip_before_filter :verify_authenticity_token</code> in <code>home/mastodon/live/config/environments/production.rb</code> ...which doesn't work and actually breaks everything :)))
* Adding <code>skip_before_filter :verify_authenticity_token</code> in <code>home/mastodon/live/config/environments/production.rb</code> ...which doesn't work and actually breaks everything :)))
 
* Trying to run the command on its own
RAILS_ENV=production bin/rails skip_before_filter :verify_authenticity_token
Unrecognized command "skip_before_filter :verify_authenticity_token"
Also not working...
 
* Looking for forgery (there's a security issue in there but we're not really sure what it exactly does... see [https://www.reddit.com/r/rails/comments/hzm7ic/is_skip_before_action_verify_authenticity_token_a/ that link for more info])
for FILE in $(find .); do echo $FILE && cat $FILE | grep forgery ; done
 
in <code>config/environments/test.rb</code>, get the line:
config.action_controller.allow_forgery_protection = false
paste it in <code>config/environments/production.rb</code>
 
<p style="font-size: 30px; font-weight: bold;">IT WORKS!!!!!!!!</p>


===To do still, later===
===To do still, later===
Line 536: Line 568:
From this [https://glitch-soc.github.io/docs/ installation guide]
From this [https://glitch-soc.github.io/docs/ installation guide]


<!-- If you encounter the <code>HEAD detached</code> error, you can fix it with:
su
  git branch #to check where you are -->
su - mastodon
cd live
 
We already added the remote git repo, so we catch up from the next steps
 
git checkout glitch-soc/main
 
which sends back an error because of the changes we made during the troubleshootings...
 
So we try copying the config files out of the git branch:
exit
cd ../mastodon/live
cp config/environments/production.rb ../configs_tmp
cp .env.production ../configs_tmp
git add .
git stash
git checkout glitch-soc/main
 
If you encounter the <code>detached HEAD</code> error, you can fix it with:
  git branch #to check where you are
git status
 
We solved this by using this command..
 
First we are moving back to the main mastodon branch:
 
git checkout main
 
then switched to glitch-soc/main:
 
git switch -c glitch --track glitch-soc/main
 
and did git pull:
 
git pull
 
Then we continued the installation, following the steps [https://glitch-soc.github.io/docs/ here]..
 
== troubleshooting glitch soc ==
we removed glitch soc by moving back to the mastodon main branch. </br>
on mastodon production.rb we changed config.force_ssl = true to  <code>config.force_ssl = false</code> (additional to setting FORCE_HTTPS=false, which manetta did before already, following this link https://github.com/mastodon/mastodon/issues/21139) <br>
'''mastodon itself is now running and its possible to log in'''<br>
''note: when generating an invite link it still appears with the ip address instead of the domain name... it is however possible to just replace the ip with the url in the generated link before sending it ''
 
we are not yet able to create new users, because we dont have an e-mail server configured. we are using the e-mail settings from gittea and the server from multiplace
 
these are the e-mail settings:
Protocol: smtp+starttls
Enable: HELO
SMTP Addr: mail.multiplace.org
SMTP Port: 587
User: reports@xpub.nl
we edit these in .env.production
 
'''!! the first users are in online !!'''
 
now trying to move on the glitch-soc branch again (following the steps in the tutorial)
 
after git pull on the glithc branch we make sure that we have the edited lines in our config/environments/production.rb file
config.force_ssl = false
config.hosts = [
  "10.0.0.19",
  "10.0.0.1"
]
 
config.action_controller.allow_forgery_protection = false
'''make sure to be in the right user or root and folder before running each of the steps of the tutorial'''<br>
switch to mastodon user by:
su - mastodon
then go to <code>/home/mastodon/live</code> to run the ruby / RAILS related commands <br>
leave mastodon user <code>exit</code>


== Troubleshooting ==
==Troubleshooting==


=== Key log files ===
===Key log files===


nginx:
nginx:
Line 551: Line 654:
  journalctl -u mastodon-web -f
  journalctl -u mastodon-web -f


== Links ==
==Links==
* https://blog.vyvojari.dev/mastodon-behind-a-reverse-proxy-without-ssl-https/
*https://blog.vyvojari.dev/mastodon-behind-a-reverse-proxy-without-ssl-https/
* https://stackoverflow.com/questions/3364492/actioncontrollerinvalidauthenticitytoken
*https://stackoverflow.com/questions/3364492/actioncontrollerinvalidauthenticitytoken

Latest revision as of 13:06, 25 September 2025

part of the Fediverse explorations and installation of the XPUB Mastodon instance (April-June 2025)

76391d8a44b12a8eea46af9be686eae2.jpg

config files + logs

where are all the config files??

  • nano /home/mastodon/live/.env.production: rbenv config, log levels can be set here
  • nano /home/mastodon/live/config/environments/production.rb: ruby production config
  • nano /etc/nginx/sites-enabled/mastodon: nginx config

And the logs?

  • journalctl -u mastodon-web -f
  • journalctl -u mastodon-streaming -f
  • journalctl -u mastodon-streaming@4000 -f
  • tail -f /var/log/nginx/access.log
  • tail -f /var/log/nginx/error.log
  • nano /home/mastodon/live/log/ - but this folder stays empty for some reason...

Install Mastodon as a software

Following this Installation guide
This guide assumes a bunch of things are already installed, but if you don't have them on your system you will have to install them. We provide here additional guidance because we don't have anything installed. In order to use the installation commands from the guide, we need to log into the root with su and then run the commands

System packages > Yarn

After enabling corepack, the documentation doesn't say it but you should install yarn by just writing yarn in the terminal

Creating the mastodon user

adduser doesn't work? here's how to resolve it:
Unhide the adduser command using the following command:

sudo update-alternatives --install /bin/adduser adduser /usr/sbin/adduser 20

Then run again

adduser --disabled-password mastodon

Setting up Mastodon

To login into the mastodon account, use:

su mastodon

Installing Ruby

All the commands should be ran one by one!!! - make sure to be in the mastodon user as indicated by the tutorial
To resolve the problem about installing rbenv with jemalloc, we do:

exit # out of the mastodon user to go back to root
apt-get install libjemalloc-dev
su - mastodon
RUBY_CONFIGURE_OPTS='--with jemalloc' rbenv install 3.4.4
rbenv global 3.4.4

Installing the last dependencies

Install bundler by running (in root)

apt install bundler
apt install gem

Then run the installation commands (in mastodon user):

bundle config deployment 'true'
bundle config without 'deployment test'
bundle install -j$(getconf _NPROCESSORS_ONLN)
yarn install

When this returns a "no such file or directory" error:

RAILS_ENV=production rails mastodon:setup

Navigate to the live folder and run:

bundle install
RAILS_ENV=production bin/rails mastodon:setup
Success-fediverse.png




SETTING UP PostGreSQL host: /var/run/postgresql
PostGreSQL port: 5432
Name of PostGreSQL database: mastodon_production
Name of PostGreSQL user: mastodon
Password for PostGreSQL user:
Redis host: localhost
Redis ports: 6379
Redis password: [empty]
Store on the cloud? No
Send e-mails from localhost? No
SMTP server: smtp.mailgun.org (to reconfigure later, leave the following SMTP prompts empty by hitting enter)
Do you want Mastodon to periodically check for updates and notify you? Yes
Save configuration? Yes
Prepare database now? Yes
Compile the assets now? Yes (we will go back to that later cause it logged errors)
Create admin right away? Yes
Username: admin
E-mail: m.berends@hr.nl
Temp password:

Switch back to root!

systemctl restart mastodon*

Acquiring an SSL certificate

Use the example.com commands, they will be editable later, but you should already have a (sub)domain name registered.
If encountering "An unexpected error" when trying to generate certificate, run

apt install python3-certbot-nginx

Setting up nginx

If you get an error while running systemctl restart nginx to apply changes, ask Manetta for help

Setting up systemd services

$EDITOR might not work, if that is the case, just replace it with nano or your preferred text editor

Finally, out of root:

sudo reboot now
Additional info

The git repo of Mastodon is in /home/mastodon/live
To remove the "dubious ownership", we ran git config --global --add safe.directory /home/mastodon/live/.git

We wanted to make sure that Mastodon services are well running:

cd /etc/systemd/system
systemctl status mastodon-web

We saw that Mastodon is not running, so we went into the log file:

journalctl | grep mastodon-web

With that, we discovered that there is a problem around the ruby installation so we went back to the Mastodon installation guide, to the ruby installation part and re-ran the commands one by one. But that didn't solve it... What we did to fix that is actually remove the .rbenv files and start again from the ruby installation.

Configuring environment

The guide

>:((((

The date is 06-06-2025. We are just now learning that LOCAL_DOMAIN and WEB_DOMAIN are supposed to be written very carefully for they cannot be changed safely. The thing is, we did change them (recklessly) and it is probably what is causing the "After logging in to the admin account" errors :)))









Troubleshooting

mastodon-web is still not running: the xvm VPN IP address is blocked so we followed these steps:

cd mastodon/live
nano .env.production

at the end of the file, add the following lines:

RAILS_LOG_LEVEL=debug
LOG_LEVEL=silly

in the beginning:

LOCAL_DOMAIN=404.xpub.nl
WEB_DOMAIN=404.xpub.nl
systemctl restart mastodon*

Error logs from /var/log/nginx/:

2025/06/02 16:40:42 [error] 19943#19943: *1355 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.1, server: 404.xpub.nl, request: "GET /favicon.ico HTTP/1.0", upstream: "http://127.0.0.1:3000/favicon.ico", host: "10.0.0.19", referrer: "https://404.xpub.nl/"
2025/06/02 16:40:42 [error] 19943#19943: *1355 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.1, server: 404.xpub.nl, request: "GET /favicon.ico HTTP/1.0", upstream: "http://127.0.0.1:3000/500.html", host: "10.0.0.19", referrer: "https://404.xpub.nl/"

Error logs from journalctl -u mastodon-web.service -f:

Jun 02 16:46:46 debby bundle[22928]: E, [2025-06-02T16:46:46.036606 #22928] ERROR -- : [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked hosts: 10.0.0.19
How we fixed the issue

Following this reddit post, we added the ip address 10.0.0.19 to "allow hosts" in the ruby config file, which we found with this command:

find . -iname production.rb

and then opened:

 nano /home/mastodon/live/config/environments/production.rb

This caused another error: it started to redirect http://10.0.0.19 to https://10.0.0.19.

To solve this... we figured out that we need to LIE to mastodon... Michael found this great blog post that explained it: https://blog.vyvojari.dev/mastodon-behind-a-reverse-proxy-without-ssl-https/

We changed the nginx config:

nano /etc/nginx/sites-enabled/mastodon

and changed this line in location @proxy:

#proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Proto https; # the lie is right here (!!!! we love it, sometimes you need to lie)

This resolved the redirect error!!

Then, we restarted nginx and mastodon... and:

NEXT ERROR

An error popped up in the journalctl:

journalctl -u mastodon-web -f

Which listed the error about the assets package not being able to download.

So we went back to the step that failed in the interactive setup wizard... and ran:

RAILS_ENV=production bin/rails assets:precompile

which gave the same errors again...

Michael suggests to run yarn install again, which was successful.

After that, we ran the command above again (assets:precompile)... (which seems to work now!!)

woohoo, different error!!! an elephant-mashing-the-keyboard-error (500), but this is good news, the server is responding.

a screenshot of an error message during the mastodon instance installation

Since now the errors were pointing at some missing assets, Michael had a hunch about the yarn install part and found this post where someone suggested to remove the yarn.lock file.

Afterwards yarn install threw an error, but we simply re ran RAILS_ENV=production bin/rails assets:precompile and then everything installed correctly.


Afterwards we opened the journal in follow mode to check what happens when someone connects to the landing page, and everything worked fine with no errors.

To avoid having a cluttered output, we commented out the debug lines we added previously at the bottom of the /live/.env.production file

After logging in to the admin account

On June 6th, we logged in to the admin account, but we started encountering various errors:

  • "Connection time out" when trying to access the Settings page, because the url redirects to the IP instead of staying on the domain.
  • The cookies error appears when we try to update stuff (such as the profile picture)

width:300px

  • Some other errors:

We then looked at the journal of mastodon-streaming

journalctl -u mastodon-streaming@4000 -f

and discovered a big amount of errors regarding the head of the url, which returns two things, 10.0.0.19 and 10.0.0.1, not 404.xpub.nl.

It then tries to get to the api section, failing in loading it because the head is wrong.

Error-node-header-url.png

We continued reading the next section of the installation (Configuring your environment), after the section we just cleared (install from source), and found out a nasty message.

Very-bad-documentation.png

Pretty dark. We did modify both local domain and web domain for sure, because when we had issues with "blocked hosts" we found a reddit post that was talking about these two variables in the live/.env.production file. We tried getting the rails console working to check ENV[WEB_DOMAIN], but we got an error saying that RAILS_ENV is not set so the console does not open.

Notes Manetta

6 June, 15:00

Diving into this error...

journalctl -u mastodon-streaming@4000 -f says:

Jun 06 14:27:46 debby node[52026]: {"level":"error","time":1749212866199,"pid":52026,"hostname":"debby","name":"streaming","req":{"id":"874c2d6d-4e89-41cc-ae80-e239872e1e27","method":"GET","url":"/api/v1/streaming/?","query":{},"params":{},"headers":{"host":"10.0.0.19","x-real-ip":"10.0.0.1","x-forwarded-for":"10.0.0.1","x-forwarded-proto":"http","connection":"close","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:139.0) Gecko/20100101 Firefox/139.0","accept":"*/*","accept-language":"en-GB,en;q=0.5","accept-encoding":"gzip, deflate, br, zstd","sec-websocket-version":"13","origin":"https://404.xpub.nl","sec-websocket-protocol":"[Redacted]","sec-websocket-extensions":"permessage-deflate","sec-websocket-key":"[Redacted]","sec-gpc":"1","cookie":"[Redacted]","sec-fetch-dest":"empty","sec-fetch-mode":"websocket","sec-fetch-site":"same-origin","pragma":"no-cache","cache-control":"no-cache"}},"err":{"type":"RequestError","message":"Unknown channel requested","stack":"RequestError: Unknown channel requested\n    at authenticationMiddleware (file:///home/mastodon/live/streaming/index.js:540:12)\n    at Layer.handle [as handle_request] (/home/mastodon/live/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/home/mastodon/live/node_modules/express/lib/router/index.js:328:13)\n    at /home/mastodon/live/node_modules/express/lib/router/index.js:286:9\n    at Function.process_params (/home/mastodon/live/node_modules/express/lib/router/index.js:346:12)\n    at next (/home/mastodon/live/node_modules/express/lib/router/index.js:280:10)\n    at Function.handle (/home/mastodon/live/node_modules/express/lib/router/index.js:175:3)\n    at router (/home/mastodon/live/node_modules/express/lib/router/index.js:47:12)\n    at Layer.handle [as handle_request] (/home/mastodon/live/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/home/mastodon/live/node_modules/express/lib/router/index.js:328:13)","name":"RequestError","status":400},"msg":"RequestError: Unknown channel requested"}

So... RequestError: Unknown channel requested

Rewinding for a second... where are all the config files again??

  • nano /home/mastodon/live/.env.production: rbenv config, log levels can be set here
  • nano /home/mastodon/live/config/environments/production.rb: ruby production config
  • nano /etc/nginx/sites-enabled/mastodon: nginx config

And the logs:

  • journalctl -u mastodon-web -f
  • journalctl -u mastodon-streaming -f
  • journalctl -u mastodon-streaming@4000 -f
  • tail -f /var/log/nginx/access.log
  • tail -f /var/log/nginx/error.log
  • nano /home/mastodon/live/log/ - but this folder stays empty for some reason...

And the tmp admin password is: 4d8f2b592bcf974140cd85d59ad78a56

Oke, what is the situation?

Let's restart the mastodon services!

systemctl restart mastodon*

Hmm, actually, after this restart, there are no errors.

I will now try to login as admin.

Oke, i get this:

  • on the screen: "Security verification failed. Are you blocking cookies?"
  • nginx access.log: 10.0.0.1 - - [06/Jun/2025:15:41:02 +0200] "POST /auth/sign_in HTTP/1.0" 422 1528 "https://404.xpub.nl/auth/sign_in" "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0", a 422 error
  • journalctl mastodon-streaming@4000: nothing
  • journalctl mastodon-web: no error, but these DEBUG + INFO logs, referring as well to error code 422
Jun 06 15:41:02 debby bundle[59371]: I, [2025-06-06T15:41:02.746978 #59371]  INFO -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] method=POST path=/auth/sign_in format=html controller=Auth::SessionsController action=create status=422 allocations=40742 duration=24.58 view=19.38 db=1.07

And.. this:

Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.719433 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache read: cache:blocked_ips ({expires_in: 10 minutes, namespace: "cache", compress: true, compress_threshold: 1024})
Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.719740 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache fetch_hit: blocked_ips ({expires_in: 10 minutes, namespace: "cache", compress: true, compress_threshold: 1024})
Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.719928 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache read: cache:rack::attack:5830724:throttle_login_attempts/ip:10.0.0.1 ({expires_in: 10 minutes, namespace: "cache", compress: true, compress_threshold: 1024, raw: true})
Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.720028 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache increment: cache:rack::attack:5830724:throttle_login_attempts/ip:10.0.0.1 ({amount: 1})
Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.721507 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache read: cache:rack::attack:485893:throttle_login_attempts/email:admin ({expires_in: 10 minutes, namespace: "cache", compress: true, compress_threshold: 1024, raw: true})
Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.721669 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache increment: cache:rack::attack:485893:throttle_login_attempts/email:admin ({amount: 1})

which makes me wonder if 10.0.0.1 gets blocked by the streaming part of Mastodon... and thus, if the HUB proxy_pass settings are again the issue..

Do we need to "allow hosts" for the streaming service as well perhaps?? Like we did with the mastodon-web service??

(btw.. the "Unknown channel requested" error does not pop up anymore..)

Added ALLOWED_PRIVATE_ADDRESSES=10.0.0.19,10.0.0.1 to /home/mastodon/live/.env.production... found it in the official docs: https://docs.joinmastodon.org/admin/config/#allowed_private_addresses, but it seems to not do anything.. so i disabled it again.

Added ALTERNATE_DOMAINS=10.0.0.1 to nano /home/mastodon/live/config/environments/production.rb... but no.

Added the same https nginx lie trick as we did with the mastodon-web part...: proxy_set_header X-Forwarded-Proto https; # another lie !!! in the nginx config... but no (i kept it there btw.. just good to keep in mind)

Hmmz... so the question remains: how to get around the "Security verification failed. Are you blocking cookies?" error...

Tried one more thing: to visit https://404.xpub.nl/web after getting the error above, and this redirected me (after a long wait) to https://10.0.0.19 ........ (???)

Trying one more thing: does postgresql work?? Following this: https://www.sql-easy.com/learn/how-to-login-to-postgresql/:

# switch to the postgres user
su - postgres

# start psql
psql

# list all databases
\l

# connect to the database: mastodon_production
\c mastodon_production

# show all tables in this database
\dt

# run this to have a better output of the info below
# enabling the expanded view
\x

# read all values in one table
TABLE table_name;
TABLE ip_blocks;

Postgresql seems to work oke, but hmm, i did not find any useful info here...

More notes of Manetta after a night of sleep

7 June 11:30

I reverted the 2nd nginx https lie in the @streaming block (to stay with the original example of the blog post Michael found)...

I found this: https://docs.joinmastodon.org/admin/config/#trusted_proxy_ip.. and added the ip 10.0.0.1... IF the reverse proxy is still THE issue... it should fix it, but our ip (10.0.0.1) is already in the default settings, so it did not do anything.. and this error still shows up in the mastodon-web logs:

Jun 07 11:45:54 debby bundle[69620]: D, [2025-06-07T11:45:54.923732 #69620] DEBUG -- : [2658241c-85ca-408f-8228-a4847c77be74] Cache increment: cache:rack::attack:5830965:throttle_login_attempts/ip:10.0.0.1 ({amount: 1})

Hmm... taking a step back... this is our situation:

┌────────────────────────────┐
│    https://404.xpub.nl     │
└────────────────────────────┘
  │
  │ 194.61.65.6
  │ HTTPS
  ▼
┌────────────────────────────┐
│            XVM             │
└────────────────────────────┘
  │
  │ proxy_pass 10.0.0.19
  │ HTTP
  ▼
┌────────────────────────────┐
│           debby            │
└────────────────────────────┘
  │
  │ location / { @proxy }
  │ HTTP
  ▼
┌────────────────────────────┐
│   Mastodon_reverse_proxy   │
└────────────────────────────┘
  │
  │ proxy_pass 127.0.0.1:3000
  │ proxy_pass 127.0.0.1:4000
  │ HTTP
  │ but with HTTPS lie
  ▼
┌────────────────────────────┐
│    Mastodon_backend_app    │
└────────────────────────────┘

Found this: https://github.com/mastodon/mastodon/issues/21139, which is a way to run Mastodon fully over HTTP (so without lying). So, i disabled our lie, and followed the two changes made in the issue. And, actually, the mastodon app runs oke! So it seems that this way is equally solving our reverse proxy SSL issue. Nice. But, the login issue is still there...

However.. if i try to log in directly on http://10.0.0.19 using the enabled HTTP connection (which i can, because my laptop is part of the HUB), i do get a different error: "Invalid E-mail address or password." Hmmm!

So i generated a new password for admin: ec94e41375995744b3712a0df3a47110... with:

su - mastodon
cd live
RAILS_ENV=production bin/tootctl accounts modify admin --reset-password

This did not fix it.

But then......... i realised... that the login screen says: EMAIL ADDRESS, not USERNAME........ !!!!

and i so i used: m.berends@hr.nl + the new password, and.......... IT WORKED!!!!!!!!

GOSH.............. so stupid!!

Trying to log in on https... it still gives the same error. But if you then refresh: it works! PFF!!!

Hmm, maybe i cheered too quickly... some of the admin setting pages still trigger this error... hmpfz...

Current status

  • inspector: 422 error
Content-Security-Policy: The page’s settings blocked an inline script (script-src-elem) from being executed because it violates the following directive: “script-src 'self' https://404.xpub.nl 'wasm-unsafe-eval'”
  • nginx: 422 error
10.0.0.1 - - [09/Jun/2025:22:41:54 +0200] "POST /invites HTTP/1.0" 422 1528 "https://404.xpub.nl/invites" "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"
  • mastodon-streaming@4000: RequestError, Unknown channel requested
Jun 09 22:26:59 debby node[72801]: {"level":"error","time":1749500819242,"pid":72801,"hostname":"debby","name":"streaming","req":{"id":"2c36e313-d8c3-42ce-a554-d9f99264ff7b","method":"GET","url":"/api/v1/streaming/?","query":{},"params":{},"headers":{"host":"10.0.0.19","x-real-ip":"10.0.0.1","x-forwarded-for":"10.0.0.1","x-forwarded-proto":"http","connection":"close","user-agent":"Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br, zstd","sec-websocket-version":"13","origin":"https://404.xpub.nl","sec-websocket-protocol":"[Redacted]","sec-websocket-extensions":"permessage-deflate","sec-websocket-key":"[Redacted]","cookie":"[Redacted]","sec-fetch-dest":"empty","sec-fetch-mode":"websocket","sec-fetch-site":"same-origin","pragma":"no-cache","cache-control":"no-cache"}},"err":{"type":"RequestError","message":"Unknown channel requested","stack":"RequestError: Unknown channel requested\n    at authenticationMiddleware (file:///home/mastodon/live/streaming/index.js:540:12)\n    at Layer.handle [as handle_request] (/home/mastodon/live/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/home/mastodon/live/node_modules/express/lib/router/index.js:328:13)\n    at /home/mastodon/live/node_modules/express/lib/router/index.js:286:9\n    at Function.process_params (/home/mastodon/live/node_modules/express/lib/router/index.js:346:12)\n    at next (/home/mastodon/live/node_modules/express/lib/router/index.js:280:10)\n    at Function.handle (/home/mastodon/live/node_modules/express/lib/router/index.js:175:3)\n    at router (/home/mastodon/live/node_modules/express/lib/router/index.js:47:12)\n    at Layer.handle [as handle_request] (/home/mastodon/live/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/home/mastodon/live/node_modules/express/lib/router/index.js:328:13)","name":"RequestError","status":400},"msg":"RequestError: Unknown channel requested"}
  • mastodon-web: nothing
  • postgresql: nothing

Next steps:

  • check if database works correctly.. the error seems to be triggered on a POST request only...
  • check which git branch is currently active
  • reinstall mastodon?

Switching to Mastodon main

We were still on the glitch-soc/main branch... this might trigger the issues actually!

So let's go back to the mastodon main branch:

git checkout main
error: Your local changes to the following files would be overwritten by checkout: config/environments/production.rb

let's copy the changed config files outside the git first:

mkdir /home/mastodon/configs_tmp/
cd /home/mastodon/live/
cp config/environments/production.rb ../configs_tmp/
cp .env.production ../configs_tmp/

restore (undo) changes in the glitch-soc/main branch:

git restore config/environments/production.rb
git restore yarn.lock

switching back to mastodon main:

git checkout main

Then copy the configs back that we changed:

cp ../configs_tmp/production.rb config/environments/production.rb 
cp ../configs_tmp/.env.production .

and restart mastodon....

Error: bundler: failed to load command: puma

Maybe repeat the rm yarn.lock step...

su - mastodon
cd /home/mastodon/live
rm yarn.lock
cd ..
rm yarn.lock 
cd live
yarn install
bundle install
RAILS_ENV=production bin/rails assets:precompile

Restarting mastodon...

Mastodon starts.. but the error is still there :(

Working with a new database

  • new db name: mastodon_production_two
  • new admin passwd: bf5b279ccf5b74feb87f5daed5e132f4
  • copying back the configs from the old .env.production file

Same errors...

  • checking if it's possible to log in to psql with the user mastodon and password:
psql -h localhost -d mastodon_production_two -U mastodon -p 5432

Hmm, this is maybe the issue then:

psql: error: connection to server at "localhost" (::1), port 5432 failed: FATAL:  password authentication failed for user "mastodon" connection to server at "localhost" (::1), port 5432 failed: FATAL:  password authentication failed for user "mastodon"

And:

psql -h /var/run/postgresql -d mastodon_production_two -U mastodon -p 5432

gives:

psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  Peer authentication failed for user "mastodon"

I changed the password of the mastodon user, but this did not help.

Troubleshooting 11-06-2025

Ah-shit-here-we-go-again.gif

Database

Today, we tried logging into the database.

To do so, we figured out that some access thing in the config file needed to be changed (see here.

Once we made that change, we managed to log into the database of postgresql, but still not into the mastodon account :(


Today, we tried logging into the database with the user mastodon... but this did give an error (see above).

To solve it, we figured out that some access thing in the config file needed to be changed (see here.

nano /etc/postgresql/17/main/pg_hba.conf

Once we made that change, we managed to log into the database of postgresql, with:

psql -h /var/run/postgresql -d mastodon_production_two -U mastodon -p 5432

but still not into the mastodon account :(

422 CSRF error

  • Adding skip_before_filter :verify_authenticity_token in home/mastodon/live/config/environments/production.rb ...which doesn't work and actually breaks everything :)))
  • Trying to run the command on its own
RAILS_ENV=production bin/rails skip_before_filter :verify_authenticity_token
Unrecognized command "skip_before_filter :verify_authenticity_token"

Also not working...

  • Looking for forgery (there's a security issue in there but we're not really sure what it exactly does... see that link for more info)
for FILE in $(find .); do echo $FILE && cat $FILE | grep forgery ; done

in config/environments/test.rb, get the line:

config.action_controller.allow_forgery_protection = false

paste it in config/environments/production.rb

IT WORKS!!!!!!!!

To do still, later

Add Glitch-Soc to the Mastodon installation

From this installation guide

su
su - mastodon
cd live

We already added the remote git repo, so we catch up from the next steps

git checkout glitch-soc/main

which sends back an error because of the changes we made during the troubleshootings...

So we try copying the config files out of the git branch:

exit
cd ../mastodon/live
cp config/environments/production.rb ../configs_tmp
cp .env.production ../configs_tmp
git add .
git stash
git checkout glitch-soc/main

If you encounter the detached HEAD error, you can fix it with:

git branch #to check where you are
git status

We solved this by using this command..

First we are moving back to the main mastodon branch:

git checkout main

then switched to glitch-soc/main:

git switch -c glitch --track glitch-soc/main

and did git pull:

git pull

Then we continued the installation, following the steps here..

troubleshooting glitch soc

we removed glitch soc by moving back to the mastodon main branch.
on mastodon production.rb we changed config.force_ssl = true to config.force_ssl = false (additional to setting FORCE_HTTPS=false, which manetta did before already, following this link https://github.com/mastodon/mastodon/issues/21139)
mastodon itself is now running and its possible to log in
note: when generating an invite link it still appears with the ip address instead of the domain name... it is however possible to just replace the ip with the url in the generated link before sending it

we are not yet able to create new users, because we dont have an e-mail server configured. we are using the e-mail settings from gittea and the server from multiplace

these are the e-mail settings:

Protocol: smtp+starttls
Enable: HELO
SMTP Addr: mail.multiplace.org
SMTP Port: 587
User: reports@xpub.nl

we edit these in .env.production

!! the first users are in online !!

now trying to move on the glitch-soc branch again (following the steps in the tutorial)

after git pull on the glithc branch we make sure that we have the edited lines in our config/environments/production.rb file

config.force_ssl = false

config.hosts = [
 "10.0.0.19",
 "10.0.0.1"
]
config.action_controller.allow_forgery_protection = false

make sure to be in the right user or root and folder before running each of the steps of the tutorial
switch to mastodon user by:

su - mastodon

then go to /home/mastodon/live to run the ruby / RAILS related commands
leave mastodon user exit

Troubleshooting

Key log files

nginx:

tail -f /var/log/nginx/error.log

mastodon-web

journalctl -u mastodon-web -f

Links