Ghost Debugging 1: Overview

This is the first of a series of pages about debugging a Ghost publishing platform running on virtual private server.

There are six posts in the series:

Rough notes are found at Notion Ghost Email Bug .

Description of issue

In Ghost 5.47.1, when user tries to sign up, confirmation email is not sent because of wrong sender. Ghost is not honoring the from field in the mail configuration object found in /var/www/ghost/configuration.development.json

Error when attempting to sign up on self hosted 5.47.1

Ghost reports SMTP rejection because of wrong sender. Inspection shows noreply@notes.nodeholder.com but config.production.json has correct address (info@nodeholder.com).

Getting oriented

Ghost runs in either production or development mode. First, login to your server, this assumes you are using a SSH key in server:/root/.ssh/authorized_keys.

local> ssh root@$ghost_droplet_ip
ghost> su ghost-mgr
ghost> cd /var/www/ghost
ghost:/var/www/ghost> ls

config.development.json  # <-- used when starting ghost from cli
config.production.json   # <-- used by systemctl
current                  # <-- symbolic link to 5.47.1
versions/5.47.1          # <-- code that runs in dev and production

Config /var/www/ghost

config.development.json - used when starting ghost from cli
config.production.json - used by systemctl
current - symbolic link to 5.47.1
versions/5.47.1 - code that runs in dev and production

Production

The production mode is configured by config.production.json and is managed by a systemctl configuration found in /lib/systemd/system/ghost_notes-nodeholder-com.service.

[Unit]
Description=Ghost systemd service for blog: notes-nodeholder-com
Documentation=https://ghost.org/docs/

[Service]
Type=simple
WorkingDirectory=/var/www/ghost
User=996
Environment="NODE_ENV=production"
ExecStart=/usr/bin/node /usr/bin/ghost run
Restart=always

[Install]
WantedBy=multi-user.target

/lib/systemd/system/ghost_notes-nodeholder-com.service

Configuration information is found in the working directory, called config.production.json found in /var/www/ghost.

{
  "url": "https://notes.nodeholder.com",
  "server": {
    "port": 2368,
    "host": "127.0.0.1"
  },
  "database": {
    "client": "mysql",
    "connection": {
      "host": "localhost",
      "user": "root",
      "password": "xxxxxxxxxxxxxxxxxxxxxxx",
      "database": "ghost_prod"
    }
  },
  "mail": {
    "from": "'NH Info' <info@nodeholder.com>",
    "transport": "SMTP",
    "options": {
      "host": "xxx.smtp.xxxxxxx.com",
      "port": 587,
      "auth": {
        "user": "info@nodeholder.com",
        "pass": "xxxxxxxxx"
      }
    }
  },
  "logging": {
    "transports": [
      "file",
      "stdout"
    ]
  },
  "process": "systemd",
  "paths": {
    "contentPath": "/var/www/ghost/content"
  }
}

config.production.json with SMTP configuration

Let's first confirm we are running in production using ghost ls:

Ghost cli interacts with system during production

Alternatively, using systemctl status:

root@ghost-sfo2-01:~# systemctl status ghost_notes-nodeholder-com.service

● ghost_notes-nodeholder-com.service - Ghost systemd service for blog: notes-no>
Loaded: loaded (/lib/systemd/system/ghost_notes-nodeholder-com.service; en>
Active: active (running) since Fri 2023-05-19 18:19:28 PDT; 1 weeks 5 days>
Docs: https://ghost.org/docs/
Main PID: 1856131 (ghost run)
Tasks: 22 (limit: 1130)
Memory: 10.6M
CGroup: /system.slice/ghost_notes-nodeholder-com.service
├─1856131 ghost run
└─1856171 /usr/bin/node current/index.js

Systemctl status for ghost service

Capture production logging via journalctl:

ghost-mgr@ghost> journalctl -f -u ghost_notes-nodeholder-com

Systemctl processes write to the systemd journal

The man page for journalctl which prints log entries from systemd journal, abbreviated from original output found in Notion task, but essentially:

[03:06:26] ERROR: Email sending failed - all recipients were rejected.
[03:06:26] DETAIL: 553 5.7.1 <noreply@notes.nodeholder.com> rejected.
[03:06:26] CAUSE: Sender address not owned by user info@nodeholder.com.
[03:06:26] INFO: Visit https://ghost.org/docs/config/#mail for email config info.
[03:06:26] ERROR ID: 442f8810-1e40-11ee-aa87-ad8c802d6d4b
[03:06:26] ERROR CODE: EENVELOPE
[03:06:26] STACK: Error at createMailError in GhostMailer.js line 70.
[03:06:26] REQUEST: "POST /members/api/send-magic-link/" returned 400, took 901ms.

Ghost log after failed signup attempt

Analysis

Sending email failed because the sender address noreply@ notes.nodeholder.com was not owned by the user info@nodeholder.com, leading to a rejection of all recipients. The error, identified by code EENVELOPE and ID 442f8810-1e40-11ee-aa87-ad8c802d6d4b, occurred during a "POST /members/api/send-magic-link/" request, which returned a 400 status code.

The mail server reports that recipient was rejected because the sending agent was not owned by info@nodeholder.com. This is because it received noreply@notes.nodeholder.com.

We see that this is response to an HTTP request POST /members/api/send-magic-link/ which was handled by nodemailer/lib/smtp-connection/index.js.

See Ghost Debugging 2: Remote development setup for configuring a remote debugging session so we can take a closer look.