Ghost Debugging 1: Overview

This is the first of a series of pages about debugging a Ghost publishing platform running on virtual private server.

There are six posts in the series:

  1. Overview (this page)
  2. Remote Development Setup
  3. Stepping Through Code
  4. SMTP Email Bug
  5. Capturing Network Packets
  6. Dealing with SMTP

Rough notes are found at Notion Ghost Email Bug .

Description of issue

In Ghost 5.47.1, when user tries to sign up, confirmation email is not sent because of wrong sender. Ghost is not honoring the from field in the mail configuration object found in /var/www/ghost/configuration.development.json

Error when attempting to sign up on self hosted 5.47.1

Ghost reports SMTP rejection because of wrong sender. Inspection shows [email protected] but config.production.json has correct address ([email protected]).

Getting oriented

Login to Ghost server

Ghost runs in either production or development mode. First, login to your server, this assumes you are using a SSH key in server:/root/.ssh/authorized_keys.

local> ssh root@$ghost_droplet_ip
ghost> su ghost-mgr
ghost> cd /var/www/ghost
ghost:/var/www/ghost> ls

config.development.json  # <-- used when starting ghost from cli
config.production.json   # <-- used by systemctl
current                  # <-- symbolic link to 5.47.1
versions/5.47.1          # <-- code that runs in dev and production
Login procedure to work as user ghost-mgr

Config /var/www/ghost

  1. config.development.json  - used when starting ghost from cli
  2. config.production.json -  used by systemctl
  3. current   -  symbolic link to 5.47.1
  4. versions/5.47.1 - code that runs in dev and production

Production

The production mode is configured by config.production.json and is managed by a systemctl configuration found in /lib/systemd/system/ghost_notes-nodeholder-com.service.

[Unit]
Description=Ghost systemd service for blog: notes-nodeholder-com
Documentation=https://ghost.org/docs/

[Service]
Type=simple
WorkingDirectory=/var/www/ghost
User=996
Environment="NODE_ENV=production"
ExecStart=/usr/bin/node /usr/bin/ghost run
Restart=always

[Install]
WantedBy=multi-user.target
/lib/systemd/system/ghost_notes-nodeholder-com.service

Configuration information is found in the working directory, called config.production.json found in /var/www/ghost.

{
  "url": "https://notes.nodeholder.com",
  "server": {
    "port": 2368,
    "host": "127.0.0.1"
  },
  "database": {
    "client": "mysql",
    "connection": {
      "host": "localhost",
      "user": "root",
      "password": "xxxxxxxxxxxxxxxxxxxxxxx",
      "database": "ghost_prod"
    }
  },
  "mail": {
    "from": "'NH Info' <[email protected]>",
    "transport": "SMTP",
    "options": {
      "host": "xxx.smtp.xxxxxxx.com",
      "port": 587,
      "auth": {
        "user": "[email protected]",
        "pass": "xxxxxxxxx"
      }
    }
  },
  "logging": {
    "transports": [
      "file",
      "stdout"
    ]
  },
  "process": "systemd",
  "paths": {
    "contentPath": "/var/www/ghost/content"
  }
}
config.production.json with SMTP configuration

Let's first confirm we are running in production using ghost ls:

Ghost cli interacts with system during production

Alternatively, using systemctl status:

root@ghost-sfo2-01:~# systemctl status ghost_notes-nodeholder-com.service

● ghost_notes-nodeholder-com.service - Ghost systemd service for blog: notes-no>
Loaded: loaded (/lib/systemd/system/ghost_notes-nodeholder-com.service; en>
Active: active (running) since Fri 2023-05-19 18:19:28 PDT; 1 weeks 5 days>
Docs: https://ghost.org/docs/
Main PID: 1856131 (ghost run)
Tasks: 22 (limit: 1130)
Memory: 10.6M
CGroup: /system.slice/ghost_notes-nodeholder-com.service
├─1856131 ghost run
└─1856171 /usr/bin/node current/index.js
Systemctl status for ghost service

Capture  production logging via journalctl:

ghost-mgr@ghost> journalctl -f -u ghost_notes-nodeholder-com
Systemctl processes write to the systemd journal

The man page for journalctl which prints log entries from systemd journal, abbreviated from original output found in Notion task, but essentially:

[03:06:26] ERROR: Email sending failed - all recipients were rejected.
[03:06:26] DETAIL: 553 5.7.1 <[email protected]> rejected.
[03:06:26] CAUSE: Sender address not owned by user [email protected].
[03:06:26] INFO: Visit https://ghost.org/docs/config/#mail for email config info.
[03:06:26] ERROR ID: 442f8810-1e40-11ee-aa87-ad8c802d6d4b
[03:06:26] ERROR CODE: EENVELOPE
[03:06:26] STACK: Error at createMailError in GhostMailer.js line 70.
[03:06:26] REQUEST: "POST /members/api/send-magic-link/" returned 400, took 901ms.
Ghost log after failed signup attempt

Analysis

Sending email failed because the sender address noreply@ notes.nodeholder.com was not owned by the user [email protected], leading to a rejection of all recipients. The error, identified by code EENVELOPE and ID 442f8810-1e40-11ee-aa87-ad8c802d6d4b, occurred during a "POST /members/api/send-magic-link/" request, which returned a 400 status code.

The mail server reports that recipient was rejected because the sending agent was not owned by [email protected]. This is because  it received [email protected].

We see that this is response to an HTTP request POST /members/api/send-magic-link/ which was handled by nodemailer/lib/smtp-connection/index.js.

See Ghost Debugging 2: Remote development setup for configuring a remote debugging session so we can take a closer look.