https://citationneeded.news/substack-to-self-hosted-ghost/

[citation needed]
a newsletter by Molly White

  * Weekly recaps
  * Podcast feed
  * Archive
  * RSS
  * About
  * Donate

Sign in Subscribe

Migrating from Substack to self-hosted Ghost: the details

I migrated Citation Needed from Substack to self-hosted Ghost. Here
is exactly how I did that.

Molly White

Molly White

Jan 13, 2024 -- 33 min read
Migrating from Substack to self-hosted Ghost: the details

Some have asked me how I feel about Substack's recent decision to ban
five (5) no-name Nazi newsletters and then say "see? we're doing what
you asked! And you screeching leftists are making a big fuss over
nothing!" I gave a quote to the Washington Post which I will just
repeat here: "It's honestly insulting, both to writers and readers on
the platform, that they think they can shut up those of us who have
serious concerns with such a meager gesture." It's completely
inadequate, particularly coupled with their promises that they won't
be changing their policy, and that they won't be taking any sort of
proactive approach to content moderation going forward.

I had a conversation with Substack co-founder Hamish McKenzie, prior
to him apparently changing his mind about this handful of
newsletters, that left me very skeptical that there will be
meaningful change at Substack with him at the wheel -- even if he is
successfully pressured to make some token gestures. That skepticism
only worsened with his extremely scummy behavior in leaking
conversations with Platformer to what Platformer described as a
"friendlier publication" -- by which they mean "anti-woke" Michael
Shellenberger's The Public -- and in actively soliciting the
supposedly organic anti-anti-Nazi letter^a of support and then
canvassing supportive authors to sign it.

Happily, I've already migrated away from the platform, so I am no
longer among the whole slew of writers who are either still deciding
whether to leave, or who have decided to leave and are just trying to
work out how.

For those of you in the former camp, I would argue that Substack's
long history of horrendous decisionmaking about what kind of content
they will not only tolerate but monetize, boost, and even actively
court is perfectly sufficient reason alone. But should you need a few
more reasons, here are a few benefits that I am personally enjoying
at the moment as a result of pulling the lever attached to my ejector
seat and spiraling off into the sunset.^b

Substack no longer takes 10% of my subscriber payments.
    This is honestly pretty huge. When you're first starting out,
    other newsletter platforms that charge flat fees like $10 or $20
    a month can seem like a daunting up-front investment when you
    don't know if anyone will care enough about what you have to say
    to pay for it. Then there's Substack, where you can sign up for
    free and not have to pay a cent. It's tempting! But as a
    newsletter grows, even somewhat small numbers of subscribers can
    make that 10% start to look pretty big. For example, if you
    manage to sign up 100 subscribers paying $10/month, you're
    already at $100/month going to Substack -- far more than flat-fee
    platforms charge for similarly sized newsletters.
I'm no longer publishing on an a16z-backed platform.
    As a writer who has been very critical of Andreessen Horowitz and
    of the venture capital model more generally, it made me
    uncomfortable to publish on a VC-backed platform whose Series A
    round was led by a16z.
I have full ownership of my writing and subscriber lists.
    Substack likes to brag that writers retain full ownership over
    their writing and membership lists, and they do to a much greater
    extent than some other platforms. But they still keep a lot to
    themselves, making truly packing up and leaving a challenging
    experience (as I will describe in more detail later on). Now, if
    I wanted to, I could dump the MySQL tables and have every single
    scrap of information on my platform saved to my local machine. I
    can run scripts to update or download or do whatever to my
    material as necessary, without worrying I'm falling afoul of
    Substack's policies against running "processes" on the site.
I can accept payments any way I like.
    Substack requires authors to agree that they will only accept
    subscription payments via their platform. As someone who wants to
    make it possible for subscribers to support in whatever way or
    amount feels reasonable to them, this meant I couldn't allow
    people to -- say -- make a one-time contribution in exchange for a
    subscription, or allow them to use a different payment processor.
I can make my newsletter look how I want.
    Separate footnotes and references sections? Coming right up. A
    font outside of Substack's five options? No problem.
I can recommend anyone I want.
    Substack has a cool feature where each Substack author can
    "recommend" any other newsletters they enjoy, via a little pop-up
    that shows to new subscribers and in various other places. The
    catch: those writers also have to be on Substack. It helps with
    Substack's network effects, but it also traps people in a walled
    garden. "Stay or lose Substack's network effects" is a huge
    bargaining chip in their advantage, and I've seen several writers
    (myself included!) reference it as a concern over leaving.

I could probably go on, but you get my gist.

Once I had decided to leave, I had to answer the question of where to
go. There are many alternatives that I've already seen various former
Substack authors choose. Among them are Ghost's "Ghost(Pro)" hosted
option, beehiiv, Buttondown, and so on. All of them look lovely. Many
third parties also sell concierge services to add functionality on
top of these of platforms, or to help with migration.

Pricing-wise, and without any concierge services, I was looking at
$165/month for Ghost Pro, $84/month for beehiiv, and $139/month for
Buttondown for a newsletter my size.

But quickly I realized that what I wanted was not a new platform, but
rather no platform. Although most platforms do a better job of
content moderation than Substack, I feared that I could find myself
in the same position down the road if my new host made a similarly
unconscionable decision. I'm also a little bit of a control freak,
and more generally, I like having as few opportunities as possible
for would-be enshittifiers to mess around with what I'm trying to do.
^c So, self-hosting seemed the obvious route to take, and once that
decision was made, Ghost's open source blogging software was the
obvious choice.^d

The process of migrating, unfortunately, is daunting. Every
newsletter is different, every destination is different, and there is
no unified how-to guide that covered what I was trying to do.

So, for the benefit of anyone out there who might be trying to do
what I did -- that is, migrate from Substack to a self-hosted instance
of the open source Ghost blogging software -- here is my version of
the guide I wish I had.

For those of you who don't care about wonky and fairly technical
details at great length, you may want to stop here. For the rest of
you brave souls, read on:

Initial setup

Server

First things first, I needed a server for Ghost. I've had a VPS with
DigitalOcean for... ten years now? that hosts my mollywhite.net
website and a whole bunch of other bits and bobs of code, but I knew
that this newsletter was going to have higher resource demands, and
didn't want to risk bringing all my other stuff offline if the
newsletter got too much traffic.

So, I spun up a separate droplet specifically for the newsletter. The
other advantage of doing this is that DigitalOcean offers one-click
"Create Ghost Droplet" functionality that does all the setup for you,
including getting the database hooked up and serving the website.

I took a shot in the dark as far as how chunky of a droplet I would
need, and picked the $14/month "Premium AMD" option (2GB RAM, 50GB
storage, 2TB transfer). It's pretty easy to increase droplet size
down the line, so this isn't a decision you really need to sweat too
much.

Once the server is provisioned, you go through a CLI set-up process.
Something went wrong the first time I did this, and MySQL wasn't
configured properly. Rather than try to untangle where it had gotten
stuck along the way, I just destroyed the droplet and started again.
No problems the second time. shrug.

After that, there's some configuration to be done in the Ghost admin
interface. This is all pretty straightforward stuff: setting the name
of the newsletter, uploading the logo, etc.

Domain

Next, I pointed a domain at the site.

I opted to change my newsletter domain from newsletter.mollywhite.net
to citationneeded.news as a part of this migration, for a number of
reasons that I won't go into here. If you are planning to keep the
same custom domain you were using at Substack, you'll want to do this
step at the very end to avoid site downtime as you're getting things
set up. Alternatively, if you don't mind people potentially getting
lost in the interim, you can disconnect your custom domain from
Substack and have your old Substack newsletter live at the default
yourusername.substack.com domain for a little bit. Just make sure you
go into Substack preferences and disconnect the domain there before
updating your DNS -- otherwise you can apparently get stuck in a weird
state where you can't get to your Substack because they think you've
still got a custom domain.

If you didn't already have a custom domain and were using the
Substack one, you'll need to buy a domain -- you can't take Substack's
with you.

I'd already purchased the citationneeded.news domain for about $7
from Namecheap back in November when I renamed the newsletter, so I
had that sitting around just redirecting to my Substack. I removed
that redirect, moved the nameservers to Cloudflare,^e and set up DNS
records in Cloudflare to point at the website (if you scroll down to
the email section, those are entries 1 and 3 in the DNS records
screenshot). I also turned off Cloudflare's caching and DNS proxying
for the time being, because I knew I was going to be actively
developing the site, and caching is a nightmare for that.

Email

Next up was email. In order to send bulk emails from Ghost, you need
to use Mailgun. Mailgun is actually the priciest part of my setup, at
$75/month for their "Foundation 100k" plan, which will allow me to
send 100,000 emails a month. After 100,000, they charge $0.001 per
message. This sounds like a lot of emails, but with more than 20,000
subscribers and somewhere around five newsletters a month, I'm likely
going to have to pay a bit more (but probably not a lot more) than
that flat $75 -- particularly because Mailgun is also used to send the
magic links for sign up and sign in, and the transactional emails to
welcome new subscribers or remind paid subscribers of upcoming
renewals.

Hooking up Mailgun to Ghost is pretty straightforward: you just snag
an API key from Mailgun and plug it into the Ghost admin interface,
et voila.

Less straightforward is doing all the configuration to reduce the
chances of email providers seeing someone suddenly sending 20,000
emails all at once from a previously unused domain and going
"SPAMMER!" From my research, it seems like there's some stuff you can
do up front, and some of it is just a waiting game as these providers
build up a reputation score for your domain/IP address and classify
you from there.

I looked into setting up a dedicated IP for my mailsending, because I
heard that maybe that was the thing to do. The reasoning there is
that with a shared IP, you run the risk of some other Mailgun
customer actually being a spammer, and then your reputation gets
bogged down by theirs. However, after a conversation with Mailgun
support (who are incredibly lovely and responsive, by the way), I
ascertained that this actually might be a bad idea, especially as I'm
just getting started:

    Dedicated IP's require consistent sending over time and if the
    sending isn't consistent it may damage the sending reputation of
    your dedicated IP address, placing the IP on blacklists and
    throttling your messages.

So, shared IP it is (at least for now). With that out of the way, I
set up more DNS records on the citationneeded.news domain. So many.
There's the record to verify with Mailgun that you control the
domain, then there's the SPF record, then there's the DMARC record
and the DKIM record, then there are the MX records, and I think
you're supposed to put a CNAME record in there for some reason too? I
don't know, I just do what I'm told. And someone's just told me about
a BIMI record, which apparently makes emails look trustworthy, so I
threw one of them in there for good measure. Why not.

Learn from my mistakes, though: I failed to set up a) the MX records
pointing at Mailgun, and b) the DMARC record before sending my first
email. This is likely why so many of the first emails bounced.

[Screenshot-2024-01-10-at-6]
[Screenshot-2024-01-10-at-6]

The first newsletter send vs. the second newsletter send. Green and
blue = good! Red = bad.

I had seen an instruction in a Mailgun setup document that said "MX
records should also be added, unless you already have MX records for
your domain pointed at another email service provider (e.g. Gmail)."
I had MX records set up on this domain that point at Cloudflare,
which forwards any emails sent to @citationneeded.news over to my
email, so I thought, "well, I wouldn't want to mess anything up by
having conflicting records." This seems to have been wrong! After a
whole bunch of emails from the first newsletter send bounced with the
error message "Sender address rejected: domain not found", I
discovered that the Mailgun MX record seems to be necessary for email
providers to verify the sender. Or maybe it was because there was no
MX record specifically for the mg. subdomain that Mailgun uses?
Either way, I added it (number 5 below), and stopped seeing those
errors.

[Screenshot_2024-01-12_at_11_10_07-AM-1]

I also added the DMARC entry (#9), which is super important to a)
prevent other people from trying to spoof emails as though they are
coming from citationneeded.news, and b) help email service providers
trust that my emails aren't spoofed and, thus, spammy. That should've
been there from the start, and would've prevented a lot of other
bounces.

Altogether, I added entries 2, 5, 6, and 8-11 to my DNS records. At a
minimum, you will need to add entries 9, 10, 11, and apparently 5 to
get your emails to reliably land where they're supposed to.

On the very optional end of things: Entry #2, the CNAME record, is
used by Mailgun to track things like email opens and clicks. I've
kept it on for now just to help me with debugging email stuff, but
will turn it off soon, because I've always found email open/
clicktracking to be kind of invasive and not super useful to me.
Entry #6, the Google Postmaster one, is not necessary to make the
emails work, but lets me see a little more information about what
Google thinks is spammy through their Postmaster tool.

Some other tools I found useful:

  * DMARCian's record wizard helps you create DMARC records
  * DMARCian's XML to human converter helps you decipher any DMARC
    reports you receive, if you choose to have them come straight to
    you
  * Postmark has a free DMARC monitoring tool to help you triage the
    bombardment of DMARC reports you will eventually receive when you
    realize having them come straight to you is probably a mistake

Entries 4 and 7 were already in place from when I set up email
forwarding through Cloudflare. Tons of different services can do this
for you for free, including some domain registration services like
Namecheap -- I just use Cloudflare because I'm already using it for
DNS and caching. You'll probably want to set something up so that you
can receive emails sent to your newsletter's domain -- otherwise if
people reply to your newsletter email, they'll just be shouting into
the void. Alternatively, you could set your personal email in the
Reply-To header on your newsletter email sends, but having not done
this myself I can't speak to how effective it is in catching all
stray emails.

Transactional emails

One thing Substack does for you out of the box that Ghost does not do
is send transactional emails -- that is, pretty much everything
besides the bulk newsletter sends that go out to everyone all at
once. Think "thank you for subscribing!" or "your subscription is
about to renew!".

Ghost does send a very brief message to new subscribers, but it's
mostly just to double-confirm their subscription, and can't be
customized to add any additional information:

[Screenshot_2024-01-12__4_34-PM]

If you want these transactional emails, and you probably do, you'll
need to set them up yourself. For those using Ghost Pro and not
wanting to write any code, I'm not sure there's any easy solution to
this besides using a service like Zapier, or working with a group
like Outpost, who sell add-ons on top of Ghost's hosted services.

I briefly experimented with using Hookdeck, a Zapier-like project
with a more generous free plan, since all I was really trying to do
was glue a Ghost or Stripe webhook to Mailgun's API, but ended up
being unsuccessful in configuring the webhook authentication. Womp
womp.

So, instead, I coded up a basic Express server, which I stuck
alongside the Ghost instance on my new DigitalOcean droplet. It's
really simple -- it's got one endpoint to listen for the Ghost webhook
event when a new user signs up, and one that listens for Stripe's
invoice.upcoming event (which signifies that a customer is about to
be resubscribed). In the first case, the server makes a call to the
Mailgun API to fire off a simple welcome email that's a little
friendlier and informative than Ghost's automatic one. For upcoming
renewals, I filter to only annual subscriptions (so monthly
subscribers aren't getting this email every month). That email is a
little more complex -- it includes some custom variables to inform
people how much they're about to be charged and when -- but is largely
the same process.

Over in Mailgun, I used their WYSIWYG template editor to create the
templates:

[Screenshot-2024-01-12-at-4]

I may also add an email that lets people know once their subscription
has actually been renewed, since no one likes surprise credit card
charges, but I'm trying to balance being communicative with not
bombarding subscribers' inboxes. Same goes for people whose
subscriptions are about to expire without renewal, to nudge to see if
they might want to resubscribe.

One note: I had originally intended to have two different welcome
emails: one for paid subscribers, and one for free subscribers. This
is what I had over at Substack. I was planning to just listen for the
Ghost member.added event, and then send the appropriate email based
on whether the member object had status: free or status: paid.
Unfortunately, I discovered that when someone signs up for a paid
subscription, they are first added as a free member (triggering the
added event with the free status), and then updated to a paid member
once the Stripe flow is completed. I couldn't think of a great way to
get around this without first sending an erroneous "free" welcome
email to paid subscribers, or making the webhook server stateful so
it could wait a little bit to see if the update came in before
sending the email. So, for now, everyone gets the same email and I
just direct people to a customized welcome page with more detail
after they've completed their signup.

Import

Next up: getting all of my posts and subscribers from Substack into
Ghost. Substack likes to boast about how you own your own content,
and can easily migrate away from Substack if you ever want to.
Unfortunately, their tooling for actually doing so leaves much to be
desired.

You can download an export of all of your Substack posts, and Ghost
has helpfully built in a handy little wizard (in Ghost Settings >
Labs > Beta features > Substack migrator) that will help you import
both your content and your subscribers. However, I would not actually
recommend using it, at least not without doing some extra work first.
That said, any content import or subscriber imports that you do can
be relatively easily deleted and redone, so it's safe to experiment
and then erase your imports later on.

Content import

If you use Substack's content export as-is, you will lose a lot and
your posts will be broken. For example, it does not actually export
any images or other media embedded throughout your posts, so you end
up with an HTML file that hotlinks to a ton of media still hosted on
Substack, which will all 404 as soon as you take your Substack site
offline. Links to other Substack posts will break, "subscribe" and
"share" buttons will all still point to Substack, etc. etc. Not
great.

Fortunately, Ghost has come to the rescue here with their migrate
tool. It's just a bunch of Node scripts, and they are easily
installed with npm or yarn. They've actually got a whole slew of them
if you're trying to migrate to Ghost from a whole bunch of different
platforms, but you'll be wanting the Substack (mg-substack) migration
tool.

I found the documentation to be a little sparse, so to answer some
questions I had: this tool is intended to be run on the ZIP file of
your post content that you've exported from Substack. You don't just
run the script standalone to do the exporting for you, it's meant to
augment the ZIP export with everything it's missing. So, when it asks
for a --pathToZip, that's the path to the Substack export file. The
--url argument is the URL to your old Substack site (not your new
Ghost URL, if they're different), which it will scrape.

It will do things like scrape all of the media assets from your
Substack, prepare them for upload them to your Ghost site, and modify
references throughout your posts to use the new paths. It will also
update links you might have made within your newsletter to your past
posts so they use the new URLs rather than Substack's URLs, and it
can update your "subscribe" buttons and the like to go to the right
place. It can even import drafts, Substack pages, and Substack
threads if you want them.

The tool is quite good as-is, however there were a few other things I
wanted it to do.

For one, it doesn't scrape and reupload audio voiceovers of posts, a
feature I've started using over the past few months. I tweaked the
script so it would (see my Github branch if you need to do the same).
That's probably the only change I made that's likely to be broadly
useful, but I also made the following tweaks:

  * Added a script to detect when I was writing an explanatory
    footnote vs. citing a source, and bucket those into separate
    "footnotes" and "references" sections
  * I'd been using an image (the blue * * *) as a separator for a
    while in Substack because I didn't like their default <hr>
    styling, so I added some functionality to the migration script to
    avoid downloading that one same separator image a bazillion
    times. Instead, I updated the script to properly replace those
    separator images with the <hr> tag, which I've styled how I like
    it with the image using CSS -- much more a11y-friendly!
  * Similarly, I'd been using blockquotes to style the "In the news"
    and "Worth a read" sections of my newsletter. This is not very
    a11y-friendly, since semantically they are not quoted content, so
    I updated the script to replace those instances with some custom
    HTML that looks largely identical but uses more appropriate HTML
    tags.
  * I discovered once I switched to a font with noticeable
    directionality in its curly quotes that a lot of my quotation
    marks were pointing the wrong way. This was the case in Substack
    also, but because of the font over there I never noticed. I
    updated the script to replace all curly quotes with straight
    quotes, because I'll be damned if I'm going to go through and try
    to fix all that. If you want to do this too, just add the
    following to the end of the processContent function in
    mg-substack/process.js:
    html = html.replace(/[""]/g, '"');
    html = html.replace(/['']/g, '\'');

With all this done, I ran the script, imported the content (Ghost
Settings > Labs > Import content), and did some spot checking to make
sure everything worked as designed. I was pretty pleased!

This took me a few tries, so I had to clear out all of the content
and reupload a few times. If you have a lot of posts, I would
recommend making a smaller test version of the import ZIP to use --
otherwise you may have to wait several minutes for each import to
complete. If you did an import and want to clear out all content from
your Ghost blog, you can easily do so via Ghost Settings > Labs >
Delete all content. Note that this does include all posts and pages,
but does not remove member data.

The only thing I missed was that the Ghost migration script
apparently does not detect and update links to past posts when you've
linked to a subsection of that post (which I do frequently). A
subscriber kindly informed me that those links are broken. I will be
writing a script to go through all my posts and fix that issue soon.

User import

So, good news and bad news. The good news is that you can migrate
your Substack subscribers (paid and free) from Substack to another
service. Substack uses Stripe for payment processing, and so as long
as you continue to use Stripe (used by Ghost, Buttondown, beehiiv,
etc.) you can swap out the newsletter platform without any
interruption to them. This is huge -- as someone who migrated from
Patreon way back when and had to ask everyone in my (much smaller)
subscriber base to manually re-subscribe, that's a nightmare.

The bad news: as with content export, Substack does not make user
export all that easy on you, despite their marketing. They do allow
you to export your users in CSV files from your Substack membership
management interface, but I discovered that these CSV files are
formatted slightly differently than the ones Substack spits out if
you use Ghost's Substack migrator tool. Critically, the ones Substack
spits out from its membership interface are missing the Stripe IDs of
paid subscribers, which are crucial to keeping subscriptions linked
as you migrate! But the export from the members interface includes a
lot of additional data I need, too, like the subset of my newsletters
^f a member person has opted to receive. So, I wrote a script to mash
all four exports together (paid member portal export, paid members
export with the Stripe data, free member portal export, free members
export with whatever other data is included from that mysterious
exporter). It also deals with the fact that some of the data from
Substack is just... wrong. For example, the column that purports to
reflect if a member has turned off emails never shows that they have,
even if they've unsubscribed from all of my newsletters.

Once I mashed that all together, I imported the users into Ghost from
the CSV.

If you do this and something's wrong, and you want to start over, it
is possible to bulk delete all Ghost members, even though it's not
immediately apparent. You have to go to your members page and then
"filter" your members list with a criterion that will match everyone
on your list -- you can use Label is "Import [date]", which is added
automatically by the Ghost migration tool -- and then click the gear
icon and choose "Delete selected members".

Everything was mostly good to go at that point, except that for some
reason, although Ghost does support having multiple sub-newsletters
per site, and allows people to subscribe or unsubscribe from them,
you can't import members' subscription preferences. This would mean
that all members would receive both the Citation Needed posts and the
Weekly Recap posts, even if they'd previously unsubscribed from one
or the other. I whacked together a quick script to read the CSV
export from Substack, which contained those preferences, and update
members's subscriptions after the fact. I would share it with you on
Github, except I seem to have lost it. If anyone needs this, let me
know -- I can probably whack it back together again pretty quickly.

At this point, everything worked great membership-wise, except... oh
no.

Fixing Stripe

Every single paid subscriber appeared to have the same subscription
plan, which is just whatever the first tier you add to Ghost happens
to be. As you may know, I've been offering pay-what-you-want
subscriptions for a while, and on Substack I accomplished this by
creating a $10/month or $100/year subscription, and then nine
discount links for 90% off, 80% off, 70% off, and so on. Way back
when, before my pay-what-you-want hack, I'd also done a couple of
one-off discounts, so there were a handful of folks with those, too.
And in the very beginning, the default subscription was $5/month or
$50/year. All of this... does not migrate well.

Although the subscriptions themselves didn't change, all of my paid
subscribers were added to the same Ghost subscriber tier, which would
make it look in their Ghost member settings as though they had the
$10/month or $100/year plan. I knew this would probably spark panic
in some subscribers who had signed up to pay considerably less, and
it's just a terrible experience for subscribers to have to assure
them "no no, even though it looks like I've totally fleeced you, I
haven't, I promise!" I could've edited the tier name and price to
look different, or have a huge disclaimer on it like "LOOK AT THE
ABOUT PAGE WHERE I EXPLAIN WHY THIS NUMBER IS WRONG", but some people
always miss these things, and it was better to just fix it.

So, first I set up my ten subscription tiers (plus the founding
member tier) in Ghost. Then I did my best to whip up a script using
Stripe's API that would go through each of my paid subscribers, check
which tier they should be in (for example, a 40% discount off the $10
/$100 subscription should have the discount removed and be reassigned
to the $6/$60 tier), remove the discount as needed, and update the
subscription without prorating. Altogether, this meant that nothing
changes on the subscriber's end in terms of payment amount,
frequency, or renewal date, but the subscriptions are technically set
up a little bit differently under the hood.

This was all a little terrifying, because changing a subscription in
Stripe runs the real risk that I could screw up and overcharge
someone -- the worst possible outcome. I tested the hell out of my
script, and added a bunch of bailout cases where the script would
skip trying to update a subscription if anything about it was
unusual.

Unfortunately, there are a ton of things that can make subscriptions
unusual. The biggest one I ran into was non-US currencies. You might
think it would be as easy as looking up the current USD conversion
rate for their currency, but this fluctuates over time, so one Brit
who signed up recently for the $10/month subscription might pay PS8/
month, whereas another who signed up longer ago might pay PS9/month.
Ultimately, I care more about people paying what they expect to pay
than I do receiving exactly the USD amount reflected in their tier,
so I went through all non-US currencies to manually migrate them and
reflect the original payment amounts and currencies involved. This
took several hours, and was perhaps the only time I've been grateful
that I don't have more people paying me.

Ghost theme tweaks

One thing I love about Ghost is you can pretty easily tweak how most
things look without also having to dive into the full nuts and bolts
of the Ghost mono-repository. I made some changes to the Ghost theme
-- some cosmetic, and some more to do with the site functionality.

In the cosmetic department, I added the full-width banner on the
homepage with the illustration of me on the laptop, to either
encourage people to subscribe or thank them for doing so. I also
added some additional links to the site footer, removed the "feature
image" from automatically displaying at the top of each post (since I
often include these just for social sharing purposes), and swapped
out the default fonts. Nothing too exciting there (unless you get
excited about fonts like I do, I suppose).

The more substantial changes were to the signup flow, and to the
welcome page.

For the welcome page, I created a custom page in Ghost by modifying
the routes.yaml file (Ghost Settings > Labs > Beta features > Routes)
and creating a welcome.hbs file in the theme. This takes advantage of
Ghost's @member.paid flag to show different content based on whether
the member is paid or free.

The signup page is also custom, with some custom JavaScript, and is
set up with the same routes file. This wasn't strictly necessary, but
with 12 signup options (the ten pay-what-you-want tiers, plus free
and founding members), the automatic page looked pretty overwhelming.

[Screenshot-2024-01-12-at-4]Default sign-up page with all of my tiers
displayed

Instead, people now choose to sign up from one of four groups, and
then can pick their pay-what-you-want tier from a dropdown:

[Screenshot-2024-01-12-at-4]Custom sign-up page

Unfortunately, I will need to dig into the Ghost mono-repo to replace
the "change plan" page, which looks much like the dizzying default
sign-up screenshot above. I hope to do so soon, but didn't want to
delay migrating any longer with something that's mostly cosmetic. The
same seems to be true if you want to make any changes to Ghost's
email styling.

Leaving Substack

With my content migrated over and my subscribers in place, it was
finally time to leave Substack.

First, I needed to ask Substack to disconnect themselves from my
Stripe account. This is critical: if you don't do this, subscriptions
created through Substack will continue to send a 10% cut to Substack,
even after you've left the platform!

I emailed Substack's support team:

    Hello,
    I am migrating my newsletter (https://newsletter.mollywhite.net/)
    off of Substack due to the company's recent refusal to deplatform
    or demonetize Nazi content.
    Can you please disable the Substack's fee on my Stripe account?
    Thank you,
    Molly White

One minute later, on Christmas Eve no less, I received a reply. How
responsive!

    Hi Molly,

    I understand you want to make changes to your newsletter's
    financial setup. However, we cannot disable the fees that are
    automatically taken as part of the transaction process on
    Substack. These fees are applied to support the services provided
    by the platform and are not optional.

    If you're looking to disconnect your Stripe account from
    Substack, which will cancel and refund all paid subscriptions to
    your newsletter, you can do so by following these steps:
    ...

[record scratch]

After a brief moment of panic, my soul returned to my body and I
remembered how widely Substack had advertised that all writers retain
control over their subscriber lists and can take them with them if
they leave.

I responded:

    No, I intend to migrate my paid subscribers from Substack (as is
    a selling point of the platform: "A Substack is the writer's
    property: the email list, content, and payment relationships
    (should you choose to monetize) is the writer's and the writer
    can take all of it with them if they ever decided to leave the
    platform.") I don't wish to cancel their paid subscriptions, I
    just wish to remove the Substack fee as I will no longer be using
    the platform. See: https://ghost.org/docs/migration/substack/#
    removing-substack-fees

I then proceeded to spend five days in terrifying limbo as I awaited
a reply (which was, in fairness, over the holidays).

Finally, a different support representative replied that they had
disconnected my Stripe account from my Substack account and removed
Substack's cut of my subscription fees. Phew.

I would like to assume that the original email was just a newbie
support representative either misunderstanding my request or not
realizing that Substack does indeed allow people to do what I was
asking. However, after seeing Substack's incredibly bad faith over
the last few weeks, a part of me wonders if this is standard
operating procedure when people make these requests, as a way of
discouraging people from leaving the platform. Perhaps that's bad
faith of me to wonder, I don't know. Either way, you should know that
it's possible that you will get this kind of reply, and to just be
persistent if you do.

Whatever you do, do not disconnect Stripe in the way they instructed
in their first reply, because all paying subscribers will be
unsubscribed, [S:and I'm pretty sure there's no undoing that:S].
(Update: I have been informed by someone more knowledgeable than I
about Stripe that this is actually undoable, so you don't need to
worry as much as I originally thought!)

Since migrating, I've learned that there are ways to turn off
Substack's connection to your Stripe account without involving
Substack support. I haven't done it myself, but the folks over at
Buttondown were kind enough to provide the how-to.

Unfortunately, when Substack disconnected my Stripe account, all my
subscribers turned into free subscribers. This is obvious in
hindsight, but I didn't think about it until it happened. Because of
this, I could no longer do the paid subscriber export that I
described above, and so any paid subscribers who signed up between
when I did the first export and when Substack disconnected my account
would be lost in the migration.

Fortunately, there were only a handful of people who signed up for
paid subscriptions during this period, so I was able to just do the
free subscriber export again, pull in the paid subscriber information
from Stripe manually, and run another import to fill in the missing
pieces. I think made my Substack account "private", which prevented
anyone new from accidentally signing up over there.

The one thing I did lose was the list of people who had unsubscribed
shortly before I migrated, many of whom cited Substack's policy
regarding Nazis in their messages. I had wanted to send any of these
folks who had completely removed themselves from my subscriber list
an email to let them know I had moved. I was able to get this
information from Substack after the fact by making another request to
their support team, but it took several days.

So, word of advice: as close as you possibly can to when Substack
disconnects your Stripe account, make sure to export your paid
subscriber list and, if you're going to want it, your unsubscribes
list.

Another thing I hadn't thought about: Substack has a number of emails
they automatically send for you that you may not even know about,
including one that reminds people when their subscriptions are about
to renew. When Substack disconnected my Stripe account, these emails
stopped being sent, because as far as Substack was concerned I didn't
have any paid subscribers. It wasn't until a week or so later, when I
went to hook up transactional emails that I realized oh no, those
emails used to be getting sent, and were no longer. As a result,
there were some people who were resubscribed without getting advance
notice -- something I feel terrible about. I've sent an email to that
group of folks and offered to refund the subscriptions in case this
was not something they wanted.

To avoid falling into this same trap, you can either make sure your
transactional emails are set up before you disconnect Stripe, or
toggle the setting in Stripe (Settings > Subscriptions and emails >
Prevent failed payments) to "Send emails about upcoming renewals".
The email will look a little different, but your subscribers will at
least get the warning!

Redirecting old links

Although I had done my best to update any links to this newsletter
within my own content, I knew there were links to
newsletter.mollywhite.net elsewhere on the web, in readers' bookmark
folders, etc. that would break once I officially moved.

To avoid this, I had to redirect all newsletter.mollywhite.net
traffic to citationneeded.news. I also had to take into account that
Substack inserts a /p/ into its post links, whereas Ghost does not.
For example, https://newsletter.mollywhite.net/p/
substackers-against-nazis becomes https://citationneeded.news/
substackers-against-nazis/.

After some fiddling in my NGINX configuration on my other server (the
one that hosts mollywhite.net), I came up with this server block:

server {
  server_name newsletter.mollywhite.net;
  location ~ ^/p/(?<path>.*)$ {
    return 301 https://citationneeded.news/$path;
  }
  location / {
    return 301 https://citationneeded.news$request_uri;
  }
}

Is it the most elegant way of doing it? I don't know! Does it work?
Sure does.

Regarding RSS: Substack keeps RSS feeds at /feed. Ghost's are at /
rss. However, Ghost automatically redirects /feed to /rss, which
means that anyone who subscribed to newsletter.mollywhite.net/feed
should receive the citationneeded.news/rss feed in their feedreader
with no updates needed on their end.

Last-minute configuration

At this point, everything was just about ready to go. This meant it
was time to prepare the server to actually "go live", and begin
handling some real traffic.

First I re-enabled Cloudflare caching. For now, this is a pretty
standard setup, except that I've added a caching bypass rule for any
pages that match the admin panel route (citationneeded.news/ghost),
which are only used by me and shouldn't be cached. I'll be tweaking
it as I go to help handle traffic spikes a little more smoothly.

I also ran mysql_secure_installation on my Ghost server to harden up
the MySQL server. This does a couple of things to make it harder for
people to compromise your SQL database (which is initially set up in
a development mode), like disabling remote login as root.

Finally, I configured ufw, Ubuntu's firewall tool.

The first email send

Finally, it was time to announce that I had moved.

Then, I penned my announcement post in the Ghost editor (which I
already love), said a small prayer, and hit the big red button:

[Screenshot-2024-01-05-at-11]Okay, it's green, whatever.

I went to my email inbox, refreshed a few times, and... nothing
happened.

Uh oh.

I waited a little longer, checked my social media to see if anyone
had said "hey cool, congrats on moving!" No dice. I braced myself,
opened the Ghost error logs, and saw that the send had failed thanks
to an authentication error with Mailgun. Turns out I'd put in my old
domain name when setting up the Mailgun connection. Dammit.

The good news is, Ghost automatically retries failed sends, so once I
fixed the connection, it automatically tried again and we were off to
the races.

Except, a huge percentage of the emails failed to land in peoples'
inboxes. And either the load from the email send or from the traffic
to the site from people who received the emails caused that dinky
little $14/month server to hang. This was worsened by the fact that I
had shared the announcement to my various social medias, which not
only increased traffic to the server, but also did what's been
variously called a "MastoDDoS" or a "Mastodon stampede". I won't go
into it here, but it's an unfortunate quirk of Mastodon's federated
design.

Nothing like having a reader clicking a link in your email
triumphantly announcing your new website only to see...

[599fadc0d0a80331]

Womp, wooomp.

Since the site was down already, I took the opportunity to beef up
the server a little bit (which requires taking the server offline). I
saw from the server logs that the RAM was maxing out, so I doubled
the memory to 4GB with 2 vCPUs with a $28/month droplet (which is
otherwise identical in terms of storage and transfer).

Once it came back up, it handled the load okay, although I wasn't
sure if this was because the load had diminished. More on that in a
sec.

After this, I saw in my Mailgun logs that emails weren't being
delivered. This was super disappointing -- people weren't receiving
the newsletter, and people were also having trouble receiving their
magic links after signing up or signing in. This turned out to be due
to (I think) a combination of the missing MX records and DMARC
record.

Another thing that can impact deliverability is bogus email addresses
on your list. Apparently if you try to send an email blast but a lot
of the emails are undeliverable because there's no valid inbox at the
other end, it can really harm your reputation. After sending my first
newsletter, I ran my email list through Mailgun's Validator tool to
weed out bad addresses. I was pretty conservative about this, since
it seemed to flag some legitimate emails as high-risk (mostly custom
domains), but I ultimately turned off email sending for around 100 or
so members (out of 21,000, not bad!) where the email had no hope at
all of getting through to someone. I had Substack's double opt-in
turned on for the vast majority of the time I was on that platform,
which probably went a long way to help this. Only three of the
invalid emails were paid members, who I'll be reaching out to -- for
all three of them, there were very similar email addresses in my
member list, so I suspect they just typoed when signing up for their
paid subscription.

Finally, I tweaked some settings so that the newsletter is sent from
[email protected], but one-off emails come from [email protected].
I'm hoping that this means that email providers won't throttle, say,
the sign-in links even if they throttle newsletter emails. It may be
that the throttling applies to the whole domain, but it's worth a
shot (plus it helps people filter things a little better).

Now that I've resolved that handful of issues, things are looking way
better. There are still some temporary failures from email providers
who think I might be a spammer, which I think is mostly a waiting
game as my sending reputation improves. I'm also encountering a few,
but not many, one-off issues that I'm debugging with Mailgun support.

My second attempt at sending email went much better than the first,
and the vast majority of emails went through. I also experimented
with sending the email, waiting a little bit, and then posting the
link to Mastodon after the initial traffic surge went back down.
Unfortunately, the initial traffic surge and the MastoDDoS were both
enough to cause the server to hang for a few minutes each, so I had
some more tweaking to do with caching and so forth. However, between
email sends, the new server is motoring along at about 10% CPU and
50% memory, so hopefully it's sized okay and I can handle the rest
with configuration tweaks.

The last email from Substack

I also sent one last email from Substack to inform subscribers that I
had moved. This was particularly necessary in my case, because a lot
of people had not received the announcement email due to the email
configuration issues I mentioned.

One word of advice: if you do this, and if you have custom email
headers configured in Substack depending on if someone's a free or
paid subscriber, turn those off first!

I forgot to do so, and so everyone got an email saying at the top
that they were a free subscriber, because as far as Substack is
concerned at this point, they are. This caused a few paid subscribers
to (reasonably) believe that their subscription had not migrated
properly.

After this email was sent, I deleted my Substack account. This felt
nice.

The end result

After all of this, I have found myself with a roughly $103/mo setup:
$28/mo for the VPS, $75/mo (plus overage tbd) for Mailgun. This is
considerably cheaper than staying on Substack, and also slightly
cheaper than both hosted Ghost and Buttondown.

However, more important to me than the exact price is the degree of
control I have over my own not-a-platform, where I can guarantee my
newsletter is not sitting on a server alongside a be-swastikaed hate
screed.

Even with a few remaining hiccups to work out, this antifascist
living room is feeling pretty nice.

One final note

I realize that this is... a lot. If you are a newsletter writer
looking to flee the Substack ship, please don't let this discourage
you. I'm already seeing other newsletters like The Sword and the
Sandwich, Today in Tabs, and Disconnect moving to various other
services (Buttondown, beehiiv, and Ghost Pro, respectively) at least
relatively seamlessly, and hopefully with a lot less custom work than
I did.

I chose the somewhat more laborious route of self-hosting, but it is
far from the only route! And if you're not a technical person but you
want to self-host, there are a ton of really capable people out there
who you can hire to help you get something up and running.

Footnotes

 1. There has got to be a shorter way of writing that...

 2. Your mileage may vary on whether or to what extent you too will
    enjoy these benefits depending on where you decide to go -- other
    platforms may charge various fees, have their own content
    moderation policies, or impose other restrictions on your
    activities.

 3. If anyone enshittifies my newsletter, it's going to be me,
    dammit!

 4. In fairness, it is possible to self-host WordPress, which is also
    open-source. It is, however, also written in PHP -- a language I
    am much less comfortable with and enjoy much less than Ghost's
    JavaScript. WordPress is also quite a bit older, and, in my
    opinion, somewhat bloated with features I don't and won't need.

 5. I realize there is some irony in using Cloudflare, given their
    own less-than-stellar reputation when it comes to extremists on
    their platform. I'm not thrilled to use their platform, but I do
    feel a little more comfortable using a free platform where I'm
    costing them money rather than something like Substack, where I
    was actively subsidizing the Nazi substacks with income generated
    from my work. If anyone has any suggestions for Cloudflare
    alternatives that have similar features (particularly with
    respect to caching and DDoS protection), I'm all ears.

 6. I have my Citation Needed newsletter, but I have what I call
    sub-newsletters: "Weekly Recaps" and, previously, the "FTX Files"
    newsletters. Some people subscribe to all of these, but some only
    receive a subset of them.

Social share image is lightly cropped from "Ghost live 2015" by
dr_zoidberg, CC-BY-SA 2.0.

(What, was this not the Ghost you were picturing?)

Loved this post? Consider signing up for a pay-what-you-want
subscription or leaving a tip to support Molly White's work, which is
entirely funded by readers like you.

Read more

The charges against Binance

Video: The charges against Binance

In December, not long after the Department of Justice announced
charges against cryptocurrency giant Binance and its CEO Changpeng
"CZ" Zhao, I started writing up an overview of everything that had
happened to the company over the past few years. I'd written
separately about a lot of it -- the CFTC

Jan 15, 2024
Issue 48 - Bitcoin has "no chance" of going to the moon

Issue 48 - Bitcoin has "no chance" of going to the moon

Bitcoin ETF fakeouts, imaginary CEOs, and a bridge hack make for an
eventful start to the new year.

Jan 9, 2024
A Mexican antifascist propaganda poster depicting a large eagle
shredding a Nazi flag

Citation Needed has a new home

Citation Needed is no longer hosted on Substack. Welcome to my
antifascist bar.

Jan 5, 2024
A grid of mostly Bored Ape NFTs

Issue 47 - Residual garbage

All my "absolute top tier apes" gone, anti-rug-pull rug pulls, and an
update on this newsletter.

Dec 26, 2023

Citation Needed features critical coverage of the cryptocurrency
industry and of issues in the broader technology world.

It is independently published by Molly White, and entirely supported
by readers like you.

Subscribe

  * Weekly recaps
  * Podcast feed
  * Archive
  * RSS
  * About
  * Donate

  * Twitter
  * Mastodon
  * Bluesky
  * YouTube
  * TikTok
  * Etc.

(c) 2024 Molly White.