Published Jul 26. 2016 - 7 years ago
Updated or edited Oct 8. 2020

The new newsletter

I reinstated the GFF Newsletter that we stopped sending a decade ago. This article is about the system driving the new newsletter.

Nerd alert

This is NOT about fly fishing or fly tying, but about site development and nerdy stuff.
Full newsletter in the archive
Full newsletter in the archive
Martin Joergensen

Sending a newsletter isn't always a trivial thing – although it can be. You have to make some decisions about how to send it and what to send, how to maintain a subscribers list and how or even if to have an archive of the newsletters that have been sent.
The simplest newsletter is a simple manually written mail sent from a mail program to a number of recipients. Pure text, identical mails, no automation of any kind.

I wanted automation

The newsletter I wanted, had to be almost fully automated and made from site content, built on the fly from different articles, images, lists and more and compiled into one HTML document where a number of individualized elements appear as tokens and are replaced at the time of sending the single mail.
I have been setting up such email newsletter systems professionally together with the people I work with many times before, and we have done so using different approaches:
- An internal system generating the mail and an external service for maintaining subscriber lists and sending the mails.
- An internal system handling mail generation and subscribers, an external system handling the mailing process.
- An internal system doing all parts of the process.
The choice usually depends on the number of newsletters, demands on speed and what services the CMS and the web server offers.

A challenge

Technically creating and sending many simultaneous mails can be a challenge. Typically there are several places where bottlenecks can appear, but the single hardest part is usually the actual sending of the mails, which can be thousands in number and can put quite a strain on a normal web server. That is why we have sometimes chosen an external service to send the mails.
We have also set up special mail handlers on the servers, able to receive a single mail template and information on a number of subscribers, and then very efficiently merging the two and sending off thousands of mails in minutes, which has been critical for some clients.

Simplenews

The Global FlyFisher newsletter has sufficiently few subscribers to not have such problems, a couple of thousand maybe. And having the mails sent over a longer period is also OK, relieving the server from sending many mails in a short time.
I can let Drupal and PHP send the mails, and because of that, the module Simplenews is the one of choice for the task. There are other newsletter systems for Drupal, but Simplenews is where I have most experience. Simplenews is also the most commonly used newsletter module in the Drupal community. It's slightly kludgy and fairly large and complex, sometimes cumbersome to work with and configure, but it's widely supported and documented with many help resources.
We have used Simplenews for several projects – both as a stand alone system, in conjunction with an external service and working with some home made modules.

The parts

The GFF installation consists of:
- A home made module than can create a mail from site content and a template and save it as a Simplenews node. It uses blocks and views as well as custom made node renderings to create an HTML mail.
- Simplenews to handle all subscription activity, create the individual newsletters, send them and maintain an archive.
- The Drupal modules Mail System, HTML Mail, and SMTP Authentication Support to expand on PHP's and Drupal's mail handling.
- The server, which actually sends the mails. In this case an external mail server since the virtual server that runs GFF has no mail facilities.
- A few home made extras including a tracking system for counting the opening rate of mails and a “link catapult” that counts the links clicked in the mails – AKA CTR, Click Through Rate.

The gff_newsletter module

The custom made module that handles the creation of the newsletters is called gff_newsletter.module, and consists of a handful of functions, which include gathering site content for the newsletter, merging it with a template file and saving it as a Simplenews newsletter node, which Simplenews can then send.
Building an HTML newsletter that looks nice in all mail clients isn't an exact science. While most browsers have come to understand HTML and CSS almost the same, mail systems such as GMail, HotMail, Thunderbird, Microsoft Outlook, Eudora and whatever mail programs or systems people are using, don't quite agree on the same common terms.
So HTML mails are usually built in a very clumsy way using tables for layout and embedded styles, not relying on modern HTML with divs, floats and central CSS.
In my module this HTML is mainly found in a template file (*.tpl.php) with a few tokens, which are replaced with the content HTML generated by Drupal itself in nodes, blocks and views.

Mail system

Since my server doesn't have any mail facilities, I use an external mail server, in my case Google's, to handle the sending, and for this I need the SMTP Authentication Support module, which connects the Drupal mail system to an external server. In order to send HTML mail you need the HTML Mail module, which again relies on the Mail System API module.
Setting this up to work with Simplenew requires a little dexterity, but is doable.
SMTP support is configured with a server, user name and password. You can use GMail if you have an account. Use smtp.gmail.com, port 465 and your whatever@gmail.com user name and the password for the account.
Mail System must be configured to work with Simplenews mail system interface in combo with SMTP, called HTMLMailSystem__SmtpMailSystem in the selection options on /admin/config/system/mailsystem
Use this both site wide and for HTML mail.

Handling URL's, images and more

When Drupal generates its content, all links and references are local, and have no server reference or protocol. So a link to the front page is simply ”/” and a link to an article in the patterns section might be ”/patterns/red-tag”. Likewise with images. The logo is simply ”/sites/all/themes/gff/logo.png”. For this to work in a newsletter I need them to be absolute, like ”http://globalflyfisher.com/patterns/red-tag” and ”http://globalflyfisher.com/sites/all/themes/gff/logo.png”. That will make them work in a mail system on a remote machine.
These local links are replaced using some clever regular expressions looking for “img src” and “a href” codes. There are modules, which can do that, but since I at the same time replace URL's with links to the link counter rather than directly to the page, I need my own. The latter enables me to count the clicks on newsletter links, a technique used by most newsletters and by Facebook as well as Google to count clicks.

Sending a newsletter

When I make a newsletter, I fire up the function in my module, which gathers the needed content, combines it with the template, saves it as a Simplenews node and sends me on to view that node. This gives me a rough preview of the newsletter, but since it's shown in Drupal, it inherits all the CSS from my Drupal theme, which influences the look both in a good and a bad way.
I have created a small function that opens the newsletter in a ”clean” popup window, showing the newsletter essentially as it will be seen in a mail client. I can edit the HTML, but rarely do so.
When I'm satisfied, I can send test mails to myself or other guinea pigs, or I can send the newsletter to the recipients. The latter won't actually send the newsletter, but mark it as ready for sending and send a number on each cron run.
Simplenews will then make the final token replacements with recipient information, unsubscribe links and more, and make sure a controllable number of mails are sent off each time cron is called until all mails have been sent.

In Eudora
In GMail
In the mail systems
Martin Joergensen
.

Log in or register to pre-fill name on comments, add videos, user pictures and more.
Read more about why you should register.
 

Since you got this far …


The GFF money box

… I have a small favor to ask.

Long story short

Support the Global FlyFisher through several different channels, including PayPal.

Long story longer

The Global FlyFisher has been online since the mid-90's and has been free to access for everybody since day one – and will stay free for as long as I run it.
But that doesn't mean that it's free to run.
It costs money to drive a large site like this.

See more details about what you can do to help in this blog post.