CMail History

Posted: 14 March 2020

As the idiom says, there's more than way to skin a cat. Internet e-mail is no exception. While users are typically only concerned with the e-mail they send arriving in the intended recipient's Inbox, their software is making decisions on how to best format the message for delivery using a variety of published standards. These days, almost all e-mail will be processed by some sort of filter which will pull apart the message and scan for keywords or malicious content. CMail was originally written to assist with validating such analysis software and therefore was designed to be able to construct and send messages using a range of options and encodings, including some which were considered obsolete at the time.

The range of options available is most apparent with attachments. CMail provides five different settings to attach individual files where most similar tools only provide one or two. Fortunately, for most purposes a single option, -a, will suffice (or -ai for images displayed in HTML messages). CMail similarly provides various settings for encoding message bodies, including transfer encoding, line length and even an option to use MIME when it's not strictly required.

Yet more settings exist to modify the behaviour of the SMTP transport, such as settings requiring TLS, limiting authentication to specific methods and forcing the use of IPv4 or IPv6. As with content encoding, each of these options was intended to exercise specific code in the software being tested and are otherwise of little use.

Fortunately, CMail remains easy to use despite all the options.

Background

In late 2004 I was made redundant from my first development job where I had been maintaining NNTP and SMTP software. On my final day, I was contacted by a relative of my soon-to-be former boss and offered a six-month part-time contract testing e-mail and Web filtering software. He was also being made redundant and was setting up a business to test the software he'd been working on after it has been sold to new offshore owners. In addition to myself, he hired another developer and after a short time, a junior QA person.

Having knowledge of the relevant standards, my role was a technical one, focusing on implementation detail and standards compliance rather than general QA work. I approached all changes with a developer's mindset and tried to work as efficiently as possible. Software testing is boring. I admire people who can run regression tests and repeat the same steps ad infinitum, but it's not something I am able to do. Automation was essential to preserve my sanity.

Validating e-mail software obviously required sending e-mail, and while everyone else in the team may have been content to send offensive statements through the content filter using Outlook Express, I was not. I also had more complex testing requirements, validating the product at protocol level.

I initially wrote a few batch files that used wMailTo to send e-mail more efficiently. wMailTo was developed by Jarle Aase, best known for War FTP Daemon. I soon discovered a trivial bug, the details of which I have forgotten. At the time STARTTLS was being implemented in the product being tested, so rather than simply fixing the bug and attempting to extend it with SSL support, I decided to write my own SMTP mailer. Thus, CMail was born.

Six-months part-time eventually turned in to eight years full-time, and I managed to find myself a niche, building test environments and undertaking general product development, with only the bare minimum of mind-numbing testing. My personal need for CMail largely disappeared, but other developers and QA staff continued to use it and the tool continued to evolve with their needs.

What's in a name?

Many people assume that because I am a big fan of the C programming language, CMail is so named because it is written in C. The truth is more mundane. The main file of CMail was cmail.c, with the C representing Command Line or Console. When the decision was made to release the tool to the world, the name was simply capitalised.

Regarding naming, version 0 is as good as any other. Many companies release version 2.x thinking people won't use a version 1.x product. CMail only receives a version bump when there is a major change, such as adding TLS. Development builds are clearly marked as such.

Design

To avoid any conflict of interest, CMail was developed strictly on my own time, using my own resources. I was regularly asked to add proprietary product features by co-workers who continued using it. While it would have been trivial to reverse engineer many of these features without my internal knowledge of the products, I always declined. The requirement to keep CMail separate resulted in a major design decision that has shaped CMail since the outset. In order to produce messages that could be consumed by the products being tested, CMail needed to both send e-mail and write e-mails to file. Outputting to file allowed CMail to create RFC-compliant messages that could have proprietary extensions added by employer-owned scripts. Essentially, this allowed SMTP to be bypassed, exercising the content analysis engine directly.

int makemessage(struct msginfo *mi, void(*fnout)(void *, char *, ...), void *argout)

To achieve dual output, CMail originally used an open source printf implementation, xprintf.c. This provided a very handy xprintf variant that could output via a callback function and was used to create socket and file outputting variants of printf. Later versions use my own printf alternative, vfstr (and variants ip_fstr/ffstr), but the basic implantation remains unchanged.

When sending via SMTP, output goes directly to the connected socket and when creating a file, to a file buffer. Doing this means CMail can send arbitrarily large attachments without buffering in memory or using a temporary file. Given CMail originally needed to send large quantities of e-mail, potentially with multiple instances sending mail in parallel or from the same computers QA members were working on, limiting resource use was an important goal.

While beneficial for testing, this design decision did not come without downsides. CMail cannot guarantee the file will be available at the time the message is sent. CMail was modified to check files existed before attempting to send, and the -skipnofile setting added requesting CMail continue regardless, which is handy when attaching files that may or may not exist. If the file cannot be read during the message creation however, CMail is forced to skip it as SMTP does not have a mechanism to handle this, other than simply terminating the connection. This may seem like odd behaviour, but it's the direct consequence of the original requirement to ensure CMail remained free and unencumbered and to bypass SMTP in the products being tested.

Another drawback of this design is lack of flexibility with logging. CMail relies on the IP code to log data in and out of the socket, but a callback with a standard interface doesn't provide a simple mechanism to intelligently switch it on and off. It could be extended of course, but there hasn't been demand for it. As a result, debug output always contains the full message body, even though it may be desirable to turn logging off when outputting attachments. Until recently, CMail also logged the full SMTP session, including passwords. This did concern users and this issue has been addressed by extending vfstr and adding an ip_fstr variant that accepts a bitmask, replacing any parameter, regardless of type, with repeated asterisks of equivalent length. This was possible because the SMTP session is handled independently from message creation.

CMail assumes that mail delivery should always be attempted. It's likely mail is being sent by scripts where errors may not be seen by a human. If authentication credentials are supplied but either not requested or invalid, CMail will still attempt delivery. The assumption that mail should always be delivered has recently been enhanced, with CMail now automatically using STARTTLS if credentials are provided but authentication is not offered prior to establishing a secure connection. This reflects a change in how SMTP services are deployed, with widespread adoption of the submission port (587) and authentication only being offered after STARTTLS.

CMail follows the Unix philosophy of do one thing and do it well. There isn't much more to add about design because other than some quirks resulting from its original use, CMail is nothing more than a tool designed to do one job. CMail sends an e-mail. Decisions on how or if to handle failure are left up to the user or calling script, which is why the most requested feature of Blat, retrying, isn't a feature in CMail.

Ch-ch-changes

Over the years many changes have been made to CMail. These have largely been driven by user requests. Support for HTML message bodies (as attachments), messages bodies from file (previously only possible via a pipe), and comments within configuration files have all resulted from user requests. The first new 'mode' added to CMail since the initial design, sending and editing pre-created e-mails, also came about as a result of a user request. If there's a feature you'd like to see in CMail, please get in touch.

There are, however, some features I regret adding, at least as they were implemented.

CMail 0.7.x and earlier supported yEnc encoding. This was an encoding method that was popular on newsgroups in the early 2000s as it was more efficient than uuencoding or base64. There was good reason for this efficiency. The encoding blatantly ignored the 7-bit limitations of news and e-mail systems, and flooded newsgroups with minimally escaped 8-bit data. CMail included support for encoding data in the yEnc format, but never implemented the checksums or ill-conceived subject format requirements. Luckily, the format never became widespread in e-mail. Detection and decoding of yEnc was never added to the products being tested, so this feature was not needed for testing purposes. Support for yEnc was removed in the 0.8.x branch, and even though the source is still there, along with a comment regarding the arrogance of yEnc's designer, it will not be returning.

For years, I resisted adding an option to attach whole directories. E-mail was not designed for sending large attachments. I finally added support for this in the form of the -awild option. The was intended to allow user to specify files with specific attributes (e.g. archive) similar to the Windows xcopy command, and additionally to include or exclude files based on wildmat expressions. In practice, this command was simply too complex for most users and in the future may be replaced with a simple option to include a directory. I should have followed the KISS principle.

Early versions of CMail supported both inline (uuencode) and MIME attachments in a single e-mail. This functionality was disabled due to compatibility issues with various mail clients. The MIME layout was also modified to improve compatibility of HTML messages with inline images on iOS.

*Nix and CMailLib

The original releases of CMail targeted both FreeBSD and Linux in addition to Windows. The *nix platforms already had numerous command line mailers, so support for these was dropped early on. Windows had fewer options, especially if SSL support was desired, and being the primary platform on which QA tested, remained the sole OS supported.

Over the years there have been many requests for a library version of CMail. There actually was a library version early on (CMailLib) that shared common code with CMail. It was only ever released to a small number of users who requested it. It directly manipulated the same data structures CMail used, and quickly became outdated as CMail evolved.

There were two reasons why I did not continue CMailLib, despite numerous requests for it. The API was little more than a wrapper around SMTP. It required knowledge of SMTP sessions to do anything more than simply send a message. The API was also powerful enough to be abused by spammers intent on using it as the basis of a bulk mailer, allowing hundreds of messages to be sent with a single connection. Most users would have been content with the ability to send a simple e-mail, but it also would have come with a significant burden on myself. While I am happy to assist when users need help with CMail, I cannot spend time fixing their code.

All mail code was removed from cmail.c in the 0.8.x branch. This file now contains little more than the command line parser and documentation. A new API was developed for CMail and other applications to make use of. Currently there is no plan to release a library version.

Compilers etcetera

CMail was originally compiled using MSVC on Windows. FreeBSD and Linux versions were compiled using GCC and hand-written Make files. Subsequently, all releases up to and including 0.7.x were built using GCC (MinGW) with editing and compilation taken care of by Code::Blocks.

With the need for 64-bit versions and FreeBSD switching to Clang as their default compiler, I decided to start moving all my projects in that direction. For the 0.8.x release, I switched to CLion as my default editor. CLion uses CMake, which despite my initial apprehension relying on any third-party tools, makes compiling on multiple platforms easy.

Originally, CMail used OpenSSL 0.9.x. Due to API changes after OpenSSL 1.0.x, I decided to investigate other options for CMail 0.8.x. I settled on LibreSSL, a fork of OpenSSL, primarily because they had designed an improved API (libtls) that could be implemented with only a few function calls.

CMail 0.8.x now uses re2c, an open source lexer generator. This is used for parsing pre-prepared e-mails to extract addresses and headers. While lexer generators are something I'd played with previously, moving to a standard toolchain for all development has simplified the ability to integrate such tools into the development cycle. It's possible CMail may use this for command line processing in the future, making cmail.c an even less significant component of CMail going forward.

It been about a decade since I routinely used CMail in my daily activities. Like many users, my use is now limited to sending automated messages. Even so, I am still committed to maintaining the product and keeping it available free of charge for anyone who wishes to use it.

Disclosure: This article may contain affiliate links. Prices are valid at the time of purchase and/or publication. Refer to our Privacy Policy.

 
 

 

©2020 Inveigle.net
Privacy Policy