Configure Your Web-based Application to Receive Email Using Wildcard Virtual Subdomains, Postfix and Linux

John Reeve | July 14th, 2011 | , , , ,

Wildcard Virtual Subdomains, Postfix and LinuxMany web-based applications, including online project management software, feature the ability to accept and process emails. For example, an incoming email may be tweeted, posted to flickr, or appended to a task. The online abundance of productivity tools has helped us get out of our inboxes, but they haven’t replaced email entirely. In fact, many web-based apps out there integrate email quite nicely.

We recently began overhauling the Intervals email integration feature, transitioning from polling our customers POP/IMAP servers toward enabling customers to email their Intervals accounts directly. To accomplish this, we needed to configure our mail servers to accept mail on wildcard subdomains using Postfix and Linux. It took a bit of tinkering and reading through a myriad of online forums and documentation to get it working. Thought I would blog how we successfully integrated Linux and Postfix virtual mailboxes directly into our Intervals account, in the hopes that someone else may find this useful if faced with the same challenge.

Step 1: Configure DNS

DNS takes a while to propagate, so the first thing you will want to do is set up your DNS so it will serve up the proper MX records for subdomain requests. You will need to add an A record for the IP of your mail server, and a wildcard MX record for the domain. It should look like this:
mx.example.com. 86400 IN A 1.2.3.4
* 86400 IN MX 10 mx.example.com.

Note: You may want to set the TTL lower until you get things working smoothly.

To test if the DNS updates for the MX record have propagated to your servers, run the following command:
dig MX subdomain.example.com
In the response you should see:
;; ANSWER SECTION:
subdomain.example.com. 86400 IN MX 10 mx.example.com.

Once you see this you know DNS has updated. But don’t wait around for DNS to resolve before starting in on the next steps. There is plenty you need to be doing in the meanwhile to configure the mail server for wildcard virtual subdomains.

Step 2: Create a user and group on the Linux server for reading and writing to the mailbox file

Once we have Postfix completely configured, emails will be written to a text-based mailbox file on the server. Postfix is going to need the UID and GID of a user that can access this mailbox file. In this step we create the user and group that Postfix, and your web-based application, will use to receive, update, and remove emails on the server.

First, we create a group:
groupadd -g 5000 vmail
Second, we create a user with no login access and the mail directory as their home directory:
useradd -d /var/spool/mail -s /sbin/nologin -u 5000 -g 5000 -G mail -M vmail
Note: The -G option adds this user to the ‘mail’ group so that it can gain write access to /var/spool/mail. On your system, this should match whatever group is specified for /var/spool/mail directory.

Third, we change the group on the mail file that was created so that our vmail user owns the file outright:
chgrp vmail /var/spool/mail/vmail
Finally, we add apache to the vmail group so that our web scripts can access the mailbox file. To do this, we edit /etc/group and add apache to the vmail line, so it looks like this:
vmail:x:5000:apache
Note: ‘apache’ should be whatever group your web server is running under. To find this out, look at the httpd.conf file.

When you are finished, you should see something like this in the /var/spool/mail directory:
-rw-rw---- 1 imail imail 65404 Jul 14 10:09 vmail

By creating a vmail user and group and associating it with the apache user and group, we avoid having to step on the toes of the postfix and mail users. This is important! We want to avoid a scenario where the apache or vmail user might accidentally, or maliciously, be compromised to gain access to the other mailbox files on the server.

Step 3: Configure Postfix to handle multiple virtual subdomains

The final step is to configure Postfix to accept email being sent to wildcard virtual subdomains. To accomplish this we use the virtual mailbox features of postfix and a few regular expressions to enable wildcards for our subdomains.

First, use your favorite text editor to create the virtual mailbox domain file in /etc/postfix/vdomains. This file contains the regular expression for matching the recipient’s email subdomain to a valid virtual domain.
/((\w[\w\-]*)\.)+example\.com/ OK

Second, create the virtual mailbox maps file in /etc/postfix/vmailbox. This file contains the regular expression for the postfix wildcard alias for matching the recipient’s email subdomain to a valid virtual mailbox.
/@((\w[\w\-]*)\.)+example\.com/ vmail

Third, add the following configuration values to the /etc/postfix/main.cf file. These configuration values tell Postfix where and how to store email sent to these virtual subdomains, and who has the permission to access the virtual mailbox file.
virtual_mailbox_domains = pcre:/etc/postfix/vdomains
virtual_mailbox_base = /var/mail
virtual_mailbox_maps = pcre:/etc/postfix/vmailbox
virtual_minimum_uid = 5000
virtual_uid_maps = static:5000
virtual_gid_maps = static:5000
virtual_mailbox_limit = 0

Note: The virtual_mailbox_limit is set to zero here to denote that the mailbox filesize should not be restricted, since the default is only 52mb. You can set this value to whatever you would like.

Finally, create a lookup table file for the virtual mailbox maps and then reload postfix. Type these commands at the Linux prompt:
postmap /etc/postfix/vmailbox
postfix reload

That’s it! Emails should start flowing into the server now, assuming DNS has propagated. Now it’s up to you to do something with the emails. If you happen to be using PHP, take a look at the imap functions or the Mail_Mbox PEAR library for working with emails in a mailbox file. While there are ways to further configure Postfix to stream emails directly to PHP, we’ve opted to keep them in their native mbox format. This way we preserve the original emails and can use other tools, as can you, in any programming language you choose to parse emails from the mailbox file. Now that you’ve set up your mail server to receive email on wildcard virtual subdomains, it’s up to you what you do with the emails. That’s the fun part. Good luck!

Resources

I could not have figured any of this out without others having already done most of the work. Here is a list of articles, forum discussions, and general documentation that helped me out.

  1. Postfix Virtual Domain Hosting Howto
  2. pcre_table – format of Postfix PCRE tables
  3. Postfix Virtual Wildcard Subdomains
  4. Postfix message and mailbox size limits
  5. DNS and sendmail: 21.3.4 Wildcard MX Records
  6. Wildcard Records
  7. Postfix – Hosting Multiple Domains with Virtual Accounts

Notes

  1. If the mail server is behind a firewall, you will need to open up port 25 so that it can communicate with other SMTP servers.
  2. This post assumes you already have Postfix installed and sending emails.
  3. To find out what types of lookup tables your Postfix system supports use the “postconf -m” command.
  4. We are using PCRE for the regular expression matching, which requires a replacment string. Since the vdomains file is only looking for a match, we can use any arbitrary replacement string. In our example, we simply use “OK.” An empty string works as well. If the replacement string is not specified, the pattern matching will still work, however, warnings will show up in the mail logs.
  5. Although the documentation for Postfix reads “separate domains, non-UNIX accounts,” you still need at least one user account devoted to handling these emails. This is because you’ll need a user for writing mail to the mailbox file, so might as well have the mailbox be for the user we created in step 2.
  6. If you’d like to do some filtering on incoming emails, the header_checks configuration parameter is a great place to start.
  7. If you’ve done all of the above and are still running into problems, post a comment with questions. I’ll do the best I can to help out.

Photo credit: Horia Varlan

9 Responses to “Configure Your Web-based Application to Receive Email Using Wildcard Virtual Subdomains, Postfix and Linux”

  1. Göran Svensson says:

    I have a dumb question. Will your set-up work for this scenario:

    I have the account nisse on my postfix box.

    If I send an e-mail to nisse@whatever.example.com

    the email will arrive in nisse mailbox.

    And the same if I send it to nisse@testing.example.com

    If so I could have found a working solution for my scenario…

  2. John Reeve says:

    Göran,

    Yes, the regular expression above will match both of those email addresses. Keep in mind, however, that it will also match anything to the left of the @ symbol. For example, emails sent to john@whatever.example.com and john@testing.example.com will end up in the same mailbox as emails sent to nisse@.

    To make it more precise, you could change the vmailbox entry to:
    /nisse@((w[w-]*).)+example.com/ vmail

  3. Morten Rasmussen says:

    Hi,

    I just tried to follow your guide.
    – I ran into some problems. Maybe you are willing to help me?

    I completed the guide, no problems.
    When i try to telnet and send a mail from there i get a “451 4.3.0 : Temporary lookup failure” when i rcpt to: test@something.mydomain.com

    Do you have any idea why that is happening?

    Morten Rasmussen

  4. John Reeve says:

    @Morten,

    I have not encountered this error before. However, I did do some googling around and it looks like Postfix may be having some issues accessing the user database:
    http://logs.tobinelectronics.com/postfix-mysql-error-451-4-3-0-temporary-lookup-failure/

    If you are using MySQL in conjuction with Postfix, then this is quite possibly the issue.
    More information can be found here:
    http://www.postfix.org/MYSQL_README.html

    Good luck!

  5. Morten Rasmussen says:

    Thanks for the reply.

    I dont use MySQL. Just checked the mail.log.

    The message gets delivered, but gets status=deferred and the error delivery failed to mailbox /var/mail/vmail: cannot open file: Is a directory.

    I know what it means, but not why the error is happening.

    Have you got any idea?

  6. John Reeve says:

    It sounds like Postfix is misconfigured. It’s trying to open the vmail file to look up the user/recipient, but instead is finding a directory. Check main.cf to make sure it’s pointing at the correct file:
    main.cf:virtual_mailbox_maps = pcre:/etc/postfix/vmailbox

  7. Morten Rasmussen says:

    Sorry for spamming!

    Just wanted to share the solution.

    In main.cf i changed

    virtual_mailbox_domains = pcre:/etc/postfix/vdomains

    to

    virtual_alias_maps = pcre:/etc/postfix/vdomains

    And changed

    /((w[w-]*).)+example.com/ OK

    to

    /((w[w-]*).)+example.com/ vmail

    Dont ask me how or why! But now i receive my mails..

    Thanks for your help!

  8. Abdullah Al Noman says:

    Not finding any directory in /var/spool/mail

  9. John Reeve says:

    First, check that the /var/spool/mail directory exists. If it does not, there are two possible reasons. One is that postfix is not installed. The second is that the distro you are using installed the postfix mailbox files to a different location.

    If the directory does exist, it should be populated with mailbox files (text files) for each user. If the mailbox files are not there, I would suspect something is wrong with the postfix configuration.

    Hope that gets you pointed in the right direction.

Leave a Reply

Intervals Blog

A collection of useful tips, tales and opinions based on decades of collective experience designing and developing web sites and web-based applications.

What is Intervals?

Intervals is online time, task and project management software built by and for web designers, developers and creatives.
Learn more…

John Reeve
Author Profile
John Reeve

John is a co-founder, web designer and developer at Pelago. His blog posts are inspired by everyday encounters with designers, developers, creatives and small businesses in general. John is an avid reader and road cyclist.
» More about John
» Read posts by John

Jennifer Payne
Author Profile
Jennifer Payne

Jennifer is the Director of Quality and Efficiency at Pelago. Her blog posts are based largely on her experience working with teams to improve harmony and productivity. Jennifer is a cat person.
» More about Jennifer
» Read posts by Jennifer

Michael Payne
Author Profile
Michael Payne

Michael is a co-founder and product architect at Pelago. His contributions stem from experiences managing the development process behind web sites and web-based applications such as Intervals. Michael drives a 1990 Volkswagen Carat with a rebuilt 2.4 liter engine from GoWesty.
» More about Michael
» Read posts by Michael

help.myintervals.com
Videos, tips & tricks