Project management can sometimes present a web design, development or creative agency, with a Catch-22-like conundrum. Given the unique and customized nature of many web sites and web-based applications, it is difficult to estimate how long it will take to develop, manage and deliver a new project until we’ve actually gone through the process. For a new project manager or new class of project, it can be overwhelming to the point we get stuck in an infinite loop of analysis paralysis.

How do we estimate a project we’ve never completed before? How do we complete a project we’ve never estimated before? There isn’t an easy or one-size-fits-all solution to this problem. We use a process we’ve dubbed “Predict, Track and Learn.” This process is one of the guiding principles behind our online project management software, Intervals. Whether you practice Agile or Waterfall project management methods, this process is applicable..

What does “Predict, Track and Learn” mean?

Every creative agency has its own way of managing projects. Some of us prefer Gantt charts and Waterfall methods, while others prefer Agile and Scrum techniques. Both require that the project manager and her team be able to estimate how long different tasks, stories, or phases will take to complete. Estimate how long the project will take — if you don’t know for sure, make an educated guess. Track your estimated deliverables closely. Once the project is completed, learn from the successes and failures and improve the next time around. Repeat with each project.

Predict

Estimating a project is not an easy process. It takes time and a number of iterative estimates to accurately predict a project. To make the estimation process go easier, break the project up into smaller chunks, for example, individual phases or tasks. As you begin to calculate the number of hours it will take to complete the smaller parts, the estimate will come together with less effort. When in doubt, guess at the number of hours. It’s better than nothing, and you can learn a lot by guessing. Once you’ve got the first draft completed, go back through the individual phases or modules, tuning each one until the total number of hours is at the desired level of accuracy. Depending on the complexity of the project, you may want to increase the total number of hours by 25 to 33% — a multiplier that helps address unknowns and our tendency to underestimate projects.

Track

Tracking your time effectively and accurately requires discipline. Our advice is to track your time as you work. Don’t wait until the end of the day or the week to jot down your time or the results will be incredibly inaccurate. There are several online time tracking applications that will help you. Once you start using one don’t be surprised to find your tracked time increasing by as much as 30%. When it comes to tracking time, paper time sheets and Excel spreadsheets are, literally, a waste of time.

Learn

When the project is completed and the clients have been billed, go back through the data and compare the actual time tracked against the estimates. Some parts of the project will have been underestimated, some overestimated. The actual time, if tracked properly, will show which areas need to be improved next time. The important thing is that now you have a history of time tracked on a specific project, broken down by it’s parts. This data will become invaluable to your business as you move forward.

Repeat

This whole process is meant to be repeated over and over again, your estimates getting more accurate with each project. In the beginning, expect a 15% margin between the estimated and actual time tracking data. That margin will decrease to less than 5% after you’ve gone through this process enough times. The number of times it takes to hone your estimation skills will depend on your abilities, luck, and the amount of custom development required for each project. Web designers and developers tend solve the same problems repeatedly, however, our development projects are not always homogeneous. By breaking down each project into smaller parts you will end up with a library of plug-and-play estimates that can be assembled for your next project.

Photo credit: Myxi

The Intervals team would like to take a moment of your blog perusing to announce a new round of recently launched features. Most importantly, we would like to unveil the new Hopper request queue feature, and it’s cousin, Email integration for appending comments to tasks and milestones, all from the comfort of your inbox. So what do these features do exactly? Read on to find out…

The Hopper

Work requests from clients and co-workers can now be emailed directly to your Intervals account, where they will be processed by the Hopper and placed in your request queue. Up until now, Intervals would regularly poll a POP or IMAP account, hosted by a third-party email provider, for new requests. While we will still support that feature, we encourage our customers to give the Hopper a whirl. The Hopper alleviates the need to manage a third-party mailbox and does away with entering the mailbox server credentials into your Intervals account. Simply tell your clients and co-workers to start sending their requests to request@mydomain.projectaccount.com (or whatever your Intervals domain) and Intervals will do the rest.

Just be sure the people sending in the requests are doing so from an email address that corresponds with a valid client contact, a user with a login, or an email address on the Hopper whitelist. If an email does not make it through the Hopper, the Intervals administrator who is in charge of the account will receive an emailing with details about why the email failed and how the account can be updated to allow the email through the next time.

How does the Hopper work?

The diagram below explains what happens to an email between the time it leaves your computer and when it finally appears in the request queue. The Hopper is primarily responsible for validating the email address against your account and then breaking out the attachments and inline graphics into documents that will go along with the request once it’s in the queue.

Email integration for tasks and milestones

The request queue is not the only feature that has received the email treatment. Tasks and milestones can now have comments emailed directly to them. Each task or milestone notification that goes out via email will contain a delimiter at the top of the email. Simply reply to the email and place your notes above the delimiter. The reply to address will be the unique email address for that task or milestone. And the comments you make will be appended as a new comment to that task or milestone.

If you would like to email a task or milestone directly, simply send an email to it’s unique email address. A task’s email address, for example, would looking something like task+12345@mydomain.projectaccount.com. Or simply click on the email icon (pictured above) to start a new email to the task or milestone in your favorite email client. Once the email is sent and received by our mail servers, Intervals will handle the rest. The new comment should appear in just a minute or two.

When a task or milestone is emailed, it’s followers, owners and assignees will receive an email notification, making it much easier to collaborate and keep the project team updated.

Task and milestone notification emails will now display the above text at the top of each message. Simply reply to the email and anything typed above this line will be appended to the task or milestone as a new comment.

Now you can update tasks and milestones directly from your inbox without having to log in to the web interface. For example, the next time you email a message to a client, use the Cc or Bcc fields in your email program to copy the message to the applicable task or milestone. Updating tasks and milestones with new comments should be a lot easier now, especially when you are away from your computer equipped with nothing but a smartphone.

Increased document storage

We love making our customers happy. We like you and we want you to like us. That’s why we’ve increased the amount of document storage across all of our Intervals plans, at no extra charge to our customers. That’s a lot of free gigabytes. Go ahead and log in to your Intervals account, or create one if you don’t have one, and start taking advantage of the extra storage space. Meanwhile, here is a breakdown of how much document storage was increased per plan:

Other notable improvements

In addition to the major updates mentioned above, we have a few other improvements we were able to sneak into this recent launch. Here are a few items we’ve added to Intervals…

Ability to delete data imports

Decided you didn’t really want to import 3,000 clients into your Intervals account? Whereas before you would only have the option to deactivate all of these imported clients and hide them from your immediate view, now you can delete all of the items associated with an import entirely. This applies to all of the data imports. Sometimes it takes a few tries to get your data imported correctly. The ability to delete data associated with botched imports will reduce the amount of clutter that can occure when importing data the first few times.

New data imports for expenses, payments, invoices

On the subject of data imports, we’ve added to the number of items that can be imported. Now project expenses, payments and invoices can be imported. We want to make it as easy as possible to get your preexisting data into your Intervals account, which is why we continue to improve upon the import functionality. Project expenses, payments and invoices can be a tedious lot to enter manually. The imports aim to annihilate all that tedium with just a few clicks of the mouse.

API support for creating people

The Intervals development team is devoted to evolving the API to encourage our customers, as well as ourselves, to build new functionality on top of the Intervals web-based application platform. We’ve seen many of our customers build some intriguing applications using the API, and even our own mobile app uses the API. To continue encourage our API developers, we’ve added the ability for people to be created via the API. Check out the API documentation for more information.

Photo credit: The Hopper by CJ Schmit

Many web developers can trace their humble beginnings to the family room floor of their childhood, where boxes of Lego bricks spilled out into our imaginative hands and became flying cars, futuristic skyscrapers, impenetrable fortresses, and whatever else we could think to build. Yes, many of us enjoy developing web-based applications because it is so much like playing with Lego. The database, programming language, and server hardware are our building blocks from which we churn out web sites and online applications.

I’m one of those web developers who got his start in Lego and never stopped. When we came up with the idea to build a Lego wall at our office, I immediately hauled in a few buckets worth of Lego and then went to the hardware store for supplies. This How-to will demonstrate how to build your own Lego wall for yourself, your kids, or just a fun event (Lego bricks make a great ice-breaker when left out at parties).

Step 1: Gather up your supplies

You will need the following supplies to get started. Depending on the surface you choose to mount you Lego wall, you may want to substitute the masonite board for a different backing.

• Some wall space
• Stud finder
• Level
• One masonite board cut to size
• Flat-headed wood screws
• Glue strong enough to hold plastic to a smooth wood surface (i.e. Liquid Nails)
• Enough 15″ x 15″ Lego baseplates to tile the masonite surface
• Legos
• Wall mountable storage for Lego bricks (We picked ours up at IKEA)

Step 2: Mount the masonite board

Once the masonite board has been cut down to the appropriate size you will need to mount it. We recommend hitting at least a few studs considering the force generated by pulling Lego bricks off of the wall. Mount the masonite board using standard wood screws, and drywall anchors when a stud can’t be found. Use a level while mounting the masonite to make it even with the floor. Make sure to use flat-headed screws and drill them in far enough so the heads are flush with the surface of the masonite (otherwise, the Lego baseplates will have bumps underneath them when mounted.) The masonite board has a tough service so you will really have to drive in the screws to get them flush.

Step 3: Glue on the Lego baseplates

Using a caulking gun or a hand-squeezable tube, dab the glue onto the backs of the Lego baseplates. Good coverage is important to make sure sections of the baseplate won’t lift up. Press the baseplate onto the wall and use tape to secure it in place. This will stop the plate from sliding down while the glue dries. After the first baseplate has been mounted, use lego bricks to mount the second plate. By spanning the two baseplates using Lego bricks, you will ensure that the plates are properly spaced so Lego pieces will fit across them. This will result in a gap of approximately 1mm, which is somewhat conspicuous, but gaurantess the baseplates are properly spaced. Tape the additional baseplates in place as well to make sure the added weight does not pull the baseplates down.

Step 4: Lego brick storage

Now decide where you want to store the Lego bricks. We chose to store them in buckets mounted below the Lego wall. We used some standard storage items we found at IKEA. You may want to just throw you Lego bricks into a box at the base of the Lego wall. Whatever you decide is up to you.

Step 5: Build something!

That’s it. The Lego wall is complete. Build anything you want. We decided to build a pixel version of our logo (which I’ll walk about in another blog post). Next up we’re planning on doing some of our favorite video game characters. The best part is that anyone can build anything they want, and the Lego wall makes for a nice break away from all the time we spend coding web sites.

Lego minifig photo credit: brad montgomery

This last July I had another opportunity to attend OSCON 2011 in Portland, Oregon. So I did what any self-described beer and technology enthusiast would do, I jumped on a plane and flew north. While in attendance, I soaked in as much knowledge as I could — insight, instruction, tips, and funny stories about the open source software running most of the Internet and Web-based applications many of us use in our day-to-day online. The following is a sampling of what I took away from a week of geek speak.

Netflix is moving massive amounts of data to the cloud

Typically I don’t attend Keynotes (mostly because there is something more interesting, like lunch, going on at the same time). But this year I made an exception, and I’m glad I did. The Netflix keynote was a 19-minute insight into the inner workings of how Netflix is moving their streaming movie data to the cloud, and in the process, moving data to new stores. Did you know that one movie takes Netflix an entire day to encode into 50 different audio and video files? Then the data is replicated three times across three timezones, and then replicated two more times to two different vendors, giving Netflix geographically redundant data backups. And while moving data around to these new servers, ratings and queues are moving to Cassandra. So the next time you can’t watch an instant movie on Netflix, it’s likely that this move into the cloud is the culprit.

Postgres 9.1 is another step in the right direction

I’ve long been an advocate for Postgres as one of the best choices for an open source database. But it has been lacking in some features, mainly streaming replication. Postgres 9.0 gave us replication, and 9.1 is taking that a step further with synchronous replication. For those of us that require zero data loss in our master-to-slave replication setups, this is just what we’ve been waiting for. In addition, Postgres is replacing contrib with more pluggable extensions available at www.pgxn.com. As it continues to evolve, it’s nice to see Postgres becoming the database of choice for projects like Django and Apple’s own Lion X Server.

Facebook is still working out where to store all of our data

Did you know that Facebook is collecting 25TB of message data alone per month? All of those messages are going into Hbase and Haystack data stores on clusters upon clusters of servers. Each cluster has a stack — running open source software such as Memcached, ZooKeeper, Hbase, Hadoop and their own Haystack — spread across 5 racks of 20 servers. That’s 100 servers per cluster. As Facebook migrates their data away from MySQL and shards it onto these clusters, they reduce the number of users affected by a single outage. This also explains how they are able to roll out new updates to a percentage of users at a time. Aside from finding a place to stash all of this data, there wasn’t any mention of what Facebook is going to do with all of our messages. <insert SkyNet reference here>.

MongoDB, and its noSQL ilk, are not to be ignored

The noSQL movement is gathering a tremendous amount of momentum and popping up in places I don’t think anyone expected it to. When discussing whether or not noSQL data stores would replace relational databases, we’ve been focusing too much on an all-or-nothing approach. Meanwhile, open source noSQL, or schema-less, databases found their niche in service oriented architectures. I can tell you from personal experience that the database is always going to present a bottleneck, and furthermore, it’s gone to be one aspect of the database that does so. Companies like Netflix and Facebook have figured this out and are resolving these bottlenecks by replacing the problematic services with Hbase, Cassandra and MongoDB. These schema-less databases can be especially useful when all you need is a basic key-value store. Over the next few years we are going to see schema-less databases adopting features from relational databases, and vice-versa. There is going to be a lot of innovation in this space as these open source databases mature.

The Netflix API is not what it used to be

When I left the talk on changes coming to the Netflix API I thought they were crazy (which I’ll explain). But as I’ve been mulling over it, I can see how their situation is truly unique and they need to do something unprecedented to address it. When Netflix created their API, they did so like any of us would — creating a standards-based API that would attract developers too their platform. And it worked. In the beginning, 99% of their API usage comes from individual developers. Now the tides have turned and 99% of the API usage comes from Netflix streaming devices. This has resulted in 20 billion API requests per month, or 20,000 per second. The solution? In short, they want to provide custom API hooks for devices to cut the number of requests per month down to 5 billion (a 75% reduction). A custom API interface per device?!?!?! Yeah, that was my thought exactly. But then, if the Roku player is always making the same four requests to display your instant queue, why not give them a custom API call so they can reduce that to one request? You’ve reduced your API requests by 75% and you’re also helping the device run faster. Netflix has a long ways to go — they are still in the conceptual phases of overhauling the API — before they pull this off, but they might just pull it off and do some pretty innovative stuff in the process.

It’s really hard to test scaling problems before deployment

Try as we might, it’s quite difficult to replicate a production environment. We can match the number of servers, the amount of bandwidth, possibly even an approximation of the load we expect to receive, but, we’ll never get it right. The truth is, we can never fully estimate what the real world is going to throw at our web-based application until we put it out there as a target. Our users, simply put, are too unpredictable. That doesn’t we shouldn’t do load testing in a staging environment, we should. It means at some point we just have to put it out there and see what happens, and be on the ready to quickly stamp out any fires that may crop up. As a web developer who has deployed features into the great unknown, it is reassuring to know that other web developers are up against the same problems and are coming up with similar solutions. It’s like they say, it’s easier to beg forgiveness than it is to ask permission. Which brings us to the topic of DevOps…

DevOps are the new SysAdmins

As web developers, it used to be we could place most the blame for web-based application failures on systems outside of our control. We could point fingers at a misconfigured database, an overworked web server, or some other constellation in the corner of the infrastructure. With the widespread adoption of Memcached, NGINX, and noSQL databases, bad coding is becoming exposed more as the bottleneck in modern web-based applications. Our apps have to perform well, and when they don’t, we have to fix them. DevOps are the new SysAdmins, meaning, if something breaks, we have to go in and get the app back on its feet. I’ve always been an advocate of developers understanding how their apps interact with the server environment where they are hosted, now it’s becoming a necessity. And if you ever get a chance to see Terry Chay speak at a conference, he’s definitely good for a few laughs ;)

Infrastructure is quickly becoming a commodity

Everywhere I went people were talking about the cloud, and have been long before I went to this conference. While the cloud is a great step forward in web-based application development, it is not a one-size-fits-all solution. The good parts? A developer can spin up any number of web server instances to meet an application’s traffic demands as they ebb and flow. Just a few clicks and more servers are deployed to handle load. Storage constraints, also, are easily addressed by allotting more disk space. Paying for usage means hardware isn’t sitting around unused. It’s easy to expand only when necessary, meaning our hosting needs can grow alongside our budget. We don’t have to project our server needs out years in advance, we can meet those needs tomorrow. But, it isn’t perfect. Servers still go down, file systems are corrupted, noisy neighbors can negatively affect our apps, and performance isn’t always the best. I’ve personally seen all of these happen in the cloud and had to wait while a system administrator troubleshoots and resolves the issue. No, the cloud isn’t perfect, but it definitely has it’s advantages. Would I move a dedicated bare-metal database server to the cloud? No. But I’m considering more letting the cloud handle commodity hosting needs, like web servers and load balancing.

Photo credits: HighlandBlade & mitmall

Many web-based applications, including online project management software, feature the ability to accept and process emails. For example, an incoming email may be tweeted, posted to flickr, or appended to a task. The online abundance of productivity tools has helped us get out of our inboxes, but they haven’t replaced email entirely. In fact, many web-based apps out there integrate email quite nicely.

We recently began overhauling the Intervals email integration feature, transitioning from polling our customers POP/IMAP servers toward enabling customers to email their Intervals accounts directly. To accomplish this, we needed to configure our mail servers to accept mail on wildcard subdomains using Postfix and Linux. It took a bit of tinkering and reading through a myriad of online forums and documentation to get it working. Thought I would blog how we did it, in the hopes that someone else may find this useful if faced with the same challenge.

Step 1: Configure DNS

DNS takes a while to propogate, so the first thing you will want to do is set up your DNS so it will serve up the proper MX records for subdomain requests. You will need to add an A record for the IP of your mail server, and a wildcard MX record for the domain. It should look like this:
mx.example.com. 86400 IN A 1.2.3.4
* 86400 IN MX 10 mx.example.com.
Note: You may want to set the TTL lower until you get things working smoothly.

To test if the DNS updates for the MX record have propagated to your servers, run the following command:
dig MX subdomain.example.com
In the response you should see:
;; ANSWER SECTION:
subdomain.example.com. 86400 IN MX 10 mx.example.com.
Once you see this you know DNS has updated. But don’t wait around for DNS to resolve before starting in on the next steps. There is plenty you need to be doing in the meanwhile to configure the mail server for wildcard virtual subdomains.

Step 2: Create a user and group on the Linux server for reading and writing to the mailbox file

Once we have Postfix completely configured, emails will be written to a text-based mailbox file on the server. Postfix is going to need the UID and GID of a user that can access this mailbox file. In this step we create the user and group that Postfix, and your web-based application, will use to receive, update, and remove emails on the server.

First, we create a group:
groupadd -g 5000 vmail
Second, we create a user with no login access and the mail directory as their home directory:
useradd -d /var/spool/mail -s /sbin/nologin -u 5000 -g 5000 -G mail -M vmail
Note: The -G option adds this user to the ‘mail’ group so that it can gain write access to /var/spool/mail. On your system, this should match whatever group is specified for /var/spool/mail directory.

Third, we change the group on the mail file that was created so that our vmail user owns the file outright:
chgrp vmail /var/spool/mail/vmail
Finally, we add apache to the vmail group so that our web scripts can access the mailbox file. To do this, we edit /etc/group and add apache to the vmail line, so it looks like this:
vmail:x:5000:apache
Note: ‘apache’ should be whatever group your web server is running under. To find this out, look at the httpd.conf file.

When you are finished, you should see something like this in the /var/spool/mail directory:
-rw-rw---- 1 imail imail 65404 Jul 14 10:09 vmail

By creating a vmail user and group and associating it with the apache user and group, we avoid having to step on the toes of the postfix and mail users. This is important! We want to avoid a scenario where the apache or vmail user might accidentally, or maliciously, be compromised to gain access to the other mailbox files on the server.

Step 3: Configure Postfix to handle multiple virtual subdomains

The final step is to configure Postfix to accept email being sent to wildcard virtual subdomains. To accomplish this we use the virtual mailbox features of postfix and a few regular expressions to enable wildcards for our subdomains.

First, use your favorite text editor to create the virtual mailbox domain file in /etc/postfix/vdomains. This file contains the regular expression for matching the recipient’s email subdomain to a valid virtual domain.
/((\w[\w\-]*)\.)+example\.com/ OK

Second, create the virtual mailbox maps file in /etc/postfix/vmailbox. This file contains the regular expression for matching the recipient’s email subdomain to a valid virtual mailbox.
/@((\w[\w\-]*)\.)+example\.com/ vmail

Third, add the following configuration values to the /etc/postfix/main.cf file. These configuration values tell Postfix where and how to store email sent to these virtual subdomains, and who has the permission to access the virtual mailbox file.
virtual_mailbox_domains = pcre:/etc/postfix/vdomains
virtual_mailbox_base = /var/mail
virtual_mailbox_maps = pcre:/etc/postfix/vmailbox
virtual_minimum_uid = 5000
virtual_uid_maps = static:5000
virtual_gid_maps = static:5000
virtual_mailbox_limit = 0
Note: The virtual_mailbox_limit is set to zero here to denote that the mailbox filesize should not be restricted, since the default is only 52mb. You can set this value to whatever you would like.

Finally, create a lookup table file for the virtual mailbox maps and then reload postfix. Type these commands at the Linux prompt:
postmap /etc/postfix/vmailbox
postfix reload

That’s it! Emails should start flowing into the server now, assuming DNS has propagated. Now it’s up to you to do something with the emails. If you happen to be using PHP, take a look at the imap functions or the Mail_Mbox PEAR library for working with emails in a mailbox file. While there are ways to further configure Postfix to stream emails directly to PHP, we’ve opted to keep them in their native mbox format. This way we preserve the original emails and can use other tools, as can you, in any programming language you choose to parse emails from the mailbox file. Now that you’ve set up your mail server to receive email on wildcard virtual subdomains, it’s up to you what you do with the emails. That’s the fun part. Good luck!

Resources

I could not have figured any of this out without others having already done most of the work. Here is a list of articles, forum discussions, and general documentation that helped me out.

1. Postfix Virtual Domain Hosting Howto
2. pcre_table – format of Postfix PCRE tables
3. Postfix Virtual Wildcard Subdomains
4. catchall alias for wildcard subdomains?
5. Postfix message and mailbox size limits
6. DNS and sendmail: 21.3.4 Wildcard MX Records
7. Wildcard Records
8. Postfix – Hosting Multiple Domains with Virtual Accounts

Notes

1. If the mail server is behind a firewall, you will need to open up port 25 so that it can communicate with other SMTP servers.
2. This post assumes you already have Postfix installed and sending emails.
3. To find out what types of lookup tables your Postfix system supports use the “postconf -m” command.
4. We are using PCRE for the regular expression matching, which requires a replacment string. Since the vdomains file is only looking for a match, we can use any arbitrary replacement string. In our example, we simply use “OK.” An empty string works as well. If the replacement string is not specified, the pattern matching will still work, however, warnings will show up in the mail logs.
5. Although the documentation for Postfix reads “separate domains, non-UNIX accounts,” you still need at least one user account devoted to handling these emails. This is because you’ll need a user for writing mail to the mailbox file, so might as well have the mailbox be for the user we created in step 2.
6. If you’d like to do some filtering on incoming emails, the header_checks configuration parameter is a great place to start.
7. If you’ve done all of the above and are still running into problems, post a comment with questions. I’ll do the best I can to help out.

Photo credit: Horia Varlan