The Intervals Blog
A collection of useful tips, tales and opinions based on decades of collective experience designing and developing web sites and web-based applications.

Looking Back on Old School Web Design & Development Methods

June 30th, 2011 by John Reeve

Old School Web Design & DevelopmentThe Web, as we know it today, has advanced a considerable amount from it’s original incarnation. HTML hasn’t changed as much as the web browser, and web designer and developers have advocated and implemented higher standards in their work. From CSS improvements to better use of semantic HTML, the Web has (mostly) become a better online environment for both designers, developers and those who casually and professionally use the Web every day.

It wasn’t always this way. It used to be the Web was used primarily for collecting and sharing information. Web-based applications were in the minority, and even then they weren’t very good. To illustrate just how far we’ve come in web design and development, here is a collection of ‘best-practice’ tips from the 1990s, before the wild west of the Web was tamed and us settlers had a whole different way of building Web sites.

Validation? What validation?

Validation was not an issue. Why? Because it was not possible to have a site that validated and looked the way we wanted it to in every browser. And faced with this predicament, we ignored what we’d been taught in design school and went with our designer’s instinct — function follows form. Besides, no one was really pushing the issue because the Web was still in its infancy. We all knew it was going to explode at some point, but we didn’t know how.

The four horsemen of the <body> element

Every web site we developed started out with the same four parameters in the <body> tag — LEFTMARGIN, TOPMARGIN, MARGINWIDTH, MARGINHEIGHT. It was the only way to make sure the margins around the Web page worked in every browser. CSS existed, but browser support was fickle, and this just worked.

Pixel precision layout with tables

The <table> tag and it’s ilk were your most used tool. Throw in a transparent GIF and no layout was too challenging. Any layout, no matter how rigid, could be designed and developed on the Web by using empty <td> elements containing a 1×1 pixel transparent GIF, resized to whatever size the margin needed to be. The table HTML tag was synonymous with a grid layout, not tabular data. And there were great apps out, like NetObjects Fusion, that made designing and developing Web sites in this manner a no-brainer. Today we see Web sites implementing tables mostly for tabular data and utilizing all parts of the table specification — head, body, and foot.

Seven different font sizes using the <font> tag

Everything from headlines to captions could be accomplished by nesting the items inside the <font> tag. Sure, we had to copy and paste the <font> tag repeatedly inside of every table cell that contained text, but it gave us more control of text display in the Web browser.

Graphical text: When the <font> just wasn’t enough

The <font> tag only gave us seven options. The elaborate comps we had designed in Photoshop, however, had more variations in text. It also had navigation text that changed color when you moused over it, and headlines branded with a similar non-standard typeface. So we cut our Photoshop text into several tiny pieces and saved each one as a GIF image that could be easily read by a human, but was invisible to search engines and screen readers. And we didn’t seem to mind maintaining Web sites that used graphical text. We just kept a PSD file laying around that we could open up, edit the copy, cut, paste, save as, and upload.

Looking back…

We’ve come a long way since the 1990s, a time when function followed form, when we treated the Web much like the wild west, a world with it’s own rules, many of which we got to make up ourselves. Thankfully, Web browsers, along with HTML and CSS standards, have improved. And the collective knowledge of today’s Web designers and developers have been leaders in making that happen, advocating for a better Web, where finally, form follows function. We’ve learned to work inside the limitations of the Web browser, recognizing it as an evolving medium where the classic rules of good graphic design still apply. In 2001, Jeffrey Zeldman wrote a great article about the state of Web design and development and where we were headed.

The separation of style from content is at ALA today, but tomorrow, or the day after, it will be at all sites. Far from seeming revolutionary, difficult, dangerous, or non-user-friendly, it will simply be the way the medium functions. It is, after all, the way the web was always meant to work.
— Jeffrey Zeldman, 2001

Fortunately, much of his vision for the future of Web design and development has been realized. Web design and development has evolved, the separation of style and content has been embraced as a best-practice, and the Web has become ubiquitous in both our personal and professional lives.

For more information on web standards, check out the Web Standards Project, a “grassroots coalition fighting for standards which ensure simple, affordable access to web technologies for all.”

I’ve only mentioned a few of the web design and development methods common in the 1990s. Please chime in and comment on your experiences.

Photo credit: agjiminez

Tags: , , ,
| 3 Comments »

Calculate the number of lines in a CSV file using PHP and Linux… without loading the file into application memory

June 21st, 2011 by John Reeve

Calculate the number of lines in a CSV file using PHP and Linux

When parsing large text files, particularly CSV files, web developers often need to know the number of lines contained in the file up front. This is useful for a number of reasons, for example, displaying a progress meter while parsing the large file, or estimating how long the parsing process might take. There are two ways to do this when using PHP for web development. Unfortunately, they both require using more memory than is necessary — just to get a line count. The good news is that the line count can be calculated very quickly using the PHP exec() function and some simple Linux utilities. But first, let’s take a look at the PHP methods…

Method #1: Parsing the entire CSV file at once using PHP

This first method uses the file() function to read the contents of the file into an array, from which the count() method can be used to derive the line count:
$linecount = count(file('filename.csv'));
Reading an entire file into memory just to get a line count? While it may not affect your code when parsing smaller files, larger CSV files will most definitely chew up memory and cause the maximum memory limit (designated in php.ini) to stop the script. And yet almost every forum out there suggests this solution when people ask how to calculate the number of lines in a CSV file. Sometimes, simple code is not so simple.

Method #2: Parsing the CSV file line-by-line using PHP

There is another method for calculating line count that requires less memory. Using the PHP’s filesystem functions — fopen(), fgets(), and fclose() to loop through the file, reading one line into memory at a time, and incrementing the line count as we do so:
$fh = fopen('filename.csv','rb') or die("ERROR OPENING DATA");
while (fgets($this->fh) !== false) $linecount++;
fclose($fh);
While this method is certainly faster than the previous, it does still require reading the entire line into memory just to count it. For CSV files with a small number of columns, this won’t matter. Parsing larger files with a greater number of columns will end up using memory that doesn’t necessarily need to be used, and will take up processing time doing so.

Method #3: Using Linux utilities to calculate the number of lines

Perhaps one of the most powerful, and most dangerous, functions available in PHP is the exec() function. This function allows us to drop down into the Linux environment running underneath PHP and issue it command line directives. Using the exec() function gives us access to the wc linux utility to print the number of newlines, words, and bytes in files.

When run on the command line, the output looks like this:
[jreeve@pelago ~]$ wc -l filename.csv
371 filename.csv

To run this in PHP and parse the output, we would do this:
list($linecount,$foo) = split(" ", exec('wc -l ' . escapeshellarg('filename.csv')));

There is only one problem with this approach. CSV files generated by different spreadsheet programs and OSes use different characters to represent a new line. While Linux uses the newline character, represented as ‘\n’, or decimal 12 (LF), Windows uses carriage-return newline pairs, represented as ‘\r\n’ or decimal 15,12 (CR+LF). To further complicate the matter, Mac OS, up until version 9, used carriage-return characters, represented as ‘\r’ or decimal 15 (CR). The wc utility has a difficult time counting the number of lines in CSV files generated using older versions of the Mac OS, and possibly other OSes and software, that use a sole carriage-return (CR) to denote each new line.

To remedy this, we use wc in combination with perl:
[jreeve@pelago ~]$ perl -pe 's/\r\n|\n|\r/\n/g' filename.csv | wc -l
371

By running the file through perl on the command line we are able to substitute variations of the newline character that we may encounter. It should be noted here that using the tr linux utility, which is often suggested, will not work because it can only substitute one character, and we need to substitute a two-character set.

To replicate this in PHP, we would do:
$linecount = exec('perl -pe \'s/\r\n|\n|\r/\n/g\' ' . escapeshellarg('filename.csv') . ' | wc -l');

Now we’ve got a fast and efficient method for getting around PHP and down into the Linux OS to do some of the heavy lifting for us. It makes sense. If there is a better tool for the job, use that tool, especially if that tool is default to the native environment we are accustomed to working in. Fortunately for us web developers, PHP and Linux go together like Ponch and Jon, making development challenges like this both efficient and fun.

Update: I neglected to mention in the article that when faced with this web development challenge we did end up using method #2 in our web-based application, Intervals. We were able to do it without too much of a performance hit. However, if performance did become an issue, I wouldn’t hesitate to implement method #3.

Photo credit: Eric Fischer

Tags: , , , , , ,
| 6 Comments »

Web-based Application Development Tip: Protect Your “Tenants”

June 17th, 2011 by John Reeve

Web Development Web-based Application Security

According to the New York times, thieves were able to capture the names, account numbers, e-mail addresses and transaction histories of more than 200,000 Citi customers, using the Citigroup customer web site. This isn’t the first time a financial institution has had its data compromised. What is so alarming about this particular security breach was how simply the web site was exploited. The article explains how the thieves were able to get at the data:

In the Citi breach, the data thieves were able to penetrate the bank’s defenses by first logging on to the site reserved for its credit card customers. Once inside, they leapfrogged between the accounts of different Citi customers by inserting vari-ous account numbers into a string of text located in the browser’s address bar.

Citigroup made a basic mistake, that surprisingly, is often made by web developers — yes, I’ve made this mistake before, but was lucky to have it caught by QA before going to production. When developing a multi-tenant web-based application that will serve up data to multiple customers, it’s important to make sure those customers can’t access one another’s data. Data needs to be sequestered and accessible to only it’s owner. This blunder is common enough that it’s known as “failure to restrict URL access to data” and is number eight on the OWASP list of top ten web site vulnerabilities.

What can we learn from Citigroups experience?

One security expert familiar with the investigation wondered how the hackers could have known to breach security by focusing on the vulnerability in the browser. “It would have been hard to prepare for this type of vulnerability,” he said.

While I understand that Citigroup needs to protect itself by not revealing too much, this security probably should have worded the vulnerability differently. It is quite possible to prepare for this type of vulnerability, but it requires a significant amount of testing throughout the web-based application for this specific vulnerability. Each and every page needs to be tested for cross-customer leakiness, and that takes a considerable amount of time. Believe me, it’s time well spent.

The expertise behind the attack, according to law enforcement officials and security experts, is a sign of what is likely to be a wave of more and more sophisticated breaches by high-tech thieves hungry for credit card numbers and other confidential information.

The fact that this vulnerability was exploited using a web scraper shows how long this type of vulnerability has been around. Most web developers, including myself, have built a script to scrape a web site at some point in their career. Before APIs were as ubiquitous as they are now, this is how we often pulled data from a web site (with the site owner’s permission, of course.) This breach exploited one of the most common vulnerabilities using web scraping techniques, both of which have been around since the web came into existence. These hackers have been around since the old school days, which makes them all the more sophisticated and dangerous.

A good web development team will know how to build safeguards into their web-based applications to protect against the most common vulnerabilities. It is a well known fact that most web site breaches exploit vulnerabilities in the web-based application itself (i.e. SQL injection, cross site scripting). But, even the best web developers are prone to overlook these vulnerabilities and should have their applications tested thoroughly by a QA team trained in finding them. And if the web-based application is going to be handling personal data of a sensitive nature, a security scan should be performed by both a security expert and an automated scanning process.

As web developers we are often entrusted with building web-based applications that will handle sensitive data for multiple customers. Our responsibility to our clients is to educate our clients about online application security and convince them to allot enough budget to develop and deliver a product that puts security first. Often times these exploits can be traced back to a web development process that was rushed or under-funded. The responsibility should not fall entirely on the web developer given the constraints a client will often impose. Ultimately, it’s the client’s responsibility to make security a priority and hire qualified web developers. If web developers and their clients don’t discuss security during a project, were just going to see more and more breaches like this one.

What can you do to protect your web site or web-based application from security breaches?

  • Review the OWASP Top Ten Application Security Risks and thoroughly test your web-based application for each vulnerability.
  • Hire a security expert to carefully review your online applications. Then task a web developer with working alongside the security expert and implementing their findings.
  • Monitor your server logs for any unusual activity. If you are not sure what to look for in the server logs, sign up with a third-party service, like Scout, that can monitor the logs for you.
  • Do regular vulnerability scans of your web sites and online applications. Again, there are third-party services, like McAfee SECURE, that can do this for you. If you handle credit card data, your processing company will require you to do this (if they haven’t already).
  • If you are handling credit card data, consider using a third-party company, like PayPal, with the tools and services to help secure credit card and other information using their web services so you don’t have to.

Photo credit: pylbug 

Tags: , , ,
| 1 Comment »

Clever release names for creative developers

June 8th, 2011 by John Reeve
Ubuntu Oneiric OcelotAndroid Honeycomb
OS X Snow Leopard

Ubuntu and its alliterative, animal themes, are the first thing that comes to our minds when we think of noticeable nomenclatures used by developers — in place of boring, old-fashioned numbers — to label significant releases. More recently they’ve released Lucid Lynx, Maverick Meerkat, and Natty Narwhal. As I write this they are working on the Oneiric Ocelot release.

Other developers have joined in the fun as well. Android has made headlines with Froyo, Gingerbread, and Honeycomb, and has announced the next major version will be dubbed Ice Cream Sandwich. Apple is running through the list their list of furry Felidae with Tiger, Leopard and Snow Leopard. The list of lists goes on, as many companies, some memorable, some not (did someone say Chicago?), affix clever code names to their releases.

Sequence-based Identifiers

The answer to life, the universe, and everything...Why replace a perfectly good numbering system with offbeat words? What’s wrong with 10.1, 10.2, 10.3 and so on? It’s boring. That’s what’s wrong. Assigning fun and imaginative names to your releases does two things. One, it gets your customers excited about your next release. It get’s them involved in a way that numbering conventions can’t do. As consumers, we love talking about when Honeycomb is going to mature and take over the tablet world. We anxiously await the release of Apple’s Lion update.

Alt.boring

Does it get much more boring than solitaire?As developers, naming releases grants us collective ownership, and boosts our morale by making everyone part of something. We don’t want to be talking about releases in numbers, they’re too corporate-esque. We want to speak in code, with intrigue, over lunch or drinks at happy hour. And when the launch arrives, the quirky release names transmogrify into even quirkier release parties, complete with themed DIY decorations and music. Alt.boring becomes alt.fun.

Creating our own theme for release names

This is what release cycles can begin to feel like when they're too long...The team here at Intervals recently decided we, too, were through with using boring old numbers to internally keep track of our releases. Numerical releases don’t lend themselves well to web-based software because of our agile and iterative development cycle. We are constantly pushing up new code, tweaking and fine tuning our online project management software. We needed something inspiring, nostalgic, and entertaining, before our brains became card catalogs filled with decimal-pocked releases.

Sometimes, it’s all in a name

8 bit Nintendo Entertainment System

Giving our releases more relevant and personable names would give us more wiggle room, more leeway in pushing up iterative changes after a major release. That’s why we came up with a naming convention for major releases based on our collective childhood experiences with console video game systems. Given the average age of our team, there was only one console that fit our demographic… the infamous 8-bit video game console known as the Nintendo Entertainment System (NES). Yeah, the one with the big gray cartridges that you had to blow on to clear the dust from the inside connector, the video console that came with a gun that neither parents nor media seemed to notice, and what was up with balancing gyros on that useless robot’s arms? That NES.

That useless robot has a name… it’s R.O.B.!

NES R.O.B.But don’t expect any upcoming releases from us named Gyromite. The NES had plenty more games available to play, more than I had realized, more than enough to creatively label each Intervals release. Everyone here has bought in to the NES theme and we’ve already gone through a few releases with the new naming convention. We started with Contra, then moved on to Double Dragon, and now we’re working on our Excitebike release. Unfortunately, it’s going to be a while before we get to my personal favorite, Metroid.

I suppose now we’ll have to go out and get us some NES belt buckles ;)

NES Release Names

Tags: , , , ,
| 5 Comments »

Four Ways Web Design and Development Agencies Can Give Back

May 26th, 2011 by John Reeve

Four ways web designers and developers can give back

Web design and development agencies are typically small businesses where the creative and engineering types flourish in a loosely structured company culture. The few formalities us web designers and developers must comply with usually revolve around deadlines, workflow, meetings, and clients. Overall, it’s not a bad gig, it’s the type of environment where web designers and developers churn out their best work, where they are driven, but not corralled. One side effect of this form of a company culture is that our minds often have idle time to spare, where creative ideas are trying to break in to our cerebral cortexes and problems of logical nature are trying to find their way out. Giving your staff an opportunity to give back charitably is a win-win and keeps the minds of creatives and engineers from stagnating. So where do you start?

1. Money

Giving money to a non-profit organization is the easiest way to give back. It’s as simple as writing a check. Plus, it’s tax deductible. There are plenty of ways to give money. One is to simply write a check from the company coffers. Another is to incentivize staff to give by matching their donations to an organization of their choice. Get creative with it. Not sure who you want to donate to? There are organizations out there, like One Percent for the Planet, that will help match you up with a non-profit organization. And once you start giving money to a an organization, you will quickly find there are plenty of other ways to get involved.

2. Time

Time, in contrast to money, is one of the most difficult things we can give to a non-profit organization. Our schedules are compressed, our responsibilities spread too thin. But there are plenty of ways to get around such barriers and donate our time, even if it is only a few hours a year. As a company we can offer as little as a few days off for staff to volunteer at a local charity, or as much as a few days off to go on a long bike ride to raise awareness for a good cause. Non-profit organizations need people to help as much as they need money. We can serve food at a soup kitchen, clean up the beach, or simply man a booth at a fair. The volunteering options become limitless when you offer to give your time.

3. Resources

We are experts in web design and development, why not offer our expertise in these areas to help out our favorite non-profit organizations? Giving of our resources and expertise requires less than even money and time, because it’s something we are used to doing every day. We spend our entire work week designing and developing web sites, how hard would it be to knock out a cool design for a local charity? Not hard at all. A great way to get involved is to simply call up a non-profit organization and find out if they have any use for your expertise (trust me, they will jump at the opportunity). We can help by designing web sites, developing small applications, donating unused space on our servers to host a web site. Get creative, think of something. We can create just about anything online we want to, and we can just as easily put those creations do good use.

4. Stuff

Web designers and developers work a lot with computers. As a result, we have a lot of computers laying around, many of them defunct and taking up space. There are plenty of non-profit organizations that will accept our discarded stuff. Whether it’s computers, desks, chairs, or leftover office supplies, there are people who could really use that stuff, and they don’t need or want the latest and greatest, just something that works.

Why?

Because giving back feels good. Because it’s better for everyone involved. Because it makes the world a slightly better place. Because one day we’re going to look back on our lives, and are we really going to wonder, gosh, I wish I had worked more and given less?

Photo credit: opensourceway

Tags: , , , ,
| 1 Comment »
« Previous Entries
Next Entries »