Running A ColdFusion Shop

This is a post I’ve been mulling for awhile and since my office is so cold that I can’t concentrate, I figured I would give it a shot.

In addition to writing applications in ColdFusion I’m also responsible for maintaining our environment. Our environment consists of 20 or so full time developers, with another 10 to 20 or so part time developers. Our code lives on around 20 servers, which are split up according to different versions and purposes. We run sub-environments for ColdFusion 6, ColdFusion 7 and Flex 1.5 (which is a much different animal then Flex 2.0.) The developer’s group as a whole maintains about 100 or so applications. These applications range in size from a simple one form troubleshooting reporting tool to our student portal that serves our entire student body, and won a Max Award for Education Experiences back in 2004.

I figured I would share some of my observations about what has made our environment more stable over the past three years. Maybe it can help; maybe I’m wrong and need correcting.

Develop, Stage, and Production

Break up your environment into a development area, a staging area, and a production area. We do this by having two servers per sub-environment: development/staging and production. We host development and staging on the same hardware, and use separate ColdFusion instances to separate staging from development. If you can’t make the investment in a separate server for production, then run it as a third instance.

Load Balance If You Can

Most of our production servers are actually load balanced clusters and not stand alone boxes. This means that patches that require a reboot don’t have to mean downtime for you users. You just “drainstop” to one node of the cluster. Update the deactivated node. Reboot and repeat. Being able to patch production servers during the day on Microsoft Patch Tuesday when there are Zero Day exploits? Priceless.

Write Code that Knows Where It Is

We have a couple of shared modules that allow code to determine if it is in development, staging, or production. It does this through a combination of file structure, CGI variables and registry information. Then variables can be set accordingly. That way, you don’t have to code around application state.

Don’t Produce, Publish

Don’t allow anyone, including your fellow administrators, to directly touch code in the production environment. Code should be published from staging. It doesn’t matter how. On most of our sub-environments we use a centrally available system that allows developers to push code to production. On one of our sub-environments it’s just a scheduled task that copies staging over to production every N minutes. This way developers, including yourself are forced to test in staging, before users start seeing errors due to a bad update. It also obfuscates more details about the production environment from our developers. If we add a cluster node to server, it doesn’t matter to the developers; publishing just takes care of it.

Share the Wealth

We do this three ways:

First, we write boilerplates that define a very loose framework than makes sure all applications do certain things “right.” Things like forcing login information to be posted over SSL, and encouraging the use of central code nuggets when proper.

Then we write central code nuggets, like encapsulated and logged LDAP calls, or authentication routines. We solve these problems once, and make sure everyone else can benefit.

Finally, we have our own Developer’s blog that we maintain for sharing internal information, and for posting announcements. We tied in publishing, so our developers have to look at it from time to time.

Tell Everyone How You Really Feel About Them

They’re painful, boring, cathartic, and frustrating all at the same time. Other people have written more about them ad nauseum. So I’m going to just say “No matter how painful they can be, you really need to do code reviews.” We typically make code review a condition of being allowed to publish to production.

Miscellaneous

Patch your machines. That’s all I have for now. Maybe I will have more later.