Running a ColdFusion Shop 2

I had more to say about running a ColdFusion shop. Hopefully I’m not overstaying my welcome on the topic.

Developers are your Users

This might be an odd bit of information, and you shouldn’t forget about the users of the applications, but developers are your users. This has a few different implications:

Development servers are production servers as far as your developers are concerned. I learned this the hard way a few years back when we lost our production environment due to a compromise. (Yeah, I had a server or two hacked, don’t judge me.) We only had a single node production environment. Part of our recovery plan was, in case of emergency, use the staging environment as the production environment. This could work for a day or two, but day three came around and the complaining started. Perhaps justified, but when you are trying to get a new production environment up, you don’t want to hear “we need to work.” In the end, you want to make sure that the development servers stay up during regular working hours.

Users have their own definition of what is and is not your responsibility. “What’s that you say? You deleted your entire directory, and want me to run a backup?” Developers will eventually either accidently wipe out files, or go down a bad path with their code and want a restore. We’ve all done it, but as an administrator you’re going to want to ensure that their problems don’t become yours. Make sure they have access to a source control product. Encourage them to use it. If you’re running on Windows enable Shadow Copy. If you do regular tape backups try and have a local backup that you can easily restore from. The idea here is that at some point you will be asked to run a restore, you want to make sure that you never really have to.

Don’t be an asshole about security. I know I just said something about being hacked, but bear with me. The first of Microsoft’s Immutable Laws of Security is:

If a bad guy can persuade you to run his program on your computer, it’s not your computer anymore.

Your developers can run code on your server; by definition you are already compromised. Don’t lock down tags. Don’t prevent object creation. A smart annoyed developer will get around you. The administrator is pretty easy to get around if you can do any sort of file manipulation. Even if this doesn’t work, there are many ways a developer can outsmart you if you are unnecessarily restrictive.

Obviously this is for groups of developers that work for the same company. I hope hosting companies don’t have this attitude. Additionally, don’t ignore external security.

Be open about your standards. A few years ago people left every code review we had saying, “Why didn’t anyone tell us about these new standards?” The reason was that the central group of administrators trolled the blogs, read articles and paid attention to what was the newest recommendations. We would talk about them, and come up with new guidelines. Problem was that we didn’t talk to the rest of the group until code reviews came up. My group started posting our standards on our Developer’s blog, and now people don’t leave code reviews feeling like they’ve been ambushed.

That’s it for now, I’m sure I remember something afterwards which can wait for an eventual part 3.

Running A ColdFusion Shop

This is a post I’ve been mulling for awhile and since my office is so cold that I can’t concentrate, I figured I would give it a shot.

In addition to writing applications in ColdFusion I’m also responsible for maintaining our environment. Our environment consists of 20 or so full time developers, with another 10 to 20 or so part time developers. Our code lives on around 20 servers, which are split up according to different versions and purposes. We run sub-environments for ColdFusion 6, ColdFusion 7 and Flex 1.5 (which is a much different animal then Flex 2.0.) The developer’s group as a whole maintains about 100 or so applications. These applications range in size from a simple one form troubleshooting reporting tool to our student portal that serves our entire student body, and won a Max Award for Education Experiences back in 2004.

I figured I would share some of my observations about what has made our environment more stable over the past three years. Maybe it can help; maybe I’m wrong and need correcting.

Develop, Stage, and Production

Break up your environment into a development area, a staging area, and a production area. We do this by having two servers per sub-environment: development/staging and production. We host development and staging on the same hardware, and use separate ColdFusion instances to separate staging from development. If you can’t make the investment in a separate server for production, then run it as a third instance.

Load Balance If You Can

Most of our production servers are actually load balanced clusters and not stand alone boxes. This means that patches that require a reboot don’t have to mean downtime for you users. You just “drainstop” to one node of the cluster. Update the deactivated node. Reboot and repeat. Being able to patch production servers during the day on Microsoft Patch Tuesday when there are Zero Day exploits? Priceless.

Write Code that Knows Where It Is

We have a couple of shared modules that allow code to determine if it is in development, staging, or production. It does this through a combination of file structure, CGI variables and registry information. Then variables can be set accordingly. That way, you don’t have to code around application state.

Don’t Produce, Publish

Don’t allow anyone, including your fellow administrators, to directly touch code in the production environment. Code should be published from staging. It doesn’t matter how. On most of our sub-environments we use a centrally available system that allows developers to push code to production. On one of our sub-environments it’s just a scheduled task that copies staging over to production every N minutes. This way developers, including yourself are forced to test in staging, before users start seeing errors due to a bad update. It also obfuscates more details about the production environment from our developers. If we add a cluster node to server, it doesn’t matter to the developers; publishing just takes care of it.

Share the Wealth

We do this three ways:

First, we write boilerplates that define a very loose framework than makes sure all applications do certain things “right.” Things like forcing login information to be posted over SSL, and encouraging the use of central code nuggets when proper.

Then we write central code nuggets, like encapsulated and logged LDAP calls, or authentication routines. We solve these problems once, and make sure everyone else can benefit.

Finally, we have our own Developer’s blog that we maintain for sharing internal information, and for posting announcements. We tied in publishing, so our developers have to look at it from time to time.

Tell Everyone How You Really Feel About Them

They’re painful, boring, cathartic, and frustrating all at the same time. Other people have written more about them ad nauseum. So I’m going to just say “No matter how painful they can be, you really need to do code reviews.” We typically make code review a condition of being allowed to publish to production.

Miscellaneous

Patch your machines. That’s all I have for now. Maybe I will have more later.