Makefile – Launch 20 Compute Engine virtual machines at once.

We’re going to try something a lot more complex in make now. I’m going to dynamically create 20 Compute Engine virtual machines that are absolutely the same. This requires quite a bit more complexity, so we’ll break it down step by step.

Let’s start with the gcloud command to create an instance.  

define create-node
   echo "Create Compute Engine Instance - " $(1) ;
   (gcloud compute instances create $(1) --machine-type "f1-micro" ;
   gcloud compute ssh $(1) --command "sudo apt-get update" ;
   say "$(1) is done.")
endef

I encapsulated this into a Makefile function. Why?  Well, as I have it here, it is a pretty simple event with adding apt-get update but I usually do more then just create the node and install software. I often set environmental information or start services, etc. So by putting all of the instance specific instructions in a function, I make it just slightly easier to grok.

Let’s go through this part step by step.  

• Define a function with the define keyword, and end it with the endef keyword
• It appears that functions must be one line, so use ; to organize multiple calls into one function
• Wrap all of the real work in a parenthesis. Why? It turns it into one operation, so that each step of the function doesn’t block parallel execution of other operations in the makefile.
• Capture the first argument – $(1) – passed into this function – we’ll use it as the name of the instance
• Create a machine using gcloud compute instances create. Note setting the machine type.  If you are creating a lot of instances, make sure you don’t run afoul of quota or spend.
• SSH into machine and run apt-get update.
• Tell us this machine is ready.   

Okay, that handles the instance creation, but now we have to loop through and create a variable amount of machines. I said 20, but I often spin up anywhere from 10 to 150 using this method.

create: 
    @echo "Initiate Create loop"
    @i=1 ; 
    while [[ $$i -le $(count) ]] ; 
        do server=`printf "load-%03d" $$i` ; 
        ($(call create-node,$$server) &) ; 
        ((i++)) ; 
    done

Again, step by step:

• Use @ so that the commands aren’t echoed to the output.
• Set up a while loop with iterator – i, that will run as long as i is less than the explicitly passed variable named count
• Use ; to make the command one logical line.
• Use printf to create a variable named server to name the instances. In this case each instance is named “load-xxx” where xxx is a sequential id number for the node that always has three digits. This makes it easier to go back later and do more group operations on the entire set of machines. 
• Call the function using the syntax $(call function_name, value_to_pass)
• Wrap call in parentheses and append a &.  This shoves the call to the background so you can create 20, or 100, or 150 of these in parallel instead of sequentially.
• We then increment the counter.   

Finally we call the whole thing with:

make create count=20

Pretty straightforward. I frequently use this technique to launch of fleet of VMs to send large amounts of load at App Engine. Next I’ll tell you how to delete them all.   

Don’t forget the count=N, or the call will bail.