Makefile – Get Compute Engine Instance IP

I have a test that requires me to spin up a VM with a service running on port 80. I’d need to test if that service is working, but the machine itself is irrelevant, so I’m not bothering to give it a hostname. But I still need to test output on port 80. So I need the external IP address of the machine and nothing else.

Enter the dynamic data trick from my first Makefile post – again.  

So I grab the value and dump it to a variable:

TESTIP = $(shell gcloud compute instances describe $() --format='value[terminator=""](networkInterfaces[0].accessConfigs[0].natIP)')

Nothing more to it.

Makefile – Start, Stop or Delete 20 VMs at once

Last time I created 20 virtual machines at once.  Now I want to stop those machines, or start them back up, or delete them. Basically, I want to do bulk operations on all of the machines that I am using in this scenario.

If you look at the create 20 VMs post, I gave each one of them a similar name, based on the pattern “load-xxx” where load is the operation I am using them for and xxx is a three digit sequential id with 0s prefixed. (This makes them order correctly in our UI.)

Because I know their names, I can count them up and not have to explicitly tell these operations how many machines I want to operate on.  To do that, I create a make variable that contains the count of all VMs prefixed by “load.”

COUNT = $$(( $(shell gcloud compute instances list | grep 'load-' | wc -l | xargs) ))

Once I have that, I can perform batch operations very simply.

To stop 20 running VMs:

    @echo "Initiate Stop loop"
    @i=1 ; 
    while [[ $$i -le $(COUNT) ]] ; 
        do server=`printf "load-%03d" $$i` ; 
        ($(call stop-node,$$server) &) ; 
        ((i = i + 1)) ; 
define stop-node
   echo "Stop Compute Engine Instance - " $(1) ; 
   (gcloud compute instances stop $(1) ) 

Just to explain, like the previous post, we loop from i to COUNT, creating a variable that contains the name of our server, and running a function call to execute the gcloud stop instances command.  Why is this a separate function?  Because I usually do more than just stop the VM.

I also wrap the call in parentheses and append the & to allow multiple calls to execute in parallel.

To start them back up:

    @echo "Initiate Start loop"
    @i=1 ; 
    while [[ $$i -le $(COUNT) ]] ; 
        do server=`printf "load-%03d" $$i` ; 
        ($(call start-node,$$server) &) ; 
        ((i = i + 1)) ; 
define start-node
   echo "Start Compute Engine Instance - " $(1) ; 
   (gcloud compute instances start $(1))      

To delete them all:

    @echo "Initiate Delete loop"
    @i=1 ; 
    while [[ $$i -le $(COUNT) ]] ; 
        do server=`printf "load-%03d" $$i` ; 
        ($(call delete-node,$$server) &) ; 
        ((i = i + 1)) ; 
define delete-node
   echo "DELETE Compute Engine Instance - " $(1) ;
    (gcloud compute instances delete $(1) --delete-disks "all" --quiet )

And in this case, I do a little bit more here in delete.  I make sure all of the disks are deleted, and I set the request to quiet. Why? Because I don’t want to confirm this 20 times, silly.

In any case, doing batch operations on my set of VMs is as easy as:

make start
make stop
make delete

There you have it, fleets of VMs responding in concert to your requests.  As it should be.

Makefile – Launch 20 Compute Engine virtual machines at once.

We’re going to try something a lot more complex in make now. I’m going to dynamically create 20 Compute Engine virtual machines that are absolutely the same. This requires quite a bit more complexity, so we’ll break it down step by step.

Let’s start with the gcloud command to create an instance.  

define create-node
   echo "Create Compute Engine Instance - " $(1) ;
   (gcloud compute instances create $(1) --machine-type "f1-micro" ;
   gcloud compute ssh $(1) --command "sudo apt-get update" ;
   say "$(1) is done.")

I encapsulated this into a Makefile function. Why?  Well, as I have it here, it is a pretty simple event with adding apt-get update but I usually do more then just create the node and install software. I often set environmental information or start services, etc. So by putting all of the instance specific instructions in a function, I make it just slightly easier to grok.

Let’s go through this part step by step.  

  • Define a function with the define keyword, and end it with the endef keyword
  • It appears that functions must be one line, so use ; to organize multiple calls into one function
  • Wrap all of the real work in a parenthesis. Why? It turns it into one operation, so that each step of the function doesn’t block parallel execution of other operations in the makefile.
  • Capture the first argument – $(1) – passed into this function – we’ll use it as the name of the instance
  • Create a machine using gcloud compute instances create. Note setting the machine type.  If you are creating a lot of instances, make sure you don’t run afoul of quota or spend.
  • SSH into machine and run apt-get update.
  • Tell us this machine is ready.   

Okay, that handles the instance creation, but now we have to loop through and create a variable amount of machines. I said 20, but I often spin up anywhere from 10 to 150 using this method.

    @echo "Initiate Create loop"
    @i=1 ; 
    while [[ $$i -le $(count) ]] ; 
        do server=`printf "load-%03d" $$i` ; 
        ($(call create-node,$$server) &) ; 
        ((i++)) ; 

Again, step by step:

  • Use @ so that the commands aren’t echoed to the output.
  • Set up a while loop with iterator – i, that will run as long as i is less than the explicitly passed variable named count
  • Use ; to make the command one logical line.
  • Use printf to create a variable named server to name the instances. In this case each instance is named “load-xxx” where xxx is a sequential id number for the node that always has three digits. This makes it easier to go back later and do more group operations on the entire set of machines. 
  • Call the function using the syntax $(call function_namevalue_to_pass)
  • Wrap call in parentheses and append a &.  This shoves the call to the background so you can create 20, or 100, or 150 of these in parallel instead of sequentially.
  • We then increment the counter.   

Finally we call the whole thing with:

make create count=20

Pretty straightforward. I frequently use this technique to launch of fleet of VMs to send large amounts of load at App Engine. Next I’ll tell you how to delete them all.   

Don’t forget the count=N, or the call will bail.

Makefile – Tell me when you are done

I have been doing a large number of tasks lately that involve executing long-running processes from a Makefile — maybe somewhere in the neighborhood of 40 seconds to 5 minutes.  They’re just long enough that I get bored and go off and do something else, but short and urgent enough that I would really want to do something (usually manually test) right after the process is done. I need to not drift off into procrastination world.  I need to be alerted when my process completes.

I have taken to adding a ‘say’ command to my Makefiles.  If you aren’t familiar, on OS X ‘say’ will have the computer speak out whatever you have written using the Text-to-Speech system setup in the OS.  So that way, when I am looking in a browser window being distracted, a disembodied voice can startle me out of my reverie and I can jump right back in as soon as possible.

I usually do something like this:

clean :
	gcloud container clusters delete $(CLUSTER) -q
	gsutil -m rm -r gs://artifacts.$(PROJECT)
	gcloud compute --project $(PROJECT) addresses delete "ip1"
	gcloud compute --project $(PROJECT) addresses delete "ip2" 
	gcloud compute --project $(PROJECT) addresses delete "ip3"
burnitdown: clean
	say “Project $(PROJECT) has been burnt to the ground.”



As always, your command names and mileage may vary. And turn down your volume.  Or don’t. 

Makefile – Clean App Engine flexible environment

One of the more interesting quirks of App Engine flexible environment is that App Engine launches Compute Engine virtual machines that you can’t spin down directly. The way to spin down App Engine flex is to delete all versions of the app.  This will close down all of the VMs, and shut down your App Engine app.

You can do it manually through the web interface, you can do it manually by listing versions in gcloud then deleting them, or you can have a Makefile do it for you.

First I use the trick I wrote about capturing dynamic data from gcloud. Then I pipe that to a Makefile command that will delete the versions.

VERSIONLIST = $(shell gcloud app versions list --format='value[terminator=" "](')
	gcloud app versions stop $(VERSIONLIST) -q

Note that I add -q to the command because I don’t want to be prompted; I just want them gone.

Makefile – Delete Forwarding Rules

I have a demo where I build a Kubernetes cluster on Container Engine to run a LAMP app. In the demo, I script out a complete build process from an empty project to the full running app.  Testing this requires a clean up that takes me all the way back to an empty project with no cluster.  

There is one thing I do not tear down – static IP addresses.  I don’t tear these down because they are locked to host names in Google Domains, and I use those IPs in my Kubernetes setup to make sure that my cluster app is available at a nice URL and not just a randomly assigned IP.

But I have been running into a problem with this. Sometimes the static IPs hold on to Forwarding Rules that are autogenerated with crazy randomized names by Container Engine. It appears to  happen only when I do a full clean.  I suspect that I am deleting the cluster before it has a chance to issue the command to delete the forwarding rules itself.

In any case, I got tired of dealing with this manually, so I made a Makefile solution. First I get the dynamic list of crazy random forwarding rule names using the Makefile technique I outlined earlier.  Then I pass that list to a gcloud command:

RULELIST = $(shell gcloud compute forwarding-rules list --format='value[terminator=" "](name)')
	-gcloud compute forwarding-rules delete $(RULELIST) --region $(REGION) -q

Note that I had to make sure I passed a region, otherwise the command would have prompted me to enter it manually.

Makefile – Get dynamic values from gcloud

Most of the time when I create something in my environment on Google Cloud Platform, I give it a specific name.  For example, I create servers and call them “ThingIWillReferenceLaterWhenIDeleteYou” or more boringly, “Server1.”

Having set names, as I alluded to, makes it easier to clean up after yourself. But there are some cases when you cannot name things when they are created. So it would be nice to get a list of these names. For example, App Engine flexible environment versions for cleaning up after a test.

You can get a list of them with this command:

gcloud app versions list

Which yields this:

default  20170405t215321  0.00           2017-04-05T21:53:39-07:00  STOPPED
default  20170405t222022  0.00           2017-04-05T22:20:40-07:00  STOPPED
default  20170405t223620  0.00           2017-04-05T22:36:38-07:00  STOPPED
default  20170405t230438  0.00           2017-04-05T23:04:59-07:00  STOPPED
default  20170405t235759  0.00           2017-04-05T23:58:27-07:00  STOPPED
default  20170407t102935  0.00           2017-04-07T10:29:55-07:00  STOPPED
default  20170407t110623  1.00           2017-04-07T11:06:45-07:00  STOPPED

Now normally I would have to add extra code to my Makefile to rip out the version names.

But gcloud actually has a robust formatting tool. So instead of running the command above I can run:

gcloud app versions list --format='json'

And get the JSON representation, which looks like this:

    "environment": {
      "FLEX": null,
      "MANAGED_VMS": {
        "FLEX": null,
        "MANAGED_VMS": null,
        "STANDARD": {
          "FLEX": null,
          "MANAGED_VMS": null,
          "STANDARD": null,
          "name": "STANDARD",
          "value": 1
        "name": "MANAGED_VMS",
        "value": 2
      "STANDARD": {
        "FLEX": null,
        "MANAGED_VMS": {
          "FLEX": null,
          "MANAGED_VMS": null,
          "STANDARD": null,
          "name": "MANAGED_VMS",
          "value": 2
        "STANDARD": null,
        "name": "STANDARD",
        "value": 1
      "name": "FLEX",
      "value": 3
    "id": "20170405t215321",
    "last_deployed_time": {
      "datetime": "2017-04-05 21:53:39-07:00",
      "day": 5,
      "hour": 21,
      "microsecond": 0,
      "minute": 53,
      "month": 4,
      "second": 39,
      "year": 2017
    "project": "redacted",
    "service": "default",
    "traffic_split": 0.0,
    "version": {
      "automaticScaling": {
        "coolDownPeriod": "120s",
        "cpuUtilization": {
          "targetUtilization": 0.5
        "maxTotalInstances": 20,
        "minTotalInstances": 2
      "betaSettings": {
        "cloud_sql_instances": "redacted:us-central1:redacted",
        "has_docker_image": "true",
        "module_yaml_path": "app.yaml",
        "no_appserver_affinity": "true",
        "use_deployment_manager": "true"
      "createTime": "2017-04-06T04:53:39Z",
      "createdBy": "",
      "env": "flexible",
      "id": "20170405t215321",
      "name": "apps/redacted/services/default/versions/20170405t215321",
      "runtime": "php",
      "servingStatus": "STOPPED",
      "threadsafe": true,
      "versionUrl": ""

Using a JSON parser might make this easier, but there is an even easier way:

cloud app versions list --format='value[terminator=" "]('

Which yields:

20170405t215321 20170405t222022 20170405t223620 20170405t230438 20170405t235759 20170407t102935 20170407t110623

What will this do?

It will list just the value of and it will separate each record it returns with a ” “, not a line break.  This allows me to drop this generated list into any command that takes multiple names and run them. The gcloud CLI takes multiple arguments in this way. 

So to make this applicable to Makefiles I have to do one more thing – take this data and put it in a variable.

VERSIONLIST = $(shell gcloud app versions list --format='value[terminator=" "](')

Here we are, ready to use this variable in other Make commands. This works for most of the other places in GCP where you see random values spitting out, like IP forwarding rules, and GKE nodes, to name two. 

To learn more about how to filter and format your gcloud commands, check out the Google Cloud Platform Blog.


Makefile – quick series

Before joining Google Cloud, I wasn’t programming as much as I used to.  So when I joined up the last build system I had used with any regularity was ANT.  Upon getting back in the routine of programming, I started down that path again, and immediately stopped.  I did not want to deal with ANT and XML when I started back up.  I also wasn’t doing anything even tangentially related to Java. So no, I’m not using ANT. I stopped doing single build files altogether and settled for folders of bash scripts.

This was… unsustainable.

Enter Mark Mandel and his constant exhortations to use Makefiles.  Eventually I listened to him, and now instead of folders of scripts, or line after line of XML, I have giant Makefiles.

Make is awesome.  And I know it is for more than just pushing files around, but that’s what I use it for.  And I love it.  

I’m running a short series on a number of productivity tips and tricks I’ve learned.  Many will be about Google Cloud. Some will not. I hope these help someone else learn to love Makefiles.