Makefile – Clean App Engine flexible environment

One of the more interesting quirks of App Engine flexible environment is that App Engine launches Compute Engine virtual machines that you can’t spin down directly. The way to spin down App Engine flex is to delete all versions of the app.  This will close down all of the VMs, and shut down your App Engine app.

You can do it manually through the web interface, you can do it manually by listing versions in gcloud then deleting them, or you can have a Makefile do it for you.

First I use the trick I wrote about capturing dynamic data from gcloud. Then I pipe that to a Makefile command that will delete the versions.

VERSIONLIST = $(shell gcloud app versions list --format='value[terminator=" "](version.id)')
 
clean: 
	gcloud app versions stop $(VERSIONLIST) -q

Note that I add -q to the command because I don’t want to be prompted; I just want them gone.

Migrating App Engine Standard to Cloud SQL v2

I recently discovered that Google Cloud SQL v2 now supports App Engine standard runtime.  This is very exciting for me. I wanted to try out the process and make sure there were no gotchas.

GAE+SQL

  • I created a new Cloud SQL v2 instance.
  • I used my syncing script from my blog post Migrating between Cloud SQL databases to move the data to the new instance.
  • I created a new App Engine module by store a new version of my app using the old code base.
  • I changed the connection string from the old database to the new one. The pattern to make this happen has changed a bit, more down below.
  • That was it.  The new code served up just fine. I served up on the old module until I made the connection string config tweak to the old code base.

New connection string

The new connection strings are only a little different than the old ones, and should require just a change to one string in your config.

The directions will tell you to look for your Instance connection name in the Instance properties of your Cloud SQL Developer Console. There are two patterns that these strings come in. 

  • V1 connection names follow the pattern projectid:instancename
  • V2 connection names follow the pattern projectid:regionname:instancename.

It’s a pretty simple change, but I can see someone accidentally (or willfully) not reading the documentation and getting tripped up on this. The new connection strings require region name; that’s all there is to it.  I’ve tested this on PHP. I assume it works everywhere. But your mileage may vary.  Golang tests are coming soon; I will update when I make that change.

Working with Cloud Vision API from PHP

I have been very excited by the Cloud Vision API recently put into Beta by Google Cloud Platform. I haven’t had a chance to play with it much, and I wanted to fool around with it from PHP on App Engine (or vanilla PHP for that matter), but there is no documentation for PHP yet.

So here is a little primer on how to do this from PHP on App Engine.

First you have to complete a few prerequisites:

Once you do this you’re ready to start developing. Because I am running PHP on App Engine I want the App Engine SDK for PHP.

I’m going to use the GUI to run this app, but you can use the command line just as easily.

cloud-vision-php-gaelauncher

The first thing I need to do is write a php.ini that properly allows use of cURL and has a good limit on uploaded files.

google_app_engine.enable_curl_lite = 1
upload_max_filesize = 5M

Then I set up a page named creds.php to hold my API key for Cloud Storage and my Cloud Storage Bucket name.

<?php 
//Create Bucket here 
// https://cloud.google.com/storage/docs/getting-started-console#create_a_bucket
$bucket = "YOUR BUCKET HERE";
// Get Service account API hereL 
// https://cloud.google.com/vision/docs/getting-started#setting_up_a_service_account
$api_key = "YOUR API KEY HERE ";

 ?>

Then I create a form page named index.php that creates an App Engine Upload URL for me. (If I wanted to not use App Engine, I could just skip the call to Cloud Storage Tools and post directly to the next file in the example: process.php.)

<?php
include_once("creds.php"); // Get $bucket
use googleappengineapicloud_storageCloudStorageTools;

$options = [ 'gs_bucket_name' => $bucket ];
$upload_url = CloudStorageTools::createUploadUrl('/process.php', $options);

?>

<!DOCTYPE html>
<html lang="en">
<head>
	<meta charset="UTF-8">
	<title>Cloud Vision API PHP Example</title>
</head>
<body>
	<form action="<?php echo $upload_url ?>" method="post" enctype="multipart/form-data">
	Your Photo: <input type="file" name="photo" size="25" />
	<input type="submit" name="submit" value="Submit" />
</form>
</body>
</html>

Then process.php does the hard work of taking the uploaded file, converting it to base64 and uploading to the Cloud Vision API.

<?php

include_once("creds.php"); // Get $api_key
$cvurl = "https://vision.googleapis.com/v1/images:annotate?key=" . $api_key;
$type = "LANDMARK_DETECTION";

//Did they upload a file...
if($_FILES['photo']['name'])
{
	//if no errors...
	if(!$_FILES['photo']['error'])
	{
		$valid_file = true;
		//can't be larger than ~4 MB
		if($_FILES['photo']['size'] > (4024000)) 
		{
			$valid_file = false;
			die('Your file's size is too large.');
		}

		//if the file has passed the test
		if($valid_file)
		{
			//convert it to base64
			$fname = $_FILES['photo']['tmp_name'];
			$data = file_get_contents($fname);
			$base64 = base64_encode($data);
			//Create this JSON
			$r_json ='{
			  	"requests": [
					{
					  "image": {
					    "content":"' . $base64. '"
					  },
					  "features": [
					      {
					      	"type": "' .$type. '",
							"maxResults": 200
					      }
					  ]
					}
				]
			}';

			$curl = curl_init();
			curl_setopt($curl, CURLOPT_URL, $cvurl);
			curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
			curl_setopt($curl, CURLOPT_HTTPHEADER,
				array("Content-type: application/json"));
			curl_setopt($curl, CURLOPT_POST, true);
			curl_setopt($curl, CURLOPT_POSTFIELDS, $r_json);
			$json_response = curl_exec($curl);
			$status = curl_getinfo($curl, CURLINFO_HTTP_CODE);
			curl_close($curl);

			if ( $status != 200 ) {
			    die("Error: $cvurl failed status $status" );
			}

			echo "<pre>";
			echo $json_response;
			echo "</pre>";
		}
	}
	//if there is an error...
	else
	{
		//set that to be the returned message
		echo "Error";
		 die('Drror:  '.$_FILES['photo']['error']);
	}
}
?>

Finally I have to create an app.yaml to serve up the two pages.

module: default
version: 1
api_version: 1
runtime: php55
threadsafe: yes

handlers:

# Needed for static image files

- url: /
  script: index.php

- url: /process.php
  script: process.php

Use GoogleAppEngineLauncher to start your app.

You should get this.

cloud-vision-php-form

Assuming you upload a picture from the top of the Eiffel Tower looking at the Champs de Mars, you’ll get something like this:

et

{
  "responses": [
    {
      "landmarkAnnotations": [
        {
          "mid": "/m/02j81",
          "description": "Champ de Mars",
          "score": 0.81389683,
          "boundingPoly": {
            "vertices": [
              {
                "x": 202,
                "y": 410
              },
              {
                "x": 1967,
                "y": 410
              },
              {
                "x": 1967,
                "y": 1318
              },
              {
                "x": 202,
                "y": 1318
              }
            ]
          },
          "locations": [
            {
              "latLng": {
                "latitude": 48.858249,
                "longitude": 2.294694185256958
              }
            }
          ]
        },
        {
          "mid": "/m/02j81",
          "description": "Paris",
          "score": 0.5426321,
          "boundingPoly": {
            "vertices": [
              {
                "x": 305,
                "y": 412
              },
              {
                "x": 1737,
                "y": 412
              },
              {
                "x": 1737,
                "y": 895
              },
              {
                "x": 305,
                "y": 895
              }
            ]
          },
          "locations": [
            {
              "latLng": {
                "latitude": 48.858546,
                "longitude": 2.3222419999999997
              }
            }
          ]
        },
        {
          "mid": "/g/1tc__sx0",
          "description": "France Eiffel Hotel",
          "score": 0.36458692,
          "boundingPoly": {
            "vertices": [
              {
                "x": 732,
                "y": 394
              },
              {
                "x": 1260,
                "y": 394
              },
              {
                "x": 1260,
                "y": 691
              },
              {
                "x": 732,
                "y": 691
              }
            ]
          },
          "locations": [
            {
              "latLng": {
                "latitude": 48.858362,
                "longitude": 2.294125
              }
            }
          ]
        }
      ]
    }
  ]
}

There you go, bare bones but simple Cloud Vision example in PHP.

If you want to dig deeper into the Cloud Vision API you can

The code for all of this is available on GitHub

Compute Engine and App Engine – a Comparison

STE-032.00_00_30_12.Still001

I want to show you a little demo of how Compute Engine and App Engine work. Both techs have their strengths and weaknesses, and I wanted to make something to showcase them. 

Compute Engine allows you to spin up Virtual Machines (henceforth to be referred to as “VMs” due to the fact that I can’t be bothered to write “irtual” and “achine”.) VMs give you a lot of control over your system. You can run a number of OSes, with variable processor, memory, and disk configurations. You interact with it by configuring a VM through the Developer Console or on the command line. You then SSH into your VM.

App Engine on the other hand just takes code.  You upload it and we run it. No SSH, no machine, just an upload site and a URL. App Engine by default gives you no control over the hardware running the code. The trade off is that we can immediately scale from zero load to any load you muster.

So how do these compare? App Engine scales in milliseconds? What does that look like? Compute Engine starts up in 10s of seconds? What does that mean? This demo shows off how you can build Compute Engine machines vs how fast you can spin up App Engine instances. This isn’t a one-is-better-than-the-other comparison; there are reasons to use both of these techs, and they aren’t mutually exclusive. Let me know what you think.

 

PHP on App Engine General Availability

appenginephpEarlier this week, Google Cloud Platform announced General Availability of PHP on App Engine.  Developers are now free to use App Engine to power their developer experience using…

Oh wait, you were already using PHP on App Engine. And have been doing so for a few months, or years.  What does this announcement mean for you?

The big bullet point here is that Google is taking the “Beta” label off the PHP on App Engine. It’s is now governed by the Service Level Agreement, and Deprecation Policy.

Now I’m not a lawyer, so all the rest of this is subject to, you know, me not being a lawyer, and therefore any interpretation herein, yada yada. You know, check with your lawyery people before taking my word for it. I’m mostly going to just describe these things, and point you to the actual documents.

Service Level Agreement

The SLA sets expectations for how much uptime Google Cloud Platform delivers, and what happens if they let you down.  It puts forth a number of uptime stats they need to hit, and what Google Cloud Platform will do if they do not meet them. It also outlines what you need to do to get compensation.

Read the SLA for more information.

Deprecation Policy

The Deprecation Policy states how long Google Cloud Platform will try and run services covered by the Deprecation Policy after a deprecation announcement, unless there is a very serious reason not to.

Read the Deprecation Policy, contained in section 7, for more information.

Please, read these with your lawyerly people. Provide them with Scotch, the promise of billable hours, and whatever else you need to give your lawyerly people to make them happy (Orphan tears?  I kid, I kid.  Please don’t sue me.)

This is a signal that PHP is joining the list of technologies that you can feel secure to choose Google Cloud Platform to host.

Atlassian Connect add-on Starter Kit for App Engine

atlassian_gcpAtlassian Connect add-ons are extensions written by third-party developers to augment Atlassian’s hosted software. We’d love for the Atlassian Community to host their add-ons with Google Cloud Platform.  To help out, I’m releasing Atlassian Connect Add On Starter Kit for App Engine on github.

Google App Engine is a great environment to get started hosting this sort of solution.

  • You just upload code.
  • We handle scaling for you.
  • App Engine serves up webservices with little to no config.
  • App Engine integrates well with other Google Developer Services

On the Atlassian side, as long as you write your services to respond the way Connect expects,  you can write your add-ons in whatever language or technology stack you want.

I wrote the Atlassian Connect Add On Starter Kit for App Engine to give a working example for getting started on App Engine using each of the supported languages.  Each language folder (Go, Java, Php, Python) contains code to create two applications:

Hello World – basically the Hello World example written by Atlassian with added App Engine configs to show the delta between a vanilla add-on and one configured to run on App Engine.

addon-world
The Hello World. Nothing special.

Language Check – this application scans a JIRA instance for issue titles in languages that differ from the default language, and then labels which language they are.  The idea is that you can then easily triage issues that should be routed to staff who can support that language. It takes advantage of the Google Translate API to accomplish this.

Add-on translating with the default language set to English.
Add-on translating with the default language set to Czech.

Now, all of this isn’t to say you cannot use other technologies in Google Cloud Platform to host your add-on.  You can also host on either Compute Engine or Container Engine. After you configure one of these solutions you can just follow Atlassian’s existing tutorials to create an add-on.

Anyway you want to do it, Google Cloud Platform can help the Atlassian developer community to host their applications.  If you want to write some code and have server configuration and scaling handled by us, App Engine is a good choice.  If you would rather have more control over or flexibility with the systems you run, either Compute Engine or Container Engine can be your answer.

Atlassian Connect Add On Starter Kit for App Engine

Managed VM Not Connecting to gcr.io on OS X

Ran into this issue today fooling around with Docker and Managed VMs on Google Cloud Platform. I was running on Mac OS X, which meant boot2docker was also in the mix. Figured this could help someone else because it baffled me for a bit.

I was trying to start up a standard runtime for python on Managed VM. I was using the following app.yaml:

module: default
runtime: python27
vm: true

api_version: 1
threadsafe: yes

resources:
  cpu: .5
  memory_gb: 1.3

manual_scaling:
  instances: 1

handlers:
- url: .*
  script: main.app

I ran the gcloud command:

gcloud preview app run ./app.yaml

In the midst of the output, this error was buried:

ERROR    2015-05-22 17:21:59,043 containers.py:283] v1 ping attempt failed
with error: Get https://gcr.io/v1/_ping: dial tcp: i/o timeout. If this
private registry supports only HTTP or HTTPS with an unknown CA certificate,
please add `--insecure-registry gcr.io` to the daemon's arguments. In the
case of HTTPS, if you have access to the registry's CA certificate, no need
for the flag; simply place the CA certificate at
/etc/docker/certs.d/gcr.io/ca.crt

What was really frustrating is that this had worked yesterday and I had changed nothing in between.  I tried a whole bunch of things. I fired a whole lot of searches on Google. In my searches I found this thread on github that suggests it’s an issue with some sort of caching in Docker. So I did the following:

  • Launched Virtual Box
  • Right clicked boot2docker-vm
  • Chose Close ->Power Off
  • Launched boot2docker again via Applications->boot2docker
  • Tried again

And now it works.

Hope this helps someone else.

PHP on App Engine Updating to PHP 5.5

A few weeks ago Google Cloud Platform released an update for PHP on App Engine that enabled PHP 5.5 on App Engine. It was all very exciting, and there was a forum post about it any everything.  At the bottom though there is a little note:

After 16th April, 2015 we will begin automatically migrating all applications to the php55 runtime.

You may have also seen emails from  “Google App Engine” reminding you of this.  In those notices it has been changed to:

in approximately 2 weeks we will begin automatically migrating your application over to use the PHP 5.5

If you need an extension on PHP 5.4 you can fill out a form to request one.

So this change is coming, and this post is yet another heads up to remind people.

Lumen on App Engine

lumen

Laravel announced today that they are launching a new PHP framework named Lumen for building API’s and microservices.  It has an emphasis on speed and it is compatible with a subset of Laravel, making for easy migrations to the larger framework.

I wanted to see how easy it would be to run a Lumen app on App Engine, and I fooled around with it a bit. You just do a simple setting in app.yaml to route all of the apps traffic to the default handler for the Lumen app:

application: [your application id]
version: one
runtime: php55
api_version: 1

handlers:

- url: /.*
  script: /[path_to_lumen_folder]/public/index.php

With that I got a basic instance running. I did run into one problem though.  The logger tried to write to disk, which is a no-no for App Engine. If this happens you will get an error like this:

Fatal error: Uncaught exception 'UnexpectedValueException' with message 'The stream or file "[Path to your lumen app]/storage/logs/lumen.log" could not be opened

One little configuration tweak got it working though. In [path to your lumen app ]vendor/laravel/lumen-framework/src/Application.php, you have to tweak the logger a bit.

First you add a reference to the right library:

use MonologHandlerSyslogHandler;

And then replace the function getMonologHandler() with this:

protected function getMonologHandler()
{
return new SyslogHandler('intranet', 'user', Logger::DEBUG, false, LOG_PID);
}

This advice was taken from the guide for Laravel on PHP for App Engine.

All in all, it is a relatively quick little framework, and having using App Engine to scale out the services it provides seems like a no-brainer.

All code shown here is licensed under Apache 2. For more details find the original source on Github.

Sharing Memcache Between Languages in App Engine

memcache diagramIn the process of performance testing the ability to swap out languages in App Engine detailed in this post, I stumbled on to something.  I was testing performance, and realized that the tests weren’t being accurate because of differences in caching. Ideally, to get the tests to be apples to apples, I would just have to get my PHP code and Go code to use the same Memcache instance and keys.  (I should have written my testing better, but then if I had I would never have stumbled into this.)

To start, follow the steps to get multiple languages working in a production instance or a development instance.

Assuming you are writing from PHP:

$memcache = new Memcache;
$memcache->set("secret", "I wrote this over in PHP");
$value = $memcache->get("secret");
echo $value;

And then to read from Go:

func handler(w http.ResponseWriter, r *http.Request) {
	c := appengine.NewContext(r)

	item, err := memcache.Get(c, "secret")
	if err != nil {
		handleError(w, err)
	}

	fmt.Fprint(w, string(item.Value))
}

It really is that easy. Now the hard part comes when you want to transfer complex data between the two.  Use JSON to encode the objects.  Both languages can handle it pretty effortlessly and Go on App Engine has JSON object handling built in as a codec to its memcache implementation. You could save it in another format like XML then read and write data like a string, while manually encoding and decoding.  You could also staple your had to your desk. Let’s not be a masochist and just do it in JSON – but I suppose it’s your choice.

Once you do that, it’s as simple as encoding to JSON in PHP:

$memcache = new Memcache;

$info['name'] = "hometown";
$info['city'] = "Philadelphia";
$info['state'] = "PA";
$info['latitude'] = "39.995664772727"; 
$info['longitude'] = "-75.138788636364";

$secret = json_encode($info,JSON_NUMERIC_CHECK);

$memcache->set("secret2", $secret);
$value = $memcache->get("secret2");	

echo $value

Then decoding in Go.

type Location struct {
	Name  string  `json:"name"`
	City  string  `json:"city"`
	State string  `json:"state"`
	Lat   float32 `json:"latitude"`
	Lon   float32 `json:"longitude"`
}

func jsonHandler(w http.ResponseWriter, r *http.Request) {
	c := appengine.NewContext(r)

	var item Location
	key := "secret2"
	_, err := memcache.JSON.Get(c, key, &item)
	if err != nil {
		handleError(w, err)
	}

	fmt.Fprintf(w, "%+vn", item)

}

Note a couple of things:

  • I omitted graceful memcache miss handling. I did so for brevity. Make sure you wrap your memcache code that handle cache misses.
  • If you are not familiar with Go, those ‘json:’ comments aren’t just comments, they’re instructions on how to encode/decode data between Go and JSON.  So you need them, or it won’t work correctly.

  • I ran into an issue with the original version of this code because latitude and longitudes were coming out of the database into PHP as strings and not floats. When you went to get them out of memcache in Go, it would through a type mismatch error.  There are 2 solutions to this:

    • cast them correctly to floats before you write to memcache

    • Use JSON_NUMERIC_CHECK in json_encode to get them to write as proper numerics when you write. This seems like the better solution

Why do this?  For starters I was doing it so  both versions of my API could take advantage of caching done by the other language.  But I am sure there are other uses:

  • Communication between these modules

  • Offloading an expensive data retrieval and processing step to Go then reading memcache from PHP.

  • I’m curious if anyone reading has any thoughts.

Note: This will work on either type of memcache solution on App Engine: shared or dedicated. Just make sure you handle cache misses gracefully.

 

All code show here is licensed under Apache 2. For more details find the original source on Github.