Sharing Memcache Between Languages in App Engine

memcache diagramIn the process of performance testing the ability to swap out languages in App Engine detailed in this post, I stumbled on to something.  I was testing performance, and realized that the tests weren’t being accurate because of differences in caching. Ideally, to get the tests to be apples to apples, I would just have to get my PHP code and Go code to use the same Memcache instance and keys.  (I should have written my testing better, but then if I had I would never have stumbled into this.)

To start, follow the steps to get multiple languages working in a production instance or a development instance.

Assuming you are writing from PHP:

$memcache = new Memcache;
$memcache->set("secret", "I wrote this over in PHP");
$value = $memcache->get("secret");
echo $value;

And then to read from Go:

func handler(w http.ResponseWriter, r *http.Request) {
	c := appengine.NewContext(r)

	item, err := memcache.Get(c, "secret")
	if err != nil {
		handleError(w, err)

	fmt.Fprint(w, string(item.Value))

It really is that easy. Now the hard part comes when you want to transfer complex data between the two.  Use JSON to encode the objects.  Both languages can handle it pretty effortlessly and Go on App Engine has JSON object handling built in as a codec to its memcache implementation. You could save it in another format like XML then read and write data like a string, while manually encoding and decoding.  You could also staple your had to your desk. Let’s not be a masochist and just do it in JSON – but I suppose it’s your choice.

Once you do that, it’s as simple as encoding to JSON in PHP:

$memcache = new Memcache;

$info['name'] = "hometown";
$info['city'] = "Philadelphia";
$info['state'] = "PA";
$info['latitude'] = "39.995664772727"; 
$info['longitude'] = "-75.138788636364";

$secret = json_encode($info,JSON_NUMERIC_CHECK);

$memcache->set("secret2", $secret);
$value = $memcache->get("secret2");	

echo $value

Then decoding in Go.

type Location struct {
	Name  string  `json:"name"`
	City  string  `json:"city"`
	State string  `json:"state"`
	Lat   float32 `json:"latitude"`
	Lon   float32 `json:"longitude"`

func jsonHandler(w http.ResponseWriter, r *http.Request) {
	c := appengine.NewContext(r)

	var item Location
	key := "secret2"
	_, err := memcache.JSON.Get(c, key, &item)
	if err != nil {
		handleError(w, err)

	fmt.Fprintf(w, "%+vn", item)


Note a couple of things:

  • I omitted graceful memcache miss handling. I did so for brevity. Make sure you wrap your memcache code that handle cache misses.
  • If you are not familiar with Go, those ‘json:’ comments aren’t just comments, they’re instructions on how to encode/decode data between Go and JSON.  So you need them, or it won’t work correctly.

  • I ran into an issue with the original version of this code because latitude and longitudes were coming out of the database into PHP as strings and not floats. When you went to get them out of memcache in Go, it would through a type mismatch error.  There are 2 solutions to this:

    • cast them correctly to floats before you write to memcache

    • Use JSON_NUMERIC_CHECK in json_encode to get them to write as proper numerics when you write. This seems like the better solution

Why do this?  For starters I was doing it so  both versions of my API could take advantage of caching done by the other language.  But I am sure there are other uses:

  • Communication between these modules

  • Offloading an expensive data retrieval and processing step to Go then reading memcache from PHP.

  • I’m curious if anyone reading has any thoughts.

Note: This will work on either type of memcache solution on App Engine: shared or dedicated. Just make sure you handle cache misses gracefully.


All code show here is licensed under Apache 2. For more details find the original source on Github.


Two Languages in App Engine Development

In my last post I outlined getting Go and PHP to act as modules in the same App Engine instance.  However I only really tested it on a “production” App Engine instance, I didn’t test it in development, because I typically use the Google App Engine SDK for each respective language separately.

When I tried the combined dispatch.yaml on the Google App Engine SDK for PHP I got the following error on a Mac running OS 10.10.2 (Yosemite):

OSError: [Errno 2] No such file or directory: '/Applications/Development/’

When I tried the combined dispatch.yaml on the Google App Engine SDK for Go I got the following error:

The development server must be started with the --php_executable_path flag set to the path of the php-cgi binary.

And when  used the -php_executable_path option with any of the copies of PHP on my system – including the ones that are buried in the PHP SDK – I got:

_PHPEnvironmentError: No input file specified.

After struggling a bit with this here is the easiest solution I found:

  • Find the location of goroot in the go_appengine folder

  • Find the location of the SDK for php by running

    which | xargs ls -l
  • Create a symbolic link to goroot in go_appengine in the PHP SDK folder that contains

After that you can test your dispatch file in development by running: dispatch.yaml api_php/app.yaml  api_go/app.yaml

Where api_php is the folder your PHP module is in, and api_go is… well you know what I’m saying.

Now, I went out of my way there to say this was the easiest way of doing it.  Not that it wasn’t a hack, or that it was a supported way of doing it.  But it does work.

All code show here is licensed under Apache 2. For more details find the original source on Github.

Two Languages in One App Engine App

AppEngine_512pxThe other day I was talking to students at a bootcamp about languages. I made the comment that language performance can vary depending on what a particular language is best at doing. When you run into performance issues it can sometimes be helpful to try rewriting pieces of your app in a particular language for a performance boost.

I thought about how that could be done in App Engine. Let’s say I have a section of an application that I wrote in PHP, but it was getting more load than expected, so I need to boost its performance. I want to try and see if Go could give me the boost I need. How hard is that to do?

Please keep in mind all of the caveats here.  Sometimes you can get a boost, sometimes it’s worth exploring. You know, it was a theoretical conversation. And for the record. This need to drop to another language doesn’t have to be performance related. It could be due to SDK or API restrictions, or developer knowledge, or just plain “I want to use another language to do this.”

In App Engine we do this through the use of modules.  Modules allow us to separate front end and back end code from each other.  But they allow us to break up large applications into manageable chunks.  In this case, we’re going to use them to allow us to break up code into multiple languages.

Let’s assume that you have an application with an app.yaml that looks like this:

application: <appengine project name>
module: default
version: 1
api_version: 1
runtime: php55
threadsafe: yes

- url: /place
  script: place.php  

- url: /details
  script: details.php   

- url: /distance
  script: distance.php

Let’s say that you want to swap out the distance method for go. The first thing you need to do is write a dispatch.yaml, which looks like this:

application: <appengine project name>

- url: "*/*"
  module: default

This will redirect all calls to your App Engine app to the Php application above. Which is what has been happening to date. But this is a setup step for later.  You then have to add the dispatch file to your application. In a command prompt, from the folder containing dispatch.yaml, run: --oauth2  update_dispatch .

Write a replacement for your distance method in Go. Go on, we’ll wait…

Ok, assuming you’ve done that you write out an app.yaml for the Go code you wrote:

application:  <appengine project name>
module: goapi
version: 1
runtime: go
api_version: go1

- url: /.*
  script: _go_app

Take note of the module name. It has to be different from the original app’s module, which should be “default.”

Once you have all of that handled you need to tweak your dispatch.yaml to replace calls made to the php version of the distance method to the Go method:

application:  <appengine project name>

- url: "*/distance"
  module: goapi

- url: "*/*"
  module: default

Rerun the dispatch update: --oauth2  update_dispatch .

And there you go, the original PHP service will answer all other calls, but the Go service will answer calls for /distance.

Running multiple language solutions in the same App Engine instance can solve some problems for you.  It also has a few interesting ramifications.  These include the ability to use the same shared Memcached instance between Go and Php. I’m going to show that off in my next blog post.

All code show here is licensed under Apache 2. For more details find the original source on Github.