Google App Engine – Squix – TechBlog

Continuous Deployment from Github to App Engine Java with CodeShip

Daniel Eichhorn — Sat, 01 Mar 2014 18:49:00 +0000

Motivation

Continuous Deployment is nice: you roll out new versions of your software all the time and not only three, four times a year with your mayor release. And everything automatically. Of course this requires a good coverage of automated unit, integration and UI tests to make sure that your current version doesn’t break anything for the user.
What tools can you use do achieve this for your open source project? If I’m not mistaken you get all this with an account at bit bucket from Atlassian. But since I currently host my open source projects on Github I had to find another solution. And it exists, even for free: http://codeship.io
Codeship already nows about many services and you only have to configure a few lines to get it up and running. For instance it supports Github as code repository and Google App Engine as deployment target. But the documentation about it is still a bit sparse. So this article describes how you can set it up

SCM

The setup of github as a repository is fairly straight forward: you use your github account to authenticate yourself through OAuth with CodeShip. Then you create a new project and in the SCM section you pick Github as your provider. After allowing codeship to access your Github repositories you already can select the repo you want to build from.

Tests

Currently I don’t have integration and UI tests yet for my open source project. Anyway, to make sure your build runs at least once I chose the JVM option in the “Select your technology to prepopulate basic commands” dropdown and added a

mvn package

in the “Modify your Setup Commands” section. I left the rest unchanged.

Deployment

And now came the tricky part. I don’t want the build to deploy to the GAE application that is defined in the appengine-web.xml. Instead I want it to be installed to a test environment with a different name running on GAE as well. So I added a second appengine-web-testenv.xml beside the default one and changed the application id to the test environment. Then I selected the “$script” option in the CodeShip deployment tab and added a

cp target/-/WEB-INF/appengine-web-testenv.xml target/-/WEB-INF/appengine-web.xml

Please replace maven.project and maven.version with the name of your maven artefact. In my case this is “extraleague-1.0-SNAPSHOT”. So this basically replaces the original appengine file with the one containing the test environment id. When appcfg.sh now tries to find the target for deployment it will find the one with the test environment.
Now we add another deployment descriptor by selecting the app engine option. I think when you first do this you have to authorise CodeShip to deploy to GAE on your behalf. After that I added./target/extraleague-1.0-SNAPSHOT/ as Path and a rest service endpoint in the URL field. This URL will be checked to decide if the deployment was successful. In my first attempt I used the index.html of my angular application for this. The problem here is that static files like the index.html did get served while the Java server part had a 500 error. That’s why I think a rest point is a better choice here.

Triggering a run and debugging

To have CodeShip do a first run you just have to commit a change to Github and there it goes. You can follow each step in the admin interface of your build and see read the log if something goes wrong. You can also change the deployment settings and rerun a previously failed build.

Remarks

I’m really happy with this setup, but I think there should be a way to avoid setting the artefact name and version into the setting path. Probably some maven command would do the trick? As next steps I’d like to setup a UI test, maybe with PhantomJS or so… Let me know if you have improvements!

Update 2014/05/10
Last week I had issues with app engine authentication and after that was solved it seemed that codeship used an old version of the app engine sdk to deploy which caused issues. So I started looking for an alternative to the above description and found it by adding the following script in the deployment section (no other build step required anymore, you can remove the app engine step):

cp src/main/webapp/WEB-INF/appengine-web-testenv.xml  src/main/webapp/WEB-INF/appengine-web.xml
cat src/main/webapp/WEB-INF/appengine-web.xml
echo '{"credentials":{"rof":{"access_token":"XXXX","expiration_time_millis":1399667604244,"refresh_token":"XXX"}}}' > ~/.appcfg_oauth2_tokens_java
mvn appengine:update
wget --retry-connrefused --no-check-certificate -T 60 http://your-test-url

Explanation
The first line is the same as before: it copies my alternative deployment descriptor in the place of the default descriptor. The second line is used for debugging and to be sure that the task will really deploy to my test instance. The third line is the most interesting. For my own security I replaced the real tokens with XXX. How do you get this JSON object? Very simple: deploy your application locally and you can copy this JSON object from .appcfg_oauth2_tokens_java. Replace your current system user with rof! The next line simply deploys the application to my test environment and the last line checks if the application was deployed successfully

Live reload with appengine-maven-plugin

Daniel Eichhorn — Sat, 19 Oct 2013 17:45:00 +0000

Google’s appengine-maven-plugin is cool. Especially for deploying your app to the cloud. But for developing it can be cumbersome, since the hot deploy mechanism is not very stable and you often have to restart the devserver. Since I wanted to get more into the whole JavaScript build chain I had a look at grunt but I just couldn’t get the configuration running with connect and livereload. And then I found this link: http://nitoyon.github.io/livereloadx/
First of all you have to install node.js as described here. To install the livereloadx run:

npm install -g livereloadx

With this command

livereloadx -l -p 35729 src/main/webapp/ --proxy http://localhost:8080/

you can start a proxy server that listens on port 35729 and checks first src/main/webapp/ for requests. If it doesn’t find the resource there it will check http://localhost:8080 for the resource. In my case the devserver run on http://localhost:8080 and the rest interface will respond to any calls that cannot be answered from src/main/webapp.

If you install the chrome plugin from
the chrome store your browser will refresh automatically

UPDATE: It seems that the current version doesn’t handle HTTP POST requests if the “prefer-local” flag is set (the “-l” in the example above). I filed a bug and I hope that this will be fixed soon: https://github.com/nitoyon/livereloadx/issues/3

UPDATE2: It has been fixed. Thank you very much nitoyon!

Exporting your App Engine Data with a Sitemap of Sitemaps

Daniel Eichhorn — Sat, 19 Mar 2011 07:09:00 +0000

Currently there are about 200’000 archived items available in the Feederator database. In the hope to attract more traffic I wanted to make this data available to search engines. But how to make this data available to search engines without putting too much burden on my App Engine Quota. Certainly one big Sitemap wouldn’t work, because the call from a search engine would reach the 30 seconds limit of a Servlet call long before all the links to the different items could be created.
Ok, that means breaking it down into smaller chunks. But how to distribute the data more or less equally? That was an easy one: since Feederator adds more or less the same quantity of new feed items to the archive I would have to separate the chunks by day. Every item already has a publishedDate, which marks the date of entry to the datastore. But that would be another problem: how to add all the different sitemaps to search engines like Google or Bing?
Turns out there’s a solution for exactly that: a sitemap of sitemaps. The super sitemap file has to look as described here. So this file would contain one entry per day and could be generated relatively easy by iterating over all days in the last three months (the duration items are stored in the archive). I wouldn’t even have to know how many items are available per day in this stage. Just assume there are some, that’s enough.
This looks like this:

http://www.feederator.org/sitemap/2010-12-19

http://www.feederator.org/sitemap/2010-12-20

http://www.feederator.org/sitemap/2010-12-21

…

In my case, the same servlet that creates the sitemapindex file also creates different per-date index files. To avoid huge quantities of XML overhead you can export the per-date sitemaps also in line by line format. The sitemap file for 2010-12-19 looks like this:

http://www.feederator.org/item/agpmZWVkZXJhdG9ychYLEg1GZWVkSXRlbU1vZGVsGLfUvwIM

http://www.feederator.org/item/agpmZWVkZXJhdG9ychYLEg1GZWVkSXRlbU1vZGVsGLjUvwIM

http://www.feederator.org/item/agpmZWVkZXJhdG9ychYLEg1GZWVkSXRlbU1vZGVsGLvUvwIM

http://www.feederator.org/item/agpmZWVkZXJhdG9ychYLEg1GZWVkSXRlbU1vZGVsGMHUvwIM

http://www.feederator.org/item/agpmZWVkZXJhdG9ychYLEg1GZWVkSXRlbU1vZGVsGNbUvwIM

http://www.feederator.org/item/agpmZWVkZXJhdG9ychYLEg1GZWVkSXRlbU1vZGVsGOTUvwIM

With one link per line. To write a servlet like that is pretty easy.

Speeding up data store queries with self-merge joins

Daniel Eichhorn — Sun, 27 Feb 2011 07:33:00 +0000

For a while now Feederator allows some kind of full text queries. When items are read for the first time into the datastore an index will be created. This index is currently a break down of the single words in the title, but can be extended to the description and other attributes of an item. My first attempt of implementing the full text search on the AppEngine resulted in a very costly algorithm.
The fulltext search index is stored as a set of strings. The datastore allows you to query for items on a set as if it would be a singular attribute, like a single String for example. So you can create a query for a set of strings index={“green”, “yellow”, “red”} in this way WHERE index == “red” which would be a hit in this case. I tried to do a query like this WHERE index == “red” AND index == “green” order by date which returned with errors telling me that I need to create a datastore index for that query. I thought that the error came from the fact that I had to filters on the attribute index. But I was wrong. The problem was that I wanted to sort the results at the same time. So, by forgetting about the in-database-sorting the query works and it works darn fast. And even better: you don’t even need an index for AND connected filters like that! Which means you can save storage space (in my case almost 1GB!).

Here are some rules of thumb that I learned from this lesson. Due to the way the datastore is organized (don’t forget its BigTable nature!) you have to pay attention when designing your data model.

You can query easily on sets
Try to avoid non-equal filters or filters that use OR. The best filters are ones that try to find an intersection between different domains on ONE entity
Sort your results in-memory to avoid the need for indexes
Sets are expensive to serialize/ deserialize. This costs occure when you write or read a set from datastore. To avoid serialization costs put your full text index on a separate child object. You query on the child object only for the key. With the key you get the parent object and then you can do in-memory sorting/ ranking

These links helped me a lot to understand how to optimize my full text search:

Sending XMPP messages from Google App Engine with html

Daniel Eichhorn — Fri, 18 Feb 2011 17:13:00 +0000

Since this took me a while to figure out I’d like to share with you. I wanted to extend Feederator to send incoming news items to the users XMPP client. What worked pretty fast was sending plain text messages:

XMPPService xmpp = XMPPServiceFactory.getXMPPService();

JID jid = new JID(xmppAddress);

Message msg = new MessageBuilder()

.withRecipientJids(jid)

.withMessageType(MessageType.CHAT)

.withBody(“New post arrived”)

.build();

if (xmpp.getPresence(jid).isAvailable()) {

SendResponse status = xmpp.sendMessage(msg);

messageSent = (status.getStatusMap().get(jid) == SendResponse.Status.SUCCESS);

if (messageSent) {

Log.info(“message sent”);

}

But to send parts of the message bold and maybe even displaying pictures was trickier. But I finally figured it out. Change the part with the message:

String message = “Plain text message”

+ “”

+ “bold text and normal text”

+ “”

+ “”;

Message msg = new MessageBuilder()
.withRecipientJids(jid)

.withMessageType(MessageType.CHAT)

.withBody(message)

.asXml(true);

.build();

By adding the .asXml(true) parameter you tell the XMPP service that you take care yourself of the content of the message. Pay attention to the fact that you always should send a normal elment for the plain text. Not all XMPP/ Jabber clients unterstand the xhtml-im extension. The specification (http://xmpp.org/extensions/xep-0071.html) says that the plain text should always tell the same as the element with markup and that clients are free to ignore the xhtml-im tag. Also note that clients are free to display the text defined by the alt attribute of a tag and not the picture.

How to delete index on App Engine for Java

Daniel Eichhorn — Wed, 22 Dec 2010 19:24:00 +0000

At the time of this writing there is no function to delete (unused) your datastore indexes directly with the Java tool set. But there is a way around: the python SDK has a function called vacuum_indexes. This is how your can do it, step by step:

Install the Google App Engine for Python SDK and follow the steps described for your operating system here: http://code.google.com/appengine/docs/python/gettingstarted/devenvironment.html
After you installed python, and the GAE for Python SDK, setup a project in a tmp folder, that has the same name as your Java project with the tools from the SDK. Pay attention on capital letters. Your project must have EXACT the same name as the Java version
[Optional] Deploy that to the app engine, but pay attention that you choose a version differently from all versions used by your java environment. You will be asked for your email and password
On the command line execute appcfg.py vacuum_indexes pathToYourProject/
Then you have to decide which index you want to delete. Be careful. Deleting the wrong index can render your app temporarily unusable…