Exporting your App Engine Data with a Sitemap of Sitemaps

Daniel Eichhorn — Sat, 19 Mar 2011 07:09:00 +0000

Currently there are about 200’000 archived items available in the Feederator database. In the hope to attract more traffic I wanted to make this data available to search engines. But how to make this data available to search engines without putting too much burden on my App Engine Quota. Certainly one big Sitemap wouldn’t work, because the call from a search engine would reach the 30 seconds limit of a Servlet call long before all the links to the different items could be created.
Ok, that means breaking it down into smaller chunks. But how to distribute the data more or less equally? That was an easy one: since Feederator adds more or less the same quantity of new feed items to the archive I would have to separate the chunks by day. Every item already has a publishedDate, which marks the date of entry to the datastore. But that would be another problem: how to add all the different sitemaps to search engines like Google or Bing?
Turns out there’s a solution for exactly that: a sitemap of sitemaps. The super sitemap file has to look as described here. So this file would contain one entry per day and could be generated relatively easy by iterating over all days in the last three months (the duration items are stored in the archive). I wouldn’t even have to know how many items are available per day in this stage. Just assume there are some, that’s enough.
This looks like this:

http://www.feederator.org/sitemap/2010-12-19

http://www.feederator.org/sitemap/2010-12-20

http://www.feederator.org/sitemap/2010-12-21

…

In my case, the same servlet that creates the sitemapindex file also creates different per-date index files. To avoid huge quantities of XML overhead you can export the per-date sitemaps also in line by line format. The sitemap file for 2010-12-19 looks like this:

http://www.feederator.org/item/agpmZWVkZXJhdG9ychYLEg1GZWVkSXRlbU1vZGVsGLfUvwIM

http://www.feederator.org/item/agpmZWVkZXJhdG9ychYLEg1GZWVkSXRlbU1vZGVsGLjUvwIM

http://www.feederator.org/item/agpmZWVkZXJhdG9ychYLEg1GZWVkSXRlbU1vZGVsGLvUvwIM

http://www.feederator.org/item/agpmZWVkZXJhdG9ychYLEg1GZWVkSXRlbU1vZGVsGMHUvwIM

http://www.feederator.org/item/agpmZWVkZXJhdG9ychYLEg1GZWVkSXRlbU1vZGVsGNbUvwIM

http://www.feederator.org/item/agpmZWVkZXJhdG9ychYLEg1GZWVkSXRlbU1vZGVsGOTUvwIM

With one link per line. To write a servlet like that is pretty easy.

How to delete index on App Engine for Java

Daniel Eichhorn — Wed, 22 Dec 2010 19:24:00 +0000

At the time of this writing there is no function to delete (unused) your datastore indexes directly with the Java tool set. But there is a way around: the python SDK has a function called vacuum_indexes. This is how your can do it, step by step:

Install the Google App Engine for Python SDK and follow the steps described for your operating system here: http://code.google.com/appengine/docs/python/gettingstarted/devenvironment.html
After you installed python, and the GAE for Python SDK, setup a project in a tmp folder, that has the same name as your Java project with the tools from the SDK. Pay attention on capital letters. Your project must have EXACT the same name as the Java version
[Optional] Deploy that to the app engine, but pay attention that you choose a version differently from all versions used by your java environment. You will be asked for your email and password
On the command line execute appcfg.py vacuum_indexes pathToYourProject/
Then you have to decide which index you want to delete. Be careful. Deleting the wrong index can render your app temporarily unusable…

deleting datastore indexes – Squix – TechBlog

Exporting your App Engine Data with a Sitemap of Sitemaps

How to delete index on App Engine for Java