Speeding up data store queries with self-merge joins

Daniel Eichhorn — Sun, 27 Feb 2011 07:33:00 +0000

For a while now Feederator allows some kind of full text queries. When items are read for the first time into the datastore an index will be created. This index is currently a break down of the single words in the title, but can be extended to the description and other attributes of an item. My first attempt of implementing the full text search on the AppEngine resulted in a very costly algorithm.
The fulltext search index is stored as a set of strings. The datastore allows you to query for items on a set as if it would be a singular attribute, like a single String for example. So you can create a query for a set of strings index={“green”, “yellow”, “red”} in this way WHERE index == “red” which would be a hit in this case. I tried to do a query like this WHERE index == “red” AND index == “green” order by date which returned with errors telling me that I need to create a datastore index for that query. I thought that the error came from the fact that I had to filters on the attribute index. But I was wrong. The problem was that I wanted to sort the results at the same time. So, by forgetting about the in-database-sorting the query works and it works darn fast. And even better: you don’t even need an index for AND connected filters like that! Which means you can save storage space (in my case almost 1GB!).

Here are some rules of thumb that I learned from this lesson. Due to the way the datastore is organized (don’t forget its BigTable nature!) you have to pay attention when designing your data model.

You can query easily on sets
Try to avoid non-equal filters or filters that use OR. The best filters are ones that try to find an intersection between different domains on ONE entity
Sort your results in-memory to avoid the need for indexes
Sets are expensive to serialize/ deserialize. This costs occure when you write or read a set from datastore. To avoid serialization costs put your full text index on a separate child object. You query on the child object only for the key. With the key you get the parent object and then you can do in-memory sorting/ ranking

These links helped me a lot to understand how to optimize my full text search:

New Features: Searching the archive

Daniel Eichhorn — Sun, 19 Dec 2010 13:50:00 +0000

I’m trying to improve Feederator.org. Sometimes changes are rather invisible to the end user. Sometimes new features extend the use of feederator rather dramatically. I enabled such a feature today: searching the archives with full text key words.
To keep your inbox clean Feederator moves items older than 3 days to the archive. Once they are in the archive you can search for them. Make sure that your most significant word comes first, otherwise it might be omitted. The search is not as good yet as Google but it is getting better. Indexed words are part of the title of a post and its categories. In addition the MIME Type of the attachment will also be indexed. Entering “2010 nzb dvdrip” will return with a list of items containing 2010 and dvdrip in the title and having a usenet enclosure.
In the time of this writing there are already more than 20’000 torrent items and more than 15’000 nzb items indexed.

Other new features:

Social bookmarking items: share items that you like with your friends in Twitter, Facebook and others
Drag And Drop: by dragging an item with the dotted handle to a feed label on the left you can copy that item to a new feed. This also works with search results: you can search for movies and once you found one that you like you can drag it to the feed that your downloader has subscribed to

full text search – Squix – TechBlog

Speeding up data store queries with self-merge joins

New Features: Searching the archive