Archives For Google App Engine

I’m forever having to look up various types of obscure GQL syntax to use in the App Engine console. I thought I’d start a blog post where I note them down, for my own sake. I’ll update this as I go.

Look up entities by encoded key string

SELECT * FROM EntityName WHERE __key__ IS KEY('dkfjbILVUYFkfbaelIIUBkuafvKJ')

Look up child entities by encoded key string

SELECT * FROM EntityName WHERE ancestor IS KEY('dkfjbILVUYFkfbaelIIUBkuafvKJ')

More to come, if you find one I’m missing let me know

As you may know, we run our OrderPipe ecommerce dashboard on Google’s App Engine – I’m a big fan of the platform, but it has some traps for new (and old) players. A recent issue required us to bulk delete entities on App Engine, and for that it seemed the best tool for the job was the ‘Mapper’ part of Map Reduce. It was a good experience learning a) how it works and b) applying it, I thought I’d document a full Map Reduce example, because the worked examples I found were all based on an older version of the library.

Background

You might be interested to read a bit about why I had to use Map Reduce – it was my own fault. A subtle bug in a recent release caused an additional Account to be created under certain situations – it made it past our staging server, because that certain situation happened infrequently enough to not cause an issue, but once it got to production, where we have many thousands of orders arriving daily, we started seeing a lot of zombie accounts being created. My first thought was we were being attacked, but alas, it wasn’t an attack, just a bug! The nett result, we were left with 10′s of thousands of unwanted entities.

The Problem

In App Engine, the datastore is highly scalable, but it’s not relational, you can’t just run a simple query to delete all rows in a table with a particular created timestamp. There are simple tools to blindly delete all entities of a type, but not entities that meet a particular condition, and more importantly, not related entities that also meet a particular condition. That’s where Map Reduce comes in, basically we shard the entities and create multiple parallel workers to break the collection into smaller parts and process through them quickly. This seems like an inefficient approach until you consider scaling beyond a single database server, or even a single database cluster.
Continue Reading…

Way back in 2010 I set up a quick little wrapper around siege for testing Magento stores and for the last nearly 2 years I’ve added features such as multiple data centers, premium subscriptions, monitoring alerts, and automated host comparisons. In this blog post I’ll walk you through an update on the new features and improvements to MageSpeedTest.com over the last few months.

I’m really happy with the site and how it helps users identify and improve Magento performance. I often see test results coming through in the order of 2-3 transactions per second and >5s per transaction. Identifying these sort of slow stores early is important – customers are simply not going to wait at least 5 seconds to see a product page these days.

At the opposite end of the performance spectrum I see public tests coming through with incredible performance profiles. For these developers and server admins it’s a way to consistently test server tweaks and tuning to get the absolute best customer experience.

In this post I wanted to quickly cover my latest Magento Speed Test changes. It’s a big update because I haven’t posted anything in a while.

Here’s the cliff notes:

  • New Locations: Japan, South America and Australia
  • A Magento Extension to make testing easy (and more soon!)
  • A REST API for accessing your monitoring results
  • A pricing update, $5 up to $9 per month
  • Featured Magento hosts
  • SSL support – test https:// sites

For more information on these items, please read on.
Continue Reading…

In this post I’ll show you how to send an Apple Notification from Google App Engine via Urban Airship. I was mightily impressed how easy it was to do earlier this week. Their docs are excellent, and I love when apps use curl for their API examples – it means the API is simple, and the examples are perfectly transparent.
Continue Reading…

I took a bit of a break for the usual Magento development last weekend and had a crack at making a simple webstore monitoring app using Google App Engine. The application was made to monitor our webstores from a reliable, US network – and who more reliable than Google? So this blog post will detail the site monitoring app, and my thoughts on the Google app engine platform. I’d also invite anyone to try my app and let me know any thoughts you have on it, or how I could improve it.

UPDATE July 2011: I have since created a much more fully features Magento monitoring tool called magespeedtest.com – it checks speed and performance regularly and emails you if it differs from your preset tolerance. I won’t be adding new features to this monitor, it’s still free and I’ll keep it up and running for as long as it makes sense to, but any future development will be on MageSpeedTest.com.

Continue Reading…