Posted on

Google App Engine GQL queries in the Datastore Viewer console

I’m forever having to look up various types of obscure GQL syntax to use in the App Engine console. I thought I’d start a blog post where I note them down, for my own sake. I’ll update this as I go.

Look up entities by encoded key string

SELECT * FROM EntityName WHERE __key__ IS KEY('dkfjbILVUYFkfbaelIIUBkuafvKJ')

Look up child entities by encoded key string

SELECT * FROM EntityName WHERE ancestor IS KEY('dkfjbILVUYFkfbaelIIUBkuafvKJ')

More to come, if you find one I’m missing let me know

Posted on

App Engine Java MapReduce Example: bulk deleting entities

As you may know, we run our OrderPipe ecommerce dashboard on Google’s App Engine – I’m a big fan of the platform, but it has some traps for new (and old) players. A recent issue required us to bulk delete entities on App Engine, and for that it seemed the best tool for the job was the ‘Mapper’ part of Map Reduce. It was a good experience learning a) how it works and b) applying it, I thought I’d document a full Map Reduce example, because the worked examples I found were all based on an older version of the library.


You might be interested to read a bit about why I had to use Map Reduce – it was my own fault. A subtle bug in a recent release caused an additional Account to be created under certain situations – it made it past our staging server, because that certain situation happened infrequently enough to not cause an issue, but once it got to production, where we have many thousands of orders arriving daily, we started seeing a lot of zombie accounts being created. My first thought was we were being attacked, but alas, it wasn’t an attack, just a bug! The nett result, we were left with 10’s of thousands of unwanted entities.

The Problem

In App Engine, the datastore is highly scalable, but it’s not relational, you can’t just run a simple query to delete all rows in a table with a particular created timestamp. There are simple tools to blindly delete all entities of a type, but not entities that meet a particular condition, and more importantly, not related entities that also meet a particular condition. That’s where Map Reduce comes in, basically we shard the entities and create multiple parallel workers to break the collection into smaller parts and process through them quickly. This seems like an inefficient approach until you consider scaling beyond a single database server, or even a single database cluster.
Continue reading App Engine Java MapReduce Example: bulk deleting entities

Posted on New locations, Magento Extension, REST API, Featured Host and Pricing

Way back in 2010 I set up a quick little wrapper around siege for testing Magento stores and for the last nearly 2 years I’ve added features such as multiple data centers, premium subscriptions, monitoring alerts, and automated host comparisons. In this blog post I’ll walk you through an update on the new features and improvements to over the last few months.

I’m really happy with the site and how it helps users identify and improve Magento performance. I often see test results coming through in the order of 2-3 transactions per second and >5s per transaction. Identifying these sort of slow stores early is important – customers are simply not going to wait at least 5 seconds to see a product page these days.

At the opposite end of the performance spectrum I see public tests coming through with incredible performance profiles. For these developers and server admins it’s a way to consistently test server tweaks and tuning to get the absolute best customer experience.

In this post I wanted to quickly cover my latest Magento Speed Test changes. It’s a big update because I haven’t posted anything in a while.

Here’s the cliff notes:

  • New Locations: Japan, South America and Australia
  • A Magento Extension to make testing easy (and more soon!)
  • A REST API for accessing your monitoring results
  • A pricing update, $5 up to $9 per month
  • Featured Magento hosts
  • SSL support – test https:// sites

For more information on these items, please read on.
Continue reading New locations, Magento Extension, REST API, Featured Host and Pricing

Posted on

Java App Engine integration testing with Spring MVC and OpenPersistenceManagerInViewFilter

In this post I’ll quickly show the technique I used for integration testing of Java services where the services rely on Spring data repositories being injected. We’re using Google’s App Engine to host and run our mobile sales dashboard for Magento/Amazon/Shopify so the tests will use the integration testing environment provided in the SDK.

Starting with a number of great articles on the subject I was able to quickly get a basic integration test up and running for App Engine with Spring. Our problem was related particularly to the PersistenceManager.

In the real application each web request is assigned a ThreadLocal PeristenceManager which is kept alive for the duration of that request, and always closed at the end, thanks to the OpenPersistenceManagerInViewFilter provided by Spring. So this is great when you’re calling data access methods on repositories from services in a web container, but means that if you’re trying to call those services during an integration test, the different repository calls will trigger exceptions that the objects involved belong to different PersistenceManagers.

The solution was to emulate the behavior of the OpenPersistenceManagerInViewFilter before and after a web request, in the setUp and tearDown of the tests themselves. For that I checked out the code for the filter and simply applied the same logic in the test.

First we make sure the Test class is setup to use Springs Junit functionality like below. Also note we are specifically setting the context to use the two test xml configurations. This is important because we actually wire a LocalPersistenceManagerFactoryBean in this Test class.

@ContextConfiguration(locations =  {"classpath:application-config-test.xml", "classpath:servlet-config-test.xml"})
public abstract class BaseSpringJunitTest
extends AbstractJUnit4SpringContextTests {
	LocalPersistenceManagerFactoryBean pmf;
	// Standard App Engine integration testing helper
	protected final LocalServiceTestHelper helper =
			new LocalServiceTestHelper(new LocalDatastoreServiceTestConfig(), new
					LocalTaskQueueTestConfig(), new LocalMemcacheServiceTestConfig());
	// ...

Most of the test xml configuration is standard, the important bit is wiring in the bean, which should look familiar from your usual container deployment:

<bean class="org.springframework.orm.jdo.LocalPersistenceManagerFactoryBean" name="persistenceManagerFactory">
	<property name="persistenceManagerFactoryName" value="transactions-optional" />

The setUp() just creates a new PersistenceManager (from the PMF we wired in) and stores it for use during the test (the same way the filter stores it for a web request).

public void setUp() {
    PersistenceManager pm = PersistenceManagerFactoryUtils.getPersistenceManager(pmf.getObject(), true);
    TransactionSynchronizationManager.bindResource(pmf.getObject(), new PersistenceManagerHolder(pm));

In the tearDown() we’ll just do the same thing the filter does after the request.

public void tearDown() {
	PersistenceManagerHolder pmHolder = 			(PersistenceManagerHolder)TransactionSynchronizationManager.unbindResource(pmf.getObject());
    PersistenceManagerFactoryUtils.releasePersistenceManager(pmHolder.getPersistenceManager(), pmf.getObject());

Once you have that class setup, you can extend it for your actual testing like this:

public class TestAccountService extends BaseSpringJunitTest {
	private AccountService accountService;
	public void testSubscribeNull() {
		accountService.subscribe(null, null);
		// check that subscribe behaved OK for null parameters...

That’s it, go forth and test!. App Engine sure does make local integration testing and developing a local copy of your app easy and painless – I suggest you try it.

PS: This is the first Java development related post I have written in a long time, apologies to the Magento developer readers out there, it may not be the last.

Posted on

Google App Engine billing changes and my website monitoring service

A long long time ago I wrote a simple little monitoring service on Google App Engine. It’s been happily ticking over for the last couple of years but with the recent changes to Google’s App Engine pricing, I’m going to have to make some changes.

This is a blog post for those who do use the service (there’s over 100 of you!) and may care about whether it is still monitoring your site. I sent this email to you all (sorry for the spam!) and will make any further updates on this site to save your inbox tears.

Hi, All

I’m Ashley, I made the website monitoring service you signed up for
here: – if you no longer use it, or
don’t care, please delete this email – I won’t be sending any more.

If you’re still with us so far, firstly sorry to spam all of you, but
there’s been some important changes by Google in the way they price
the App Engine platform, and apps like this monitor service, are not
free any more.

I’ll do my best to do a quick rewrite of the monitor in a way that can
keep the running costs at zero, but if I can’t, I’m going to have to
shut it down. If that does happen, I’ll gladly open source the code,
so that you can all run your own version of it, on your own app engine

As promised I won’t email any of you about this again, I know how
annoying emails like this are. If you use and care about the service
working, subscribe at for updates on the
future of the monitor, and the code if/when it is open-sourced.

Any questions, please just ask me.

Kind regards,

So either there’ll be an update about what the new service does and does not do, or there’ll be some code so you can run it yourself. D-Day is Nov 7.

I will just say, I’m still a big fan of App Engine, despite the changes – it’ll remain my platform of choice for future projects until something better comes along.