Archives For August 2011

Long time, no post. Sorry about that. I’ve been very busy working on an app for aggregating ecommerce orders on your mobile device. Check out the sneak peek from twitter. I’m also looking for some private beta testers so if you’re keen to try it out and willing to give us some helpful feedback, flick me an email.

Anyway this is just a quick update on some features (arguably fixes) I added tonight for The first is support for multiple sitemap files in a sitemap.xml index file. The second is support for gzipped sitemap files (but only if they are references in the sitemap index).

I have also implemented a couple of measures to prevent the number of links getting out of hand. Firstly for very large sitemaps once I have 10000 links for a single site, I stop collecting them. Secondly if a site does have over 10000 links, I cache them instead of fetching them on each and every speed test.

This caching has been implemented to preserve system resources, now that the servers are monitoring and testing hosts regularly there’s a decent amount going on and I don’t want the test performance to suffer due to downloading millions of URLs from sitemaps unnecessarily.

There are still some outstanding issues with – if you have emailed me about them be assured I will get to them!

Stay tuned for some posts on my experiences developing the mobile web app, using jQuery mobile and Google App engine.