Just a quick little note. I took the work I had done in this article on siege testing Magento and basically made it into a little script you can run to test your own Magento store from the comfort of your web browser.
You can see the app in action at Magento Speed Test.
The features include:
- Testing for a specified amount of time with a specified number of concurrent users.
- Having your results emailed to you when they are ready.
- Optionally choosing to publish your results to the home page
- Tweet your results.
Note that you do need a standard Magento sitemap.xml in order for the testing to work properly. But you should have one anyway.
It’s still in beta mode – but I’d really love for people to try it out and give me some feedback and ideas. Here are a couple of features I think would be cool:
1) Run tests every 12 or 24 hours against various hosting company demo stores. That way we’ll get to see how they are performing and who is the fastest (for my little test at least).
2) Show a leader board for the home page, not just the 5 latest published results. I could add comments/notes so people can learn tips and tricks for speeding up Magento.
3) Less cool but important, I need to optimize the App with some memcaching – the Google App Engine Gods get angry when you run lots of un-cached queries.
But I’d appreciate hearing your feature ideas.
Used it, like it!
Apart from the fact I got terrible results, which will ruin my Christmas holiday as a perfectionist just can’t tolerate crap server performance.
A third party testing tool that doesn’t need setting up is great as performance testing otherwise can be difficult to get going.
And when this is out of beta, you can expect reliable testing, that is the same every time.
So nice tool indeed, thanks for making it available! I’d even pay a one-off fee to lower the 10 minute delay. As I now need to test some 100 combinations of server tweaks and speed tools, I’ll end up with a lot of waiting for the 10 minutes to pass.
I understand it’s to avoid DOS attacks of sorts so perhaps you can do something like Google Webmaster Tools and have us upload a unique html file or meta tag value to proof the server is ours, and then it allows more heavy testing.
Anyway, I always want more more more, don’t mean to be ungrateful! 🙂
Are there any guidelines or recommendations as to what is acceptable?
Perhaps you could rate the results good bad or ugly when they are published.
@J.T. Good idea to shorten up the 10 minute delay. The Google style ‘place an html file in the root of your store’ approach is probably the best.
The kicker is that once that file is there, anyone could initiate a test against you. I could make the file contents email specific – which would give a little security by obscurity. So the file contents+domain+email must all match in order to do unlimited tests.
Maybe the correctly coded html file could be what is sold in a bid to make money from this little experiment!
I’ll have a tinker with it this week, but definitely will take your feedback on board.
@Paul – Good question, it’s a little bit hard to mark a single test as good or bad – if you’re paying $20/month then the definition of good is very different to if you are paying $500/month.
Here’s the best one I’ve seen, one to strive for. I won’t say who it’s from but they shared it with me via Skype yesterday. I’m sure when they are ready, they’ll publish their results.
This is an enterprise solution though, you’ll struggle to get performance like that for $50/month!
How about when people elect to publish their results they can optionally include how much they are paying for the hosting. That way I could do some sort of graph that shows price/performance and then others can see where they sit on the graph.
I’ll add it to the feature ideas list – it’d be good to put results in context.
Good ideas guys, please keep them coming.
Perhaps the HTML verification file could contain the actual number of tests, or the time frame, you allow.
Or a simple XML file:
AllowFrom DateTime
AllowTo DateTime
AllowTestsPerHour 10
But that’s making it quite elaborate already. The default behaviour could simply be as you have it now with some optional advanced functionality for us nerds.
The performance versus hosting fees is a neat though potentially depressing idea. If sub one sec response and over 20 transactions is possible, I sure have my work cut out!
Happy 2011 by the way!
Hi, J.T. Happy New Year to you too!
That’s a good idea having the file dictate the thresholds. It could also have:
NotifyEmail a@b.com
NotifyThreshold 10
So that it emails a@b.com if 10 tests have been run or something (logging the IP that initiated them).
I need to clone myself…