Laying Siege to Magento: Performance testing your Magento webstores

December 6, 2010

As I alluded to in my excuses for not posting in ages blog post the other day I am working on a new Magento hosting review for a company called MageMojo. In doing so I’m trying out the performance testing tool called Siege. Siege was mentioned several times at the Magento Developers Paradise, so I have been wanting to try it ever since.

This is a little article that summarizes the steps I took to do the testing, in case you want to test your own Magento stores.

Update: I have since created a tool that will allow you to run testing like this from the comfort of your own web browser. Check out the automated Magento performance testing at MageSpeedTest.com.

Get a Server

Being all the way over in New Zealand pretty much means performance testing Magento from here is out of the question. I fired up an Amazon EC2 small instance for the purposes of testing for a few hours, 20c – I’ve said it once, I’ll say it again, commodity computing is the future . I won’t bore you with a tutorial on starting an EC2 server, there are enough perfectly good guides already online.

You could optionally use a server you already have, or run this from your own computer at the office or home. It would arguably be a fairer test of latency, not running the tests in a production data centre!

Definitely do not run it from the server that is running Magento, that would be a poor test; a) because there would be no latency and b) because the server would be using resources to run the tests, and serve the webstore, that’s not very fair on it!

Install Siege

wget http://www.joedog.org/pub/siege/siege-latest.tar.gz
 
tar xvzf siege-latest.tar.gz
 
cd siege-2.71/
 
# if you're on a fresh ubuntu, get this package first
sudo apt-get install build-essential
 
./configure
 
make && sudo make install

Now you can begin a siege on your store, but first, an important warning…

If you use Siege on your production store, be careful it does not backfire.

Too many requests on your production server could easily make it unusable for other visitors, so be careful. Start small, with low numbers of concurrent users for short durations and keep an eye on your servers memory usage. You can halt an in-progress siege with ctrl+c at any time.

Oh and I’m sure the people who read my blog don’t need to be told this but; if you use Siege maliciously to _performance test_ other people’s websites (from multiple Amazon EC2 instances) that’s not performance testing anymore – that’s a denial of service attack, I strongly advise against it!

Run a Basic Siege Test

Now you can run your first Siege test. In this test we just hit one URL with some number of concurrent users (10) for some amount of time (1 minute):

siege -c10 -t1M http://www.google.com

Run a More Complex Test against Magento

First grab a bunch of categories and products from the sitemap.xml (you have a sitemap, right?) and put them into a urls.txt file.

#categories
curl http://yourmagentostore.com/sitemap.xml | sed 's/\<url\>/\<url\>\n/g' | grep 0.5 | sed 's/.*loc>\(.*\)<\/loc.*/\1/g' > urls.txt
#products
curl http://yourmagentostore.com/sitemap.xml | sed 's/\<url\>/\<url\>\n/g' | grep 1.0 | sed 's/.*loc>\(.*\)<\/loc.*/\1/g' >>urls.txt

And then run the test like this for 30 seconds with 25 concurrent users, adjust to suit:

siege -i -c25 -t30s -f urls.txt

Note: I bet a 100 million dollars there is a more elegant way to grab the urls from a sitemap. Someone post one in the comments so I can buy an island.

Looking at Results

Siege results look something like this:

Transactions:		         149 hits
Availability:		      100.00 %
Elapsed time:		       13.87 secs
Data transferred:	        1.26 MB
Response time:		        0.37 secs
Transaction rate:	       10.74 trans/sec
Throughput:		        0.09 MB/sec
Concurrency:		        4.02
Successful transactions:         149
Failed transactions:	           0
Longest transaction:	        0.42
Shortest transaction:	        0.36

The most import numbers to look for are Transaction rate and Response time they will give you an indication of how quickly your server is responding and how many connections it can handle per second.

I hope this little demo has been helpful. I’ll be following up with the full test results for MageMojo later this week, I also compared the Nexcess results – very interesting.

I’d be very interested to hear from anyone with tips and ideas for performance testing, likewise if you try this and have any trouble feel free to ask for help.

Edit: Pefromance…doh….
Edit: Updated Siege URL and version, thanks to Eric@MageMojo for letting me know.

Ashley

Posts

Hi! I'm Ashley Schroder, a Software Engineer from New Zealand - this is a collection of notes on my experiences with Ecommerce Web Development, particularly Magento Development.

Are Your Customers Getting The Magento Emails You Send?

MageSend Magento Email sending with Amazon SES

I have made a premium Magento Extension called MageSend. MageSend makes it simple and easy to send using Amazon's highly reliable SES email service.

The extension is $99, and comes with a complete money back guarantee, please check it out, I am confident it will resolve any email sending problems with your Magento store.

24 responses to Laying Siege to Magento: Performance testing your Magento webstores

  1. I just invested 99 million dollars in getting the world’s top coders to try and come up with a better way to extract the URLs from the sitemap. Your method held up as the best method. They held their heads in shame and admitted defeat. Please PayPal me the 100 million dollars asap, as the interest on the 99m is killing me already…

  2. Haha, I think we should wait a couple of days! There might be some other contender.

  3. Might be cheating, but what about good old php:
    php -r ‘$x=new SimpleXMLElement(file_get_contents(“http://domain.com/sitemap.xml”));foreach($x->url as $n) echo $n->loc.”n”;’

  4. Hi Ashley,

    Think i have seen that method somewhere before 😉

    using siege to identify caching mechanism and server environment is also possible by using the -g parameter.

    siege -g http://enterprise.procontractors.eu/

  5. @Vinai Apologies this took a while to show up – somehow got flagged as spam on the way through, I’m pleased to see such a great one liner! Thanks for stopping by.

  6. Inspired by this article I checked out siege more thoroughly. Thats how I stumbled over sproxy by the same author. It’s a proxy that saves all requests (GET and POST) in the siege url.txt format. That way it is possible to simulate user session with siege. The Sessions-ID needs to be passed as a GET parameter, though, Cookies don’t seem to work.

  7. hey how’s the testing result for the Magento Stores hosted on Amazon EC2? I’m using a Large instance.. 7GB of memory on a fedora box.

    my AVG response time is over 1 second.. after:

    1. fooman speedster
    2. compliation
    3. APC cache

    (now fooman’s not activated on the site, as I’m changing CSS atm)

    this is why I’m considering spending some money to get this:

    http://www.tinybrick.com/magento-modules/sales-tools.html

    you think it’s worth it? Or kristof heard you have a good speed extension coming out as well..

    cheers
    nathan
    ( 0508 NATHAN ) my free call

  8. @Nathan,
    if you can tune the PHP APC settings to apc.stat = 0 and apc.shm_size = 512 (thats Megabytes) you will have better results then with the Magento Compiler module (turn in off when using apc.stat = 0).
    Whenever you change a php file you will need to restart apache or flush the apc system cache by calling apc_clear_cache() though.

  9. hi thanks a LOT, there are some huge differences.. from 1 sec wait time to 500ms wait time..

    You just saved me $500 US…haha..

    Merry Christmas everyone.

    cheers
    nathan

  10. @Nathan @Vinai – uncanny timing! I have been doing some testing on different EC2 setups these last couple of days. Mainly I’m trying to set up a cluster that most of the time just has 1 large server in it, but can ramp up an extra one or two when the first is under load (like when you send out a newsletter, or get a blog review/coverage).

    I’m going to write up my setup instructions and publish more thorough results of the EC2 clustering side of things when I get a spare moment (mid 2012).

    In terms of optimization on the actual servers I did only the very basics (APC+more mysql memory). I’ll def. try your stat suggestion Vinai sounds like a quick win.

    My initial testing with 1-3 large instances behind a load balancer, with one of the 3 large instances being the ‘mothership’ mysql server:

    Response time:		        0.67 secs
    Transaction rate:	       19.42 trans/sec
    

    The good thing about this setup is that if you use the EC2 auto scaling, it could add up to say 10 larges and probably scale pretty linearly. The next improvement would be to pull out the mysql server but I have read that it does not improve performance by much so it’s probably not the bottleneck.

    Stay tuned for full setup and results of my experiments and please, keep the tips and results coming. It’s so hard to know if we’re on the right track in isolation.

    Cheers,
    Ashley

  11. Hi Ashley,

    try

    cat sitemap.xml | xpath -q -e “//loc/text()” > urls.txt

    for a nice way to extract urls from the sitemaps.

    Bye,

    Fabrizio

  12. Hi Fabrizio, very nice I haven’t had experience with xpath on the commandline. The current Magento specific algorithm grabs only specific priority URL’s – but I am seeing a need to get all URL’s so this may just be the best way.

    Thanks!

  13. Hi Ashley,

    grab urls with a specific priority with following command using xpath:

    curl –silent http://…/sitemap.xml | xpath -q -e “/urlset/url[priority=0.5]/loc/text()” > urls.txt

    Bye,

    Fabrizio

  14. Nice one Fabrizio!

  15. Hi – tool looks great, but once the pages are cached, aren’t you just testing the server, rather than Magento?

    Is there a way to start a session, add a product to the cart & checkout in order to test actual customer transaction time?

    thx!

  16. Currently writing an free extension that will produce profiles for siege based on visitors url profiles. It will be configurable in the admin. For instance creation of sessions like:

    Sesssion Name: Standart Checkout
    Url Path:
    Browse Catalog – Product View – Add to Cart – Checkout – Billing – Shipping – Shipping Method – Payment – Review – Place Order – ….

  17. Toby Hemmerling November 2, 2011 at 5:08 am

    @Ashley: Thanks for explaining the tool and for creating magespeedtest.com

    @Ivan: That would be huge for my team. Will you comment here when it’s ready?

  18. Hi Ashley,
    Great tool at http://www.magespeedtest.com.

    It says that the tests are done from 4 data centres on 3 continents – can you please tell us which continents? I looked through this post, couldn’t find the detail (I may have missed it). Like you, I’m in the southern hemisphere. So a US host can sometimes end up having slow latency from here.

    Thanks for your help,
    TIm

  19. Ashley, forget my last question – I just saw the answer!

  20. The answer for everyone else’s benefit:
    North America
    Europe
    Asia

    (and soon, another Europe and one in South America!)

  21. Ashley,

    One more question – can I use MageSpeedTest to test my site if it has a different domain to my email? In other words, can I test our Magento site while it’s still on the devserver (to see the impact of adding extensions etc) – rather than wait till the site is live?

    Example scenario:

    My email is email@mydomain.com
    Devserver is http://www.mydevserver.com

    Thanks,
    Tim

  22. Yes, you can test any, but they are rate limited and I may ask you to prove you own/run the server to keep testing it if your tests are sustained. I haven’t had to yet, mind you.

  23. Hi Ashley,

    I’m a premium member for the speed test and I keep getting several server time out errors when testing 20 users, 45 secs. Is this normal? Any ideas how this can be fixed?

    Thanks for your help
    AJ

    Example:
    ** Preparing 20 concurrent users for battle.
    The server is now under siege…
    [alert] socket: -869157120 select timed out: Connection timed out
    [alert] socket: -848177408 select timed out: Connection timed out

  24. I am searching for Magento Monitoring. Can you advise something?
    I want to track if magento is ready for selling LED stuff…