Tuesday, June 8, 2010

TurboGears and Amazon EC2 Benchmarking, Part 1: Methodology

This is going to be a multi-part series. As of right now, I've got at least three parts, and I may be adding another one in. I've just gathered a lot of data from Amazon, and the TurboGears EC2 images I made. Enough of it, in fact, that I'm not sure how to present all of it. For now, I'll explain how I gathered the data.

First, I signed up for Amazon's Relational Database Service (RDS). This allowed me to move the database off of my EC2 instances. It did subject me to the limits of their setup, and their network, but this provides two advantages:

  1. Web servers and database servers are typically separated in real world deployments. Using RDS allows me to have a similar configuration.
  2. This allows me to show the speed of TurboGears. It's a fairly safe bet that Amazon has tuned their RDS service very well. Therefore, any speed issues that occur are likely to belong entirely to TurboGears.

From there, I used the EC2 Amazon Machine Images I announced last week. I would start up an instance, create a quickstarted application, and then test against both the index page (which produces no database hits) and against the index page with authentication cookies, which did produce database hits.

That's the summary.

For 2.0.3, I did the following:
  1. Start up the instance of the TG 2.0.3 AMI I created. Use the correct one to get an instance size I've not yet measured.
  2. ssh into the machine
  3. Become the "turbogears" user
  4. Create a quickstart app named "tgtestapp"
  5. Run "python setup.py develop" to register the application
  6. Copy development.ini to tgapp.ini, as expected by the AMI
  7. Edit tgapp.ini: Set debug to False, and set the correct sqlalchemy.url to point to the RDS database.
  8. Use "paster setup-app tgapp.ini" to initialize the database
  9. Use "tglinker tgtestapp 0.1dev" to configure the remaining mod_wsgi files
  10. Restart Apache
  11. Visit the site in a browser to make sure the site is visible
  12. Run the commands I made for benchmarking, and record the requests per second value from them.

For 2.1b2, I did the following (very similar to 2.0.3):
  1. Start up the instance of the TG 2.1b2 AMI I created. Use the correct one to get an instance size I've not yet measured.
  2. ssh into the machine
  3. Become the "turbogears" user
  4. Create a quickstart app named "tgtestapp" using genshi templates
  5. Run "python setup.py develop" to register the application
  6. Copy development.ini to tgapp.ini, as expected by the AMI
  7. Edit tgapp.ini: Set debug to False, and set the correct sqlalchemy.url to point to the RDS database.
  8. Use "paster setup-app tgapp.ini" to initialize the database
  9. Use "tglinker tgtestapp 0.1dev" to configure the remaining mod_wsgi files
  10. Restart Apache
  11. Visit the site in a browser to make sure the site is visible
  12. Run the commands I made for benchmarking, and record the requests per second value from them.
  13. Destroy the current quickstarted app (rm -rf)
  14. Make a new one, using mako templates
  15. Run "python setup.py develop" to register the application
  16. Use "tglinker tgtestapp 0.1dev" to configure the remaining mod_wsgi files
  17. Restart Apache
  18. Visit the site in a browser to make sure the site is visible
  19. Run the commands I made for benchmarking, and record the requests per second value from them.
I repeated all of the steps (for both 2.0.3 and 2.1b2) for all Amazon EC2 instance sizes.

The commands I used for running the actual benchmarks are as follows:

  • ab -c 10 -n 400 http://localhost/
  • ab -c 10 -n 1000 http://localhost/
  • ab -c 10 -n 1000 -C 'tgtestapp=06c4a524f1a6c3196b1c491e95517c61c1770ef7efeaaacdcc29437c70cc8737438fc02d' -C 'authtkt="3a5c5890ea35f1bea327bf5b43123bbe4c0ef565manager!"' -C 'authtkt="3a5c5890ea35f1bea327bf5b43123bbe4c0ef565manager!"' http://localhost/
I have noticed a sometimes significant delay in mod_wsgi getting started serving the site. The first command overcomes that delay, and gets everything flowing as smoothly as possible. The second command focuses on raw rendering speed. The third looks at speed when the database server is in the mix, as it will generate hits.

Many people insist on multiple runs when doing benchmarks. To a degree, they are right. They are looking to know the absolute values, and benchmarking is inherently non-absolute. I did not do multiple runs, because I was looking for approximations of real world performance. Any one run will provide that.

Even those times when I said "Wait, that can't be right", and re-ran the numbers, I found the results were always within a few percentage points of each other. I quickly figured out that, for the needs of this article, anything more than one run would be superfluous.

You will note that the pages being retrieved are not simple "hello world" pages. The entire TurboGears stack gets tried out here. This is a full, real world example of what you can expect to see from TurboGears on Amazon's EC2 service. As such, multiple runs are of questionable value in this case (at best). If enough people are bothered by this, I'll do re-runs, and average out the results (or get the median, whichever), but I really do not feel it is necessary for this particular test.

Preparing the results for viewing on blogspot is actually taking a bit of time. I'll get them up in the next two days, so that everybody can see the numbers that I see.

Next: Requests per Second

No comments: