Wednesday, June 9, 2010

TurboGears and Amazon EC2 Benchmarking, Part 2: Requests per Second

That all important question has been asked, and is now answered: How fast is TurboGears? I started this series yesterday when I described my methodology. I conclude soon with my analysis of what I did, and what I found.

The numbers below are not what you might expect to see. Many people, when doing benchmarks, do the equivalent of writing "hello world", and say "That's how fast my web framework is." No, it's not. It's considerably slower than that. I've seen far too many benchmarks like that. I'll not link to any of them, as I don't wish to encourage more like that. I want to get an idea of how a full featured application will perform, not some dinky "hello world".

When looking at these numbers today, keep one thing in mind: These are not "hello world" requests. These requests include templating, caching, sessions, database requests and transactions, authentication, authorization, and all of the other little pieces that make an actual web application into an application versus something that manages to simply print "hello world" on the screen.

I'll be back tomorrow with some analysis, both on the numbers, and my own biases in the whole process. For now, look these over. I think you'll find them interesting.

In the below table, the columns have the following meanings:
  • EC2 Instance Name: This is the marketing name of the instance as visible on Amazon's page about EC2.
  • EC2 Instance Type: This is the instance name that you will see when you create a new instance on EC2.
  • TG Version: The version of TurboGears that produced these results
  • Genshi/Mako: When running these tests, I varied the template engine between these two where possible. This column indicates the template engine that produced these results.
  • Index or Database: The default index page for a quickstarted application does not hit the database. This column indicates whether I was forcing a database hit, or just hitting the index page.
  • Req/Second: How many requests per second were being processed. I truncated the results (so nothing after a decimal).


TurboGears on EC2 Results
EC2 Instance NameEC2 Instance TypeTG VersionGenshi/MakoIndex or DatabaseReq/Second
Small Instancem1.small2.0.3GenshiIndex24
Small Instancem1.small2.0.3GenshiDatabase13
High-CPU Mediumc1.medium2.0.3GenshiIndex100
High-CPU Mediumc1.medium2.0.3GenshiDatabase64
Largem1.large2.0.3GenshiIndex86
Largem1.large2.0.3GenshiDatabase44
Extra Largem1.xlarge2.0.3GenshiIndex148
Extra Largem1.xlarge2.0.3GenshiDatabase80
High-Memory Extra Largem2.xlarge2.0.3GenshiIndex171
High-Memory Extra Largem2.xlarge2.0.3GenshiDatabase94
High-Memory Double Extra Largem2.2xlarge2.0.3GenshiIndex284
High-Memory Double Extra Largem2.2xlarge2.0.3GenshiDatabase171
High-Memory Quadruple Extra Largem2.4xlarge2.0.3GenshiIndex388
High-Memory Quadruple Extra Largem2.4xlarge2.0.3GenshiDatabase229
High-CPU Extra Largex1.xlarge2.0.3GenshiIndex217
High-CPU Extra Largex1.xlarge2.0.3GenshiDatabase123
EC2 Instance NameEC2 Instance TypeTG VersionGenshi/MakoIndex or DatabaseReq/Second
Small Instancem1.small2.1b2GenshiIndex28
Small Instancem1.small2.1b2GenshiDatabase14
High-CPU Mediumc1.medium2.1b2GenshiIndex124
High-CPU Mediumc1.medium2.1b2GenshiDatabase58
Largem1.large2.1b2GenshiIndex96
Largem1.large2.1b2GenshiDatabase41
Extra Largem1.xlarge2.1b2GenshiIndex168
Extra Largem1.xlarge2.1b2GenshiDatabase70
High-Memory Extra Largem2.xlarge2.1b2GenshiIndex196
High-Memory Extra Largem2.xlarge2.1b2GenshiDatabase86
High-Memory Double Extra Largem2.2xlarge2.1b2GenshiIndex349
High-Memory Double Extra Largem2.2xlarge2.1b2GenshiDatabase150
High-Memory Quadruple Extra Largem2.4xlarge2.1b2GenshiIndex468
High-Memory Quadruple Extra Largem2.4xlarge2.1b2GenshiDatabase194
High-CPU Extra Largex1.xlarge2.1b2GenshiIndex283
High-CPU Extra Largex1.xlarge2.1b2GenshiDatabase126
EC2 Instance NameEC2 Instance TypeTG VersionGenshi/MakoIndex or DatabaseReq/Second
Small Instancem1.small2.1b2MakoIndex115
Small Instancem1.small2.1b2MakoDatabase22
High-CPU Mediumc1.medium2.1b2MakoIndex480
High-CPU Mediumc1.medium2.1b2MakoDatabase93
Largem1.large2.1b2MakoIndex282
Largem1.large2.1b2MakoDatabase63
Extra Largem1.xlarge2.1b2MakoIndex415
Extra Largem1.xlarge2.1b2MakoDatabase94
High-Memory Extra Largem2.xlarge2.1b2MakoIndex630
High-Memory Extra Largem2.xlarge2.1b2MakoDatabase131
High-Memory Double Extra Largem2.2xlarge2.1b2MakoIndex976
High-Memory Double Extra Largem2.2xlarge2.1b2MakoDatabase220
High-Memory Quadruple Extra Largem2.4xlarge2.1b2MakoIndex1354
High-Memory Quadruple Extra Largem2.4xlarge2.1b2MakoDatabase274
High-CPU Extra Largex1.xlarge2.1b2MakoIndex776
High-CPU Extra Largex1.xlarge2.1b2MakoDatabase181

No comments: