Testing Plone Sites by Using FunkLoad

Using FunkLoad to test Plone sites becomes very easy because of the following products:

Here are excellent blogs talking about using FunkLoad with Plone:

• How to Write FunkLoad Testing in Few Minutes: http://blog.redturtle.it/redturtle-blog/how-to-write-funkload-test-in-few-minutes
• New collective.funkload releases: http://blog.redturtle.it/redturtle-blog/2010/05/04/new-collective.funkload-releases
• Martin Aspeli talked about Funkload for Plone site in is Blog: http://www.martinaspeli.net/articles/tools-for-a-successful-plone-project

collective.loadtesting

As it mentioned on README, it aims to provide standardized Plone performance benchmarking for core developers, product developers and integrators.  It provides a baseline performance benchmark for core Plone development.  It also allows you to write your own loadtests specific to your application.

Some tweaking in buildout.cfg

We need force buildout to use specific versions for zope.testing and funkload:

[versions]
zope.testing = 3.9.5
funkload = 1.11.0

zc.buildout will automatically pick version 3.10 for zope.testing and 1.13.0 for funkload.  However the zope.testing verion 3.10 move out testrunner to a separate module zope.testrunner and funkload version 1.13.0 removed function set_running_flag (http://svn.nuxeo.org/trac/pub/changeset/54211).  Both will cause errors when we execute funkload testing.

collective.coreloadtests to load test CacheFu

The buildout config file (cachefu-funkload-1.11.0.cfg) is for creating CacheFu instance and CacheFu PloneSite so we can execute all test cases from coreloadtests on two comparable Plone sites: default Plone site and Plone site with CacheFu enabled.  The following is what I did to load test CacheFu:

• install funkload and make sure it is working properly
• check out collective.loadtesting: 
  $ svn co http://svn.plone.org/svn/collective/collective.loadtesting/trunk loadtesting
  $ cd loadtesting
• download
  $ wget http://plonexp.leocorn.com/xp/plonedemo/cachefu-funkload-1.11.0
• execute bootstrap:
  $ python bootstrap.py
• run build out:
  $ bin/buildout -c cachfu-funkload-1.11.0.cfg
• Run the FunkLoad Monitor
  

  $ bin/fl-monitor-ctl etc/monitor.conf start

• testing default Plone site and generate benchmark report:
  

  $ bin/instance start
  $ bin/fl-run-bench -s collective.coreloadtests -t test_ReadOnly --cycles=1
  $ bin/fl-run-bench -s collective.coreloadtests -t test_ReadOnly
  -- this will generate a bench xml file in folder var/funkload for the test result
  $ bin/fl-build-report --html var/funkload/ReadOnly-bench-20100909T102115.xml
  -- this will generate a html report in current folder with name test_ReadOnly-20100909T102115

• testing CacheFu Enabled Plone Site and generate benchmark report:
  

  $ bin/cachefu-instance
  $ bin/fl-run-bench -s collective.coreloadtests -t test_ReadOnly --cycles=1
  $ bin/fl-run-bench -s collective.coreloadtests -t test_ReadOnly
  -- this will generate a bench xml file in folder var/funkload for the test result
  $ bin/fl-build-report --html var/funkload/ReadOnly-bench-20100909T141301.xml
  -- this will generate a html report in current folder with name test_ReadOnly-20100909T141301

• Generate the differential report for default Plone site bench report and CacheFu enabled Plone site bench report:
  

  $ bin/fl-build-report --diff test_ReadOnly-20100909T102115 test_ReadOnly-20100909T141301

The following link is the differential report generated on my local machine:

http://www.leocorn.com/funkload-reports/diff_ReadOnly-20100909T_141301_vs_102115/index.html

As we can see from the difference, we could get significant performance improvement by just enable CacheFu for a Plone site.