Tuesday, June 3, 2014

Server Load Testing using Gatling: Some Lessons Learned

I had to load test a server using Gatling to see how many requests per second it could handle. The server built up a cache over time as it stored the input parameters, so that it would not have to look them up on subsequent identical queries. This led to a development I had not anticipated.

Gatling's http behavior is to issue a new request immediately upon receiving a response, unless there is a deliberate pause inserted. My test did not originally include such a pause, as I thought I would find the max rate through stress. In the case of cached queries, the server responded in single-digit milliseconds, which led to Gatling spawning a huge number of requests per second, overwhelming the server. Once I understood this, I inserted a pause into the Gatling scenario. I used a 1 second pause, so that each Gatling "user" could only send at most one request per second. Thus, the overall throughput rate is measured by the number of users: 1000 users at 1 request per second each would mean a total requests-per-second rate of 1000. In this way, I had to keep upping the number of users to find the limit, which was somewhat backwards than what I had originally intended.

Along the way I learned some other useful techniques, such as how to create and use dynamic values for query parameters. I wanted a different random value each time. To do this I used "session attributes". You preface one "exec" with another which calls a function.

exec(func)
      .exec( // http stuff
)

You could also use a "feeder" function to pre-generate such values or a feeder with .csv files for static data.

Here is the sample source code (using Gatling 2.0 jar files)  (SampleScenario.scala)

package basic

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import io.gatling.jdbc.Predef._
import io.gatling.http.Headers.Names._  // to become HeaderNames in later versions of 2.x
import scala.concurrent.duration._
import bootstrap._  // to be deprecated in later versions of 2.x, to be commented out then
import scala.util.Random
import io.gatling.core.feeder.Feeder


trait WWWLoadCommandLineArgsSample {
  val users = Integer.getInteger("users", 1)
  val rampup = intToFiniteDuration(Integer.getInteger("rampup", 0))
  val random = new scala.util.Random
  var test_duration = intToFiniteDuration(Integer.getInteger("test_duration", 60))
}

class BasicSimulationGatling2 extends Simulation with WWWLoadCommandLineArgsSample{

  val chars = ('A' to 'Z')
  val quarterUsers = (users.toFloat / 4.0).toInt
  val halfUsers = (users.toFloat / 2.0).toInt

  val chooseRandomACodes = exec((session) => {
    val code1 = (random.nextInt(898) * 100) + 10000
    val code2 = code1 + 100
    val a_code: String = "[" + code1.toString() + "-" + code2.toString() + "]"
    session.set("a_code", a_code)
  })

  val chooseRandomBCodes = exec((session) => {
    val b_code : String = chars(util.Random.nextInt(chars.length)).toString + random.nextInt(10).toString() + chars(util.Random.nextInt(chars.length)).toString
    session.set("b_code", b_code)
  })

  val chooseRandomCCodes = exec((session) => {
    val c_code : String = random.nextInt(9).toString()  + random.nextInt(9).toString()  + chars(util.Random.nextInt(chars.length)).toString
    session.set("c_code", c_code)
  })

  val httpConf = http
    .baseURL("http://www.the_server.com")
    .acceptCharsetHeader("ISO-8859-1,utf-8;q=0.7,*;q=0.7")
    .acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
    .acceptEncodingHeader("gzip, deflate")
    .acceptLanguageHeader("fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3")
    .disableCaching

  val scn_a = scenario("A Codes")
    .during(test_duration) {
    exec(chooseRandomACodes)
      .exec(
        http("a_code")
          .get("/")
          .queryParam("aCode", "${a_code}")
          .check(status.is(200))
      ).pause(1000 milliseconds)
  }

  val scn_b = scenario("B Codes")
    .during(test_duration) {
    exec(chooseRandomACodes)
      .exec(
        http("b_code")
          .get("/")
          .queryParam("bCode", "${b_code}")
          .check(status.is(200))
      ).pause(1000 milliseconds)
  }

  val scn_c = scenario("C Codes")
    .during(test_duration) {
    exec(chooseRandomCCodes)
      .exec(
        http("c_code")
          .get("/")
          .queryParam("cCode", "${c_code}")
          .check(status.is(200))
      ).pause(1000 milliseconds)
  }

  // divide the scenarios among the total users, share the rampup, duration and protocolConfig
  setUp(scn_a.inject(ramp(halfUsers) over (rampup)), // ramp to become 'rampUsers' in later versions of 2.x
    scn_b.inject(ramp(quarterUsers) over (rampup)),
    scn_c.inject(ramp(quarterUsers) over (rampup)))
    .protocols(httpConf)
}

To Run:

export JAVA_OPTS="-Dusers=1500 -Dramp=120 -Dduration=3600"
./gatling.sh -s basic. BasicSimulationGatling2 -sf user-files/simulations/basic

Where gatling.sh =

#!/bin/sh
OLDDIR=`pwd`
BIN_DIR=`dirname $0`
cd ${BIN_DIR}/.. && DEFAULT_GATLING_HOME=`pwd` && cd ${OLDDIR}

GATLING_HOME=${GATLING_HOME:=${DEFAULT_GATLING_HOME}}
GATLING_CONF=${GATLING_CONF:=$GATLING_HOME/conf}

export GATLING_HOME GATLING_CONF

echo "GATLING_HOME is set to ${GATLING_HOME}"

JAVA_OPTS="-server -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms512M -Xmx512M -Xmn100M -Xss2M -XX:+HeapDumpOnOutOfMemoryError -XX:+AggressiveOpts -XX:+OptimizeStringConcat -XX:+UseFastAccessorMethods -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSClassUnloadingEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly ${JAVA_OPTS}"

CLASSPATH="$GATLING_HOME/lib/*:$GATLING_CONF:${JAVA_CLASSPATH}"

java $JAVA_OPTS -cp $CLASSPATH com.excilys.ebi.gatling.app.Gatling $@

Tuesday, November 5, 2013

Simplest Apache/PHP REST API

On Fedora (17 in my case)
with apache/php installed and running

1) enable URL mapping and map the URL to the PHP target file for REST processing

edit
/etc/httpd/conf/httpd.conf
  make sure this line is in the file:
 LoadModule rewrite_module modules/mod_rewrite.so

and then add
   RewriteEngine On
   RewriteLogLevel 3
   RewriteRule ^/rest_test/simple/(.*)$ /var/www/siab/src/rest_test.php?num=$1


2) restart the service:
    sudo service httpd restart



3) create the PHP API target file

/var/www/siab/src/rest_test.php looks like this:

<?php
$num = $_GET['num'];
$myData = array('results' => $num);
echo json_encode($myData);
exit(0);
?>

in browser: localhost/rest_test/simple/3
outputs:
   {"results":"3"}


4) of course in reality you want to do a lot more than this (like return status codes, do backend processing, handle POST as well as GET) but these are the very basics in a nutshell

-----------------------

for optional parameters:

RewriteRule ^/rest_test/simple/([^/]+)/?([^/]*)/?([^/]*)/?$ /var/www/siab/src/rest_test.php?num=$1&num2=$2&num3=$3

<?php
$num = "";
$num2 = "";
$num3 = "";

if (isset($_GET['num']))
   $num = $_GET['num'];
if (isset($_GET['num2']))
   $num2 = $_GET['num2'];
if (isset($_GET['num3']))
   $num3 = $_GET['num3'];
$myData = array('num' => $num, 'num2' => $num2, 'num3' => $num3);
echo json_encode($myData);
exit(0);
?>

Tuesday, July 17, 2012

Two Python Open Source Projects for Test

I've recently added two open source projects to github, useful for software testing in python.

One is a general purpose test automation framework, suitable for running any and all tests. I've use it for web testing, api testing, serial port/firmware testing, network traffic scanning, tv remote control testing, game testing - all sorts of stuff

You can find it on github as pytaf26

The other is an abstraction layer for Selenium using Python bindings. I've found - to my consternation - that some things work better in Selenium RC than they do in WebDriver, and vice versa. In fact, in my current work I've had to switch between the two quite frequently. To ease the pain, I wrote an abstraction layer - two classes that use the same method signatures - so that  I only have to write the tests once, and pass along a command-line switch to determine which version of Selenium to use. The details of each version lie under the hood of the easy-to-remember api methods.


You can find it on github as py-selenium-layers

Tuesday, April 10, 2012

Pytaf

My new open source project - PYTAF (Python Test Automation Framework) - is in the process of being announced on Freecode.com. The project can be accessed directly through GitHub. It's a distillation of the various test harnesses I've developed over the past several years, beginning in Java, polished in Ruby, until finally fully formed in Python. This version is actually in Python 3, which I'm not using yet in everyday work. It's not much different, but developing this project was partly a mechanism for learning about it.


I'm happy to put it out there in the world, as happy as putting out any of my ebooks or earlier open-source projects. If one person finds it useful, that will be 'abundance'.

announcement also here on opencode

Thursday, April 5, 2012

Porting PTest from Python 2.6 to Python 3.0


work-in-progress ... experiment porting my python automation test framework from Python 2.6 to Python 3.0 (on Windows 7, it should be noted)

python.org/ftp/python/3.2.2

---------------------
import changes
---------------------
  import httplib becomes import http.client as httplib
  import urllib2 becomes import urllib as urllib2
  import urllib also becomes import urllib as urllib

  import MySQLdb - no such module. download pymsql instead and, after installing, import pymysql

  lxml
   from http://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml, lxml-2.3.4.win-amd64-py3.2.‌exe
    failed to install - couldn't find python3.2 in the freaking registry
  so ... not using lxml  (BeautifulSoup is way better anyway)

  import thread becomes import _thread

BeautifulSoup 4 - https://groups.google.com/forum/#!msg/beautifulsoup/VpNNflJ1rPI/sum07jmEwvgJ
and after installing
from BeautifulSoup import BeautifulSoup becomes   from bs4 import BeautifulSoup

import Queue becomes import queue

--------------------------
execution changes
--------------------------


#set the right TEST_HOME
set TEST_HOME=C:\Users\tlichtenberg\workspace\tlichtenberg_win\QA\ptest3

# use the right python
c:\python32\python ptest3.py -c api_config.json -t test_api

---------------------
code changes
---------------------

print statements are now functions
     print "this is my statement"
becomes
     print("this is my statement")

 MySQLdb.Connection(... = pymysql.connect(...

thread.get_ident() becomes _thread.get_ident()

http response.read() returns a bytes object, not a string. so it needs to be
response.read().decode() to turn it into a string

for k,v in dict.iteritems():
   print k,v
becomes
for (k,v) in dict.items():
    print(k,v)


--------------
unicode
________


all text is unicode
  u"\xe9" no longer supported
  backslashes in strings are now just backslashes, not escapes

? not sure about this yet ...
instead of u'\x80abc'.encode('utf-8')
                  b'\x80abc'.decode("utf-8", "replace")




Friday, January 20, 2012

Basic Django on Apache2

My Django project is called 'automation' and lives in the directory /home/tlichtenberg/workspace/ptest/django/src/automation

It has CSS files in the regression/static/styles directory under the home project. I needed to add some stuff to Apache to allow it to find the files it needs to run the Django project

--------------------------------

My Apache2 httpd.conf:


WSGIScriptAlias / /home/tlichtenberg/workspace/ptest/django/src/automation/django.wsgi

AliasMatch ^/([^/]*\.css) /home/tlichtenberg/workspace/ptest/django/src/automation/regression/static/styles/$1

Alias /static/ /home/tlichtenberg/workspace/ptest/django/src/automation/regression/static/

<Directory /home/tlichtenberg/workspace/ptest/django/src/automation/regression/static>
Order deny,allow
Allow from all
</Directory>

<Directory /home/tlichtenberg/workspace/ptest/django/src/automation>
Order deny,allow
Allow from all
</Directory>

------------------------------

The django,wsgi file is critical for setting the path and connecting Django to Apache. It looks like this:


import os
import sys

path = '/home/tlichtenberg/workspace/ptest/django/src/automation/'
if path not in sys.path:
   sys.path.append(path)

os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

------------------------------------------

The 'settings' defined by django,wsgi points to the Django project's 'settings.py' file, which defines a lot of the paths that Django needs. Also important is the 'urls.py' file, like settings.py, it is a standard for all Django projects and defines all the URLs associated with the web applications being served by Django. For example:

settings.py:

  STATIC_ROOT = '/home/tlichtenberg/workspace/ptest/django/src/automation/regression/static/'
  STATIC_URL = '/static/'
  ROOT_URLCONF = 'urls'
  INSTALLED_APPS = (
    'my_project',
    'my_other_project',
 )

urls.py
   (r'^my_project/$', 'my_project.views.my_project_main'),
defines that any url beginning with 'my_project' (e.g. http://localhost/my_project) will direct the code to the my_project_main method in the project's views.py file.

---------------------------------------------------


 


Pyjamas on Django for Web Automation


I've been working on developing a web application for use in automating some product testing. The commands being implemented are communicated via HTTP, so there isn't strictly a need for a browser interface just to test the api's, but certain functionality requires doing stuff on the device, then going to a different website and verifying some side effects (e.g. purchases recorded, emails sent, etc ...) Using a web application for these cases lets me do some automation using Selenium.

When I first thought about the design, I assumed I would do a Java Servlet or Java Server Pages, but since I've been mostly working in Python recently, I thought I'd see if there was anything I could do that way. Sure enough, I came across a project called Pyjamas, a Python port of Google Web Toolkit. I was interested right away, because not only was it Python, but it could run in Django, and I was already running a Django server on Apache2 on Ubuntu. This web app would theoretically plug right in. And it did, after some minor bumps in the road.

Pyjamas is a sort of miracle wonder drug, where you create your UI in Python, and then "build" it into JavaScript/HTML/Ajax using its 'pyjsbuild' tool. So you can make real-world web apps without having to look under the hood, so to speak. It provides a simple GUI kit, similar to Java AWT 1.0 - primitive but useful. If I was a better UI designer, the above would look a lot nicer, but since I'm primarily a backend coder, this will just have to do :}

Codewise, you import a lot of Pyjamas modules:
   from pyjamas.ui.TextBox import TextBox

and then go about building your UI, creating panels and sticking widgets into them. I won't go into great detail here. There are a lot of excellent examples in the Pyjamas download and elsewhere, but basically, something like this to get going ...

  class RemoteController:
     
    def onModuleLoad(self):
        self.remote = DataService()    
        dockPanel = DockPanel()
        ...
       # create all your widgets in here, add them to you main panel,
       # and then add the main panel to the RootPanel object
       self.myTextBox = HTML()
       self. myTextBox.setID('my_text_box')
       dockPanel.add( myTextBox )
       RootPanel().add(dockPanel)

The DataService class is interesting - this is what lets you connect into functions running on Django, and mine looks like this:
 from pyjamas.JSONService import JSONProxy
 class DataService(JSONProxy):
    def __init__(self):
        JSONProxy.__init__(self, "/services/", ["method_one", "method_two"])


In my Django views.py, I just have to define those methods:
  from django.pimentech.network import * # use the pimentech module described here to define the JSON service and decorator

  service = JSONRPCService()

 @jsonremote(service)
 def method_one(request):
    print >> sys.stderr,  'method_one called"
    request.session['ip'] = text # set the ip as specific to this session
    ecp.set_url(text)
    return "put this into my text box, please"

now in your Django urls.py you define the /services/ path used by the DataService class above:
  (r'^services/$', 'YOUR_PROJECT.views.service'),

when your method returns from Django, you handle it in your Pyjamas class like this:

 def onRemoteResponse(self, response, request_info):
        try:    
            if request_info.method == 'method_one':
                self.myMyTextBox.setText(response)

Other GUI stuff should be familiar if you've done any similar programming - with clickListeners, mouseListeners, keyListeners and so on.

finally, have your main and have it call onModuleLoad:

if __name__ == "__main__":
    app = RemoteController()
    app.onModuleLoad()

I have myTextBox an id just like Ajax would. I did this for Selenium. My Selenium test could now find the widget By.Id or by xpath using the id:
    text = selenium.get_text("//*[@id='my_text_box']")

  For Selenium you usually want to be able to get text out of text fields but the PyjamasTextBox and TextArea widgets don't actually put the text into the HTML they render. Instead, I used an HTML() widget. It's like a Label and DOES embed the text in the web page so you can get to it from Selenium.

Some other stuff:

CSS - It's nice to set the background colors of your various widgets, and you can do that using inline CSS  in the Python code like this:
   from pyjamas import DOM
   DOM.setStyleAttribute(myWidget.getElement(), "background-color", "#8fbc8f")

 If you have multiple concurrent users, you want your web app to be session based. This is done, in Django, by adding a few things to your settings.py. I also had some issues with CSRF so I needed to disable that (you may not want to, depending on your security concerns. I had none in my test environment)

   MIDDLEWARE_CLASSES = (
    'regression.middleware.DisableCSRF',            
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.middleware.csrf.CsrfResponseMiddleware',
    'django.middleware.cache.UpdateCacheMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.cache.FetchFromCacheMiddleware',
)

 CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache',
        'LOCATION': 'c:/django_cache',
    }
}

CACHE_MIDDLEWARE_ALIAS = "default"
CACHE_MIDDLEWARE_SECONDS = 600
CACHE_MIDDLEWARE_KEY_PREFIX = ""