/bin/recykl

Working on performance tests and tuning is an interesting and challenging job. One has to prepare tests, execute and also evaluate them. The preparation and execution is not always easy, but it is manageable. The evaluation of the tests comes with an interesting problem. The problem is called data visualization.

I was testing a system spread across several servers and there were several technologies involved (Microsoft perfmon, Apache jMeter) in the testing and monitoring. Therefore, the visualization was a challenge.

It is not a problem to put all the statistics into one spreadsheet and try to create a chart. BUT there is usually a lot of tests and the preparation of each chart takes ages. Secondly, the traditional spreadsheets are good to show sales for four regions, but they have problems with hundreds or thousands of data samples. The chart looks like a saw (in MS Excel) or is very CPU intensive (OpenOffice.org). So, automation and a good visualization technique is needed.

I selected gnuplot for the visualization. It is fast, it creates loads of different charts and it is managed from a script console… so, automation is possible. In my approach, the preparation of data files and generation of the gnuplot script is done using ruby scripting language. The resulting chart is nice and its generation took me just a few seconds, including deciding what should be displayed.

For those with no patience, the script is available for download here.

For the others, here is a short summary of what the script is doing for you. It …

• Takes several CSV input files as parameters. Thus you can have a chart with data from different servers.
• Lists all the columns that are in the given CSV file.
• Gives you the possibility to select the columns of interest.
• Allows you to specify time format of the timestamp in the file. Default is set to the perfmon format.
• Scales the data to show all of the data sets in the same chart.
• Generates a gnuplot script for you. You can then customize the script to your needs.

How it works

0. Create a “temp” directory in the directory where you run the script.

1. Run

ruby process.rb  SERVER3.csv

2. The script then lists all the columns from the CSV file

Select fields (time first, separated by space) from file SERVER3.csv
0 : "(PDH-CSV 4.0) (Central Europe Standard Time)(-60)"
1 : "\\SERVER3\LogicalDisk(C:)\Disk Bytes/sec"
2 : "\\SERVER3\LogicalDisk(D:)\Disk Bytes/sec"
3 : "\\SERVER3\Memory\Available MBytes"
4 : "\\SERVER3\Memory\Page Faults/sec"
5 : "\\SERVER3\Network Interface(HP NC7782 Gigabit Server Adapter)\Bytes Total/sec"
6 : "\\SERVER3\Network Interface(MS TCP Loopback interface)\Bytes Total/sec"
7 : "\\SERVER3\PhysicalDisk(0 C:)\Disk Bytes/sec"
8 : "\\SERVER3\PhysicalDisk(1 D:)\Disk Bytes/sec"
9 : "\\SERVER3\Process(java)\Page Faults/sec"
10 : "\\SERVER3\Processor(0)\% Processor Time"
11 : "\\SERVER3\Processor(1)\% Processor Time"
12 : "\\SERVER3\Processor(2)\% Processor Time"
13 : "\\SERVER3\Processor(3)\% Processor Time"
14 : "\\SERVER3\Processor(4)\% Processor Time"
15 : "\\SERVER3\Processor(5)\% Processor Time"
16 : "\\SERVER3\Processor(6)\% Processor Time"
17 : "\\SERVER3\Processor(7)\% Processor Time"
18 : "\\SERVER3\Processor(_Total)\% Processor Time"
...
36 : "Server monitoring"
Enter your options:

3. Enter the fields you want in the chart. The timestamp field needs to be the first one; fields are separated by spaces.

0 4 9 18

4. Specify the time format. The default is set to perfmon (US English Windows). Press enter, if you want to use the default.

Specify time format. Put HH:mm:SS into round brackets. Default is: DD/MM/YYYY (HH:mm:SS).XXX
DD/MM/YYYY (HH:mm:SS).XXX

5. The script is generated.

gnuplot script created. Generate a picture with the following command
pgnuplot script.plt

6. Edit the script if necessary. If you do not edit the script, you will get a chart like the one above.

set terminal png
set output "chart.png"
set xdata time
set timefmt "%H:%M:%S"
set format x "%H:%M"
set yrange [0:100]
cd "temp"
plot "SERVER3.csv.tmp" using 1:(0.01*$2) with lines title "0.01 * \\\\SERVER3\\Memory\\Page Faults/sec" smooth bezier, \
"SERVER3.csv.tmp" using 1:(0.01*$3) with lines title "0.01 * \\\\SERVER3\\Process(java)\\Page Faults/sec" smooth bezier, \
"SERVER3.csv.tmp" using 1:(1*$4) with lines title "1 * \\\\SERVER3\\Processor(_Total)\\% Processor Time" smooth bezier

7. Generate the chart with the following command

pgnuplot script.plt

8. Check file chart.png

Limitations

• The script is very simple and definitely will not work for all data files. Nevertheless, for an experienced ruby programmer it should not be a problem to modify it.
• Hopefully there are not many bugs in there. If you find some, give me a note (comment under this article is appreciated).

I hope you will find the script useful.

I just made a bad experience with ruby I would like to share.

I wanted to upgrade my SuSE Linux 9.1 to version 10. Failed… during the upgrade process my machine ended up in a strange state. I was not able to boot neither from hard drive nor from Windows XP installation disk. I had to use my old Win 2000 installation CD to make it running.

As a consequence, I had to recreate the whole development environment. After few hours of work I started my application…

Before the installation it was running without any problem. After the environment was recreated, I was not able to render some views. It showed me strange syntax errors:

ActionView::TemplateError (compile error
./script/../config/../app/views/story/_showing_tags.rhtml:4: syntax error
_erbout.concat "	"
                	  ^) on line #4 of app/views/story/_showing_tags.rhtml:
…
…
…
_erbout.concat "                        ";  end ; _erbout.concat "\n"
_erbout.concat "</div>\n"
_erbout
end
Backtrace: ./script/../config/../app/views/story/_showing_tags.rhtml:4:in `compile_template'
c:/win32app/ruby/lib/ruby/gems/1.8/gems/actionpack-1.12.5/lib/action_view/base.rb:307:in `compile_and_render_template'
c:/win32app/ruby/lib/ruby/gems/1.8/gems/actionpack-1.12.5/lib/action_view/base.rb:292:in `render_template'
c:/win32app/ruby/lib/ruby/gems/1.8/gems/actionpack-1.12.5/lib/action_view/base.rb:251:in `render_file'
c:/win32app/ruby/lib/ruby/gems/1.8/gems/actionpack-1.12.5/lib/action_view/base.rb:266:in `render'
c:/win32app/ruby/lib/ruby/gems/1.8/gems/actionpack-1.12.5/lib/action_view/partials.rb:59:in `render_partial'
c:/win32app/ruby/lib/ruby/gems/1.8/gems/actionpack-1.12.5/lib/action_controller/benchmarking.rb:29:in `benchmark'
c:/win32app/ruby/lib/ruby/1.8/benchmark.rb:293:in `measure'
…
…
...

In fact, the rails were complaining about the tabs before the embedded ruby commands

After changing the tabs to 2 spaces the application started… complaining about another characters.

I did some research and I was pointed to a ruby version. Before the reinstallation I was running 1.8.2. After the installation I took the latest one. Bad decision. It was caused by the ruby interpreter version.

So, if you face weird syntax errors with ruby 1.8.4 and you are sure about the syntax, try to change the version. I downgraded to 1.8.2 and it started to work again.

While doing some Ruby on Rails code refactoring, I realized I’d like to visualize the database schema of the application.

An easy possibility is to print out db/schema.rb. This seems a bit too linear and little visual to me. OK, I’d better keep thinking.

An enterprise approach would be to take a CASE machinery and feed it with the create DB scripts. That is solid and reliable, but a kind of overkill. What’s more, a Rails app with convention over configuration can do well without foreign keys and so it usually does. This means that relations between tables are mostly implied by column names rather than explicitly declared as DB constraints. Thus, hardly any CASE tool can derive such relations from a DB schema.

OK, let’s try to go the Rails way and put together existing ingredients, add a little cream and see what we can get:

• there are many UML modeling tools out there, although few of the free ones can reverse-engineer a DB schema
• there’s the SchemaDumper in ActiveRecord, which generates the schema.rb based on DB reflection
• there’s this standard XMI format for data interchange between modeling tools

OK, so let’s try and tweak the SchemaDumper to generate XMI, then import it into a UML modeling tool and get the schema diagram from there.

So, I morphed SchemaDumper into UmlDumper, which generates XMI in the widely adopted Rose flavor. E.g. a Typo blog database will result in schema.xml like this:

Then I import this XMI into a UML tool – StarUML in my case – to get the basic picture. Yellow boxes are DB tables, grey boxes are presumed “virtual” tables referenced by columns in other tables.

I play a little with the layout and incorporate the “virtual” tables into the hierarchy. For completeness, I also add a few model classes, which show up only in Rails code, not in the database (manual entries in blue color):

I export the diagram from the tool to print it, share it and enjoy.

Now, that’s it! The approach is quite universal (not yet true for UmlDumper), I can quickly reuse it in my further Rails projects, and so can you.

Note: for one-time use I created a rake task.

namespace :uml do
  desc "Generate an XMI db/schema.xml file describing the current DB as seen by AR. Produces XMI 1.1 for UML 1.3 Rose Extended, viewable e.g. by StarUML"
  task :schema => :environment do
    require 'lib/uml_dumper.rb'
    File.open("db/schema.xml", "w") do |file|
      ActiveRecord::UmlDumper.dump(ActiveRecord::Base.connection, file)
    end 
    puts "Done. Schema XMI created as db/schema.xml."
  end
end

For frequent use (fine-tuning the export), it is better to start the Rails console and call UmlDumper from there to avoid re-loading the whole Rails stack all over again.

There are several possibilities how to use ferret in RoR. This post will show the easy way – using the acts_as_ferret plugin.

To show the syntax and code, I will use the same data objects as in the Full text search in ruby on rails 2 – MySQL

Installation

Ferret installation is easy

gem install ferret

will do the job.

In addition, it is necessary to install the acts_as_ferret plugin.

script/plugin install svn://projects.jkraemer.net/acts_as_ferret/tags/stable/acts_as_ferret

Setup

The most simple setup is

class Article > ActiveRecord::Base
  acts_as_ferret	
end

This is enough to make the full text engine working. Now you can test it in the Rails console

Article.find_by_contents("sybase")

If you have a lot of data to be indexed, be patient with the first run. It is slow, because the index needs to be built.

The acts_as_ferret with no argument indexes automatically all fields of the Article, including arrays of child objects. This behaviour could be overwritten. You can narrow the field set

# Index only id and body, not title
acts_as_ferret :fields => [ 'id', 'body' ]

Or you can widen the field set.

acts_as_ferret :fields => [ 'id', 'body', 'title', 'long_article' ]
 
# Compute the article length
def long_article
  self.body.length > 40
end

Note 1: see usage of long_article in Query syntax below

Note 2: once you change the structure of the index, you need to rebuild it. The easiest way is to stop your application and delete the index/~environment~/~Indexed object~ folder. It will be created automatically with the next search request.

Query syntax

Since ferret is a port of the lucene engine, it uses the same query syntax. I will show only a few queries that you can use.

For details see Lucene documentation

  # Search for pages with "sybase" keyword
  Article.find_by_contents("sybase")
 
  # "sybase" and "replication" keywords
  Article.find_by_contents("sybase replication")
 
  # "sybase" or "replication"
  Article.find_by_contents("sybase OR replication")
 
  # short articles about sybase
  Article.find_by_contents("long_article:(false) *:sybase")
 
  # articles containing similar words like "increase"
  # will return e.g. increasing
  Article.find_by_contents("increase~")

Pagination

Ferret is fast, ferret is flexible, but… it is not an active record object, so you cannot use the pre-defined pagination. You have to implement it on your own. Here is how we did it in our project www.tamtami.com.

1. Create full text search function in the model

  def self.full_text_search(q, options = {})
    return nil if q.nil? or q==""
    default_options = {:limit => 10, :page => 1}
    options = default_options.merge options
    options[:offset] = options[:limit] * (options[:page].to_i-1)
    results_ids = []
 
    num = self.ferret_index.search_each("*:(#{q})", {:num_docs => options[:limit], :first_doc => options[:offset]}) { |doc, score|
      results_ids << self.ferret_index[doc]["id"]
    }
    results = Article.find(results_ids)
    return [num, results]
  end

or more elegant, as proposed by Jens Kraemer

  def self.full_text_search(q, options = {})
    return nil if q.nil? or q==""
    default_options = {:limit => 10, :page => 1}
    options = default_options.merge options
    options[:offset] = options[:limit] * (options.delete(:page).to_i-1)  
    results = Article.find_by_contents(q, options)
    return [results.total_hits, results]
  end

2. Create method that creates paginator in application.rb

  def pages_for(size, options = {})
    default_options = {:per_page => 10}
    options = default_options.merge options
    pages = Paginator.new self, size, options[:per_page], (params[:page]||1)
    pages
  end

3. Perform the search in the controller

  def search
    @query=params[:query]
    @total, @articles = Article.full_text_search(@query, :page => (params[:page]||1))	  
    @pages = pages_for(@total)
  end

4. Use it in the article view

...
   <%= pagination_links(@pages, :params => {:query=>@query}) %>
...

Final word

The ferret fulltext engine is fast, flexible, but needs more programming than MySQL full text index.

So, we have a public web application accessible 24×7. Even if we do not process financial transactions or control airport traffic, this poses a kind of stress upon us. To relieve it, functional testing is our true companion.

By functional testing I mean simulating an end user’s traversal through the application. This means that, contrary to unit tests, we are testing what the user sees, not what the developer has written. The tests are thus less sensitive to software design changes, internal bug fixes, and even URL refactoring.¹

As a tool, we have chosen Watir, an open-source library for controlling a web browser. It is light-weight and it does what it says on the box: “drives the … browser the same way people do. It clicks links, fills in forms, presses buttons… also checks results, such as whether expected text appears on the page.”

Technically, Watir is a library in Ruby² for controlling Microsoft Internet Explorer on Windows (with support for Linux and Firefox planned in the next major release). The tested web application may be anything (ASP, JSP, PHP, etc.) producing browsable web pages.

Writing tests in Watir is fun in itself. Firstly, you can see the page in the browser and watch the elements being manipulated by an invisible user – buttons, links etc. get highlighted when the test suite accesses them. Secondly, you can even use Ruby console to write the tests interactively so that you command the browser like a starship.

def fBU1_log_in( username = "Adam", password = "Adam" )
    assert(@ie.pageContainsText("Login"))
    @ie.text_field(:id, "user_username").set(username)
    @ie.text_field(:id, "user_password").set(password)
# Clear the security warnings for HTTPS
    startClearers 
    @ie.button(:value, "Log in").click
# Show English version for test even if the user has 
# another language preference
    if not @ie.pageContainsText("Welcome ") 
        @ie.image(:alt, "English").click 
    end
    assert(@ie.pageContainsText("Welcome "+username))
end

Still, as we soon found out, there needs some home work to be done in order to achieve representative results of the tests:

1. define test scenarios
2. prepare test data

Test scenarios are needed to cover the whole functionality landscape, and to agree with the developers, what the application should do. In theory, this should completely be done at the beginning of the application lifecycle, but in an agile approach (as we had it with Recykl.com), you just define the main use cases at the beginning and fine-tune the details based on end user/customer feedback. Anyway, having the scenarios automated in tests is a good validation of the functional design.

Test data for the functional tests are indispensable for the predictability of the test results. In our case, another important requirement was isolation of the data for the individual tests, as we decided to use just one suite of test data for the whole suite of functional tests. For faster test execution the data is loaded once, not at the beginning of each test in the suite (which is a common Ruby test fixtures practice). To achieve isolation, we successively run the tests under different users, so the risk of interfering is reduced to a minimum.

Now, after some time of using Watir for the testing, there are some lessons learned:

1. the most painful and unpredictable errors result from timing, e.g. in asynchronous AJAX calls – sometimes it’s difficult even for a human to tell if we are still waiting for something to happen or that’s all
2. use some kind of automated reporting of results (unfortunately Watir currently has a very limited support here)
3. tests are not carved in stone – you will have to update them as your functional design evolves (and it does in a successful application) – so write them flexibly. For example, do not rely on details like exact wording of a link, but instead use a regular expression matching just the main keyword, e.g. /[Ss]ubmit/
4. even though it takes long to run the tests, try to run them after each major changeset – you’ll catch bugs and also easier keep up with the intentional changes

Having that in mind, doing functional testing with Watir has significantly increased our confidence in the functioning of our web application and saved us some hot moments in its (sometimes stormy) agile lifecycle.

Note 1: Although this looks like THE testing, other kinds of testing still have their important role: unit testing to make sure the application pieces are runnable after developers’ changes, performance testing to see the behavior of the application under heavier load than just one user, security testing and reviews to identify the possible weak spots exposed to the wild internet.

Note 2: Both Watir and Ruby are open-source products. Ruby is an object-oriented scripting language somewhere between Perl and Smalltalk. It is has garbage collection like C# or Java, but installs from a 5MB Linux RPM or 27MB Windows installer and starts up in an instant. Watir is an 800kB little gem ready to start after download.