Archive for the ‘Open source’ Category

Nested set plugin for Grails

We are happy to announce that our NestedSet plugin for the Grails framework has been released. The plugin creates a hierarchical structure on top of plain database records. A description of the “nested set” data structure can be found in Wikipedia.

The NestedSet Grails plugin is

  • non-intrusive – it does not change your data model, just adds one table
  • easy to use – just add “static nestedSet = true” to the domain class
  • versatile – it supports multiple hierarchies

Installation instructions and usage details can be found at the NestedSet plugin homepage.

If you find it useful, tell your friends. If there is a problem, please register at grails.cz and create a new issue.

Utilizing Caches.rb with Ferret

We needed to cache a Ruby class method calling the Ferret indexing engine.
Yurii Rashkovskii developed a great library called Caches.rb.

When I googled it out, it seemed very simple to use and promised to do EXACTLY what I need (even the default timeout was JUST IT). I especially liked the very Rails-like tutorial Don’t tell, show me!. However, it required quite some effort to make it work, mainly because of the rather sparse documentation. Still, in the end the usage is very elegant, the solution is simple and it does what it promises. Thank you, Yurii!

To help our esteemed readers get faster over that less agreeable middle phase, here are a few tips:

  • Downloading it: I tried gems but the gem list server seemed to be overloaded, and when it worked at last, I just got an older version (0.2.0). When checking out (or exporting) version 0.4.0 from SVN trunk directly, the trick was in finding out the latest working SVN URL:
    ruby script/plugin install http://svn.verbdev.com/rb/caches.rb/trunk

  • With all the typical Rails mixin stuff petrified in my mind, it took me a while to notice that caching should be declared AFTER the definition of the method to be cached, and not at the beginning of the class definition.
  • The example in the documentation shows it, but it’s easily overseen.

  • We do use Rails, so I included class_cache_storage Caches::Storage::Global as suggested here
  • For some reason (I suspect my shallow knowledge of Ruby ;-), I did not manage to successfully require ‘caches.rb’ from the plugin installation dir _#{RAILS_ROOT}/vendor/plugins/caches.rb/lib_, so I copied it to _#{RAILS_ROOT}/lib_, which helped.
  • For similar reasons, I had to use a workaround to extend the class definition with caching, instead of the recommended way of extending conf/environment.rb by ActiveRecord::Base.extend Caches::ClassMethods

So our class looks as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
require 'ferret'
include Ferret
require 'caches.rb'
 
class FIndex
  extend Caches
  ...
  def self.search(user=nil, results_per_page=10)
    ...
  end
  class_cache_storage Caches::Storage::Global
  class_caches :search
end

N.B.: The class to be cached works with Ferret and not the DB, so it did not inherit from ActiveRecord.

  • For a pure RoR developer, the terminology may be a little confusing (which indicates that Yurii can deal with more programming languages than just Ruby):
    • class methods mentioned in the library description are obviously just a shorthand for “the methods of a class”
  • in Yurii’s documentation, Ruby class methods are called static methods (as known in C/C++ or Java), and their caching is supported by class_caches (feature not available in caches.rb 0.2.0 from gems or RubyForge, but included in the more recent versions from SVN, like 0.4.0 we are using)

Visualize your Rails schema

While doing some Ruby on Rails code refactoring, I realized I’d like to visualize the database schema of the application.

An easy possibility is to print out db/schema.rb. This seems a bit too linear and little visual to me. OK, I’d better keep thinking.

An enterprise approach would be to take a CASE machinery and feed it with the create DB scripts. That is solid and reliable, but a kind of overkill. What’s more, a Rails app with convention over configuration can do well without foreign keys and so it usually does. This means that relations between tables are mostly implied by column names rather than explicitly declared as DB constraints. Thus, hardly any CASE tool can derive such relations from a DB schema.

OK, let’s try to go the Rails way and put together existing ingredients, add a little cream and see what we can get:

  • there are many UML modeling tools out there, although few of the free ones can reverse-engineer a DB schema
  • there’s the SchemaDumper in ActiveRecord, which generates the schema.rb based on DB reflection
  • there’s this standard XMI format for data interchange between modeling tools

OK, so let’s try and tweak the SchemaDumper to generate XMI, then import it into a UML modeling tool and get the schema diagram from there.

So, I morphed SchemaDumper into UmlDumper, which generates XMI in the widely adopted Rose flavor. E.g. a Typo blog database will result in schema.xml like this:

xmi.gif

Then I import this XMI into a UML tool – StarUML in my case – to get the basic picture. Yellow boxes are DB tables, grey boxes are presumed “virtual” tables referenced by columns in other tables.

raw.gif

I play a little with the layout and incorporate the “virtual” tables into the hierarchy. For completeness, I also add a few model classes, which show up only in Rails code, not in the database (manual entries in blue color):

main.gif

I export the diagram from the tool to print it, share it and enjoy.

Now, that’s it! The approach is quite universal (not yet true for UmlDumper), I can quickly reuse it in my further Rails projects, and so can you.

Note: for one-time use I created a rake task.

namespace :uml do
  desc "Generate an XMI db/schema.xml file describing the current DB as seen by AR. Produces XMI 1.1 for UML 1.3 Rose Extended, viewable e.g. by StarUML"
  task :schema => :environment do
    require 'lib/uml_dumper.rb'
    File.open("db/schema.xml", "w") do |file|
      ActiveRecord::UmlDumper.dump(ActiveRecord::Base.connection, file)
    end 
    puts "Done. Schema XMI created as db/schema.xml."
  end
end

For frequent use (fine-tuning the export), it is better to start the Rails console and call UmlDumper from there to avoid re-loading the whole Rails stack all over again.

Functional testing with Watir

So, we have a public web application accessible 24×7. Even if we do not process financial transactions or control airport traffic, this poses a kind of stress upon us. To relieve it, functional testing is our true companion.

By functional testing I mean simulating an end user’s traversal through the application. This means that, contrary to unit tests, we are testing what the user sees, not what the developer has written. The tests are thus less sensitive to software design changes, internal bug fixes, and even URL refactoring.1

As a tool, we have chosen Watir, an open-source library for controlling a web browser. It is light-weight and it does what it says on the box: “drives the … browser the same way people do. It clicks links, fills in forms, presses buttons… also checks results, such as whether expected text appears on the page.”

Technically, Watir is a library in Ruby2 for controlling Microsoft Internet Explorer on Windows (with support for Linux and Firefox planned in the next major release). The tested web application may be anything (ASP, JSP, PHP, etc.) producing browsable web pages.

Writing tests in Watir is fun in itself. Firstly, you can see the page in the browser and watch the elements being manipulated by an invisible user – buttons, links etc. get highlighted when the test suite accesses them. Secondly, you can even use Ruby console to write the tests interactively so that you command the browser like a starship.

def fBU1_log_in( username = "Adam", password = "Adam" )
    assert(@ie.pageContainsText("Login"))
    @ie.text_field(:id, "user_username").set(username)
    @ie.text_field(:id, "user_password").set(password)
# Clear the security warnings for HTTPS
    startClearers 
    @ie.button(:value, "Log in").click
# Show English version for test even if the user has 
# another language preference
    if not @ie.pageContainsText("Welcome ") 
        @ie.image(:alt, "English").click 
    end
    assert(@ie.pageContainsText("Welcome "+username))
end

Still, as we soon found out, there needs some home work to be done in order to achieve representative results of the tests:

  1. define test scenarios
  2. prepare test data

Test scenarios are needed to cover the whole functionality landscape, and to agree with the developers, what the application should do. In theory, this should completely be done at the beginning of the application lifecycle, but in an agile approach (as we had it with Recykl.com), you just define the main use cases at the beginning and fine-tune the details based on end user/customer feedback. Anyway, having the scenarios automated in tests is a good validation of the functional design.

Test data for the functional tests are indispensable for the predictability of the test results. In our case, another important requirement was isolation of the data for the individual tests, as we decided to use just one suite of test data for the whole suite of functional tests. For faster test execution the data is loaded once, not at the beginning of each test in the suite (which is a common Ruby test fixtures practice). To achieve isolation, we successively run the tests under different users, so the risk of interfering is reduced to a minimum.

Now, after some time of using Watir for the testing, there are some lessons learned:

  1. the most painful and unpredictable errors result from timing, e.g. in asynchronous AJAX calls – sometimes it’s difficult even for a human to tell if we are still waiting for something to happen or that’s all
  2. use some kind of automated reporting of results (unfortunately Watir currently has a very limited support here)
  3. tests are not carved in stone – you will have to update them as your functional design evolves (and it does in a successful application) – so write them flexibly. For example, do not rely on details like exact wording of a link, but instead use a regular expression matching just the main keyword, e.g. /[Ss]ubmit/
  4. even though it takes long to run the tests, try to run them after each major changeset – you’ll catch bugs and also easier keep up with the intentional changes

Having that in mind, doing functional testing with Watir has significantly increased our confidence in the functioning of our web application and saved us some hot moments in its (sometimes stormy) agile lifecycle.


Note 1: Although this looks like THE testing, other kinds of testing still have their important role: unit testing to make sure the application pieces are runnable after developers’ changes, performance testing to see the behavior of the application under heavier load than just one user, security testing and reviews to identify the possible weak spots exposed to the wild internet.

Note 2: Both Watir and Ruby are open-source products. Ruby is an object-oriented scripting language somewhere between Perl and Smalltalk. It is has garbage collection like C# or Java, but installs from a 5MB Linux RPM or 27MB Windows installer and starts up in an instant. Watir is an 800kB little gem ready to start after download.

Freedom and open source is not for all

I have been living under totalitarianism for 15 years. And I must say, it was a great time. It was funny and joyful, because it was my childhood. I never wanted to live in a free country. In fact, I thought, I am living in a free country. I just did not know the taste of freedom.

But there were people that knew what is the taste of freedom and that the totalitarianism is a stinky place to live. Fortunately, there was the Velvet revolution in 1989 and the freedom came to my country.

Some time ago I found the free software and open source movement and frankly speaking, the more I know it, the more it reminds me the difference between totalitarianism and freedom.

The open source means for me

  • possibility to choose the best option for me. No restrictions given by “dictator”. But it also means responsibility for the choice that was taken.
  • possibility to go my way, not to follow the main stream. Possibility to select any minority software and be different and to take advantage of being different.
  • responsibility – one must be responsible for himself… and believe me, this is one of the hardest things in changing totality for freedom; changing proprietary software for free software.
  • annoyingly lot of choices. I know, it is an advantage, but this is one of the reasons why changing proprietary software for open source is so painful.
  • evolution – systems are built on other systems, different teams working on the same code, the best libraries survives. Seeing difference between Windows and Linux shows how strong is this principle.

On the other hand, closed source represents

  • simple life. Any feature either exists or does not exist. I do not have any impact on it. I always can say that the XY company made wrong decision.
  • hard life, once I really need something that was not “planned” by the proprietary software designer. Usually a workaround is needed, but it often made the resulting system more complex… and try to change any part of the proprietary code! You will have neither code of your system, nor support of the system.
  • no choice. Once I live in proprietary software world, I do not have much choices. The companies creates lock-in strategies and it is painful to change the vendor.
  • I do not have any impact on the functionality of the closed software. OK, one can write a mail to the company asking for a feature, but the result is the same like changing totalitarianism via “elections”.
  • massive marketing – the same is essential for any totalitarianism.

The most important thing I have learned about totalitarianism and freedom is simple: “People that were living the whole life under totalitarianism, does not understand the free people. People living the whole life in freedom does not understand the totalitarianism.”

And last, but not least: “For some people, living in totalitarianism is more simple and comfortable than living in a free country.”

So please, be patient and let the proprietary software users live in their world. Give them chance to free themselves and prepare them simple “escape tunnel”. And do not be angry if they stay with closed source software, because freedom and open source is not for all.