Live fulltext search in Ruby on Rails

Some time ago I promised to create a small tutorial about live fulltext search. A fulltext search, that gives you results as you type.

Ingredients:

  • Ruby on rails
  • ferret gem (gem install ferret)
  • acts_as_ferret gem (gem install acts_as_ferret)
  • auto_complete plugin (from the application root: ruby script/plugin install auto_complete)

What we will do

  1. Create an empty application – simple book database
  2. Add fulltext search capabilities
  3. Create the live search
    1. Create search pane partial – the one that will display the search box
    2. Create the search results partial – that will render the hints (search results)
    3. Modify controller to respond to the search pane

Create a book database application

We will create a small application for book management. It will store, list, update books and it will also provide the live search.
So, lets create the skeleton of the aplication:

# Create the rails application
rails books
# create database books
echo "create database books"  | mysql -u root -p
cd books

Configure database login and password in app/config/database.yml.

development:
  adapter: mysql
  database: books
  username: root
  password: password
  host: localhost
  port: 3306

Create skeleton of the application. From root of the application run:

ruby script/generate scaffold Book title:string abstract:text

Create the books table

rake db:migrate

Start up the development server

ruby script/server

Now, browse to http://127.0.0.1:3000/books and type in some data.

Add fulltext search capabilities

Change the app/models/book.rb to support fulltext search

require "acts_as_ferret"
 
class Book < ActiveRecord::Base
    acts_as_ferret
end

You can check in the console, that the fulltext is enabled. Just start the console via
ruby script/console and put there

Book.find_by_contents("book").

It should return a result set, similar to this:

=> #<ActsAsFerret::SearchResults:0x2540f54 @results=[#<Book id: 2, title: "Book secondo", abstract: "Book about book", created_at: "2008-07-07 23:16:38", updated_at: "2008-07-07 23:16:38">, #<Book id: 1, title: "First book", abstract: "This is a first book", created_at: "2008-07-07 23:16:23", updated_at: "2008-07-07 23:16:23">], @total_hits=2>

Create the live search

Finally, create the live search.

Create search pane partial

The search_pane will be used to display search box.

Create a partial _search_pane.html.erb in app/views/books and put there simple tag. The tag create Ajax Autocompleter that calls auto_complete_for_search_query method of the default controller (in our case it will be books)

<%= text_field_with_auto_complete :search, :query %>

Add javascript include and partial rendering to the books template app/views/layouts/books.html.erb.

Do not forget! The javascript include must be in the head of the template.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
       "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
  <meta http-equiv="content-type" content="text/html;charset=UTF-8" />
  <title>Books: <%= controller.action_name %></title>
  <!-- HERE -->
  <%= stylesheet_link_tag 'scaffold' %>
	<%= javascript_include_tag :defaults %> 
</head>
<body>
 
<!-- AND HERE -->
<%= render :partial=>"books/search_pane" %>
 
<p style="color: green"><%= flash[:notice] %></p>
 
<%= yield  %>
 
</body>
</html>

Create the search results partial

The search_results will format the results of the full text search and will “offer” the resulting records. Create a partial app/views/books/_search_results.html.erb and add there the formatting code:

<ul>
	<% for book in @books %>
	<li><%= link_to h(book.title), :controller=>"books", :action=>"show", :id=>book %></li>
  	<% end %>
</ul>

Modify controller

Add the following line at the beginning of the books_controller.

protect_from_forgery :only => [:create, :update, :destroy]

Create a method in books_controller that will search for the books

 def auto_complete_for_search_query
   @books = Book.find_by_contents(params["search"]["query"]+"*", {:limit => 5})
   render :partial => "search_results"
 end

We do not want to generate the whole page layout, so it is necessary to specify it in the books controller:

layout 'books', :except => [:auto_complete_for_search_query]

And now, navigate to http://127.0.0.1:3000/books and start searching. As soon as you start typing into the search box, it shows results. Click on one of the proposed links to see what happens. Source code is here.

Ad-hoc fulltext search in RoR ActiveRecord

I came to a situation where I needed to search my Active record, but I did not know which field contains the information. The solution with Ferret was just three steps away…

Let’s say, you want to search Stories for ‘Giant’ keyword. You have to create a Ferret index in memory (ferret gem needs to be installed), index all active records and gather all IDs matching the keyword.

1
2
3
4
5
6
7
index=Ferret::I.new
 
Story.find(:all).each { |s| index << {:id=>s.id, :content=>s.inspect} }
 
index.search_each('Giant', :limit=>100) do |id, score| 
  puts "Active record ID: #{index[id][:id]} with score #{score}"
end

… now you have the full power of the Ferret engine in your hands.

RoR in enterprise – lessons learned

After a while my first enterprise prototype is finished and I have to summarize what was right and what was wrong during the period of prototyping.

Really nice surprise for me was the way of communication. The requirements were formulated more precisely then any requirement before, but not from the beginning. When I did start, the requirements were very vague, but after first screens and first few features the communication was excellent and clear. This was the phase of fundamental principles and relations creation.

After some time, the flow of requirements was stronger and stronger. It was necessary to start requirements management – yes, it is true. You cannot get rid of it.

In the middle of development users started to use it and a new set of “handy little” features was requested.

Later the system became very important and the users started to solve real production issues using the system.

Now, the system is almost finished. I mean – the necessary features are there, but sometimes it is not consistent. Especially the stuff that is used rarely. The enterprise is pushing me to pass it into production regime.
And what are my lessons learned?

  • The start will be painful. Be prepared to completely redesign the code and model.
  • Scaffolding is for some time more than enough.
  • It is not necessary to focus on good graphical desing from the beginning. Focusing on features is more important. In my case I did implement a nice design after 80-90% of features was implemented. And in fact, I did it because the environment was really ugly, not because customer did ask me to do it.
  • Make sure you find the right group of people to prototype with. It is impossible to create a prototype without business experts.
  • It is not important how much you boost your performance using good tools. You will always have more requirements than could be implemented. Requirements management is a must.
  • Communicate clearly that you are working on a prototype. Otherwise you will be forced to make it a real application. Here I started to think about using grails for prototyping. Nevertheless, I have no experience to decide if it was a good idea or not.
  • Be prepared for success! The application that will be created is in line with the requirements and heals the biggest pains of the business users. It is highly probable that the users will love it.
  • Do not stick with one technology. Anything that makes the process faster is valuable. Eg. I did use Sybase PowerDesigner to generate “history keeping” triggers automatically. Adding a new table to my model was just few clicks and assigning the right trigger template.
  • And last, but not least. Listen! Listen more! And make sure you understand that you are not the one who knows the business. You came there to help them to communicate their needs, not to show them how to do their business.

Now I must say, that Ruby on Rails is a great tool for project communication. It is able to communicate user requirements very efficiently and precisely.

RSpec for Ruby on Rails

Behaviour driven development is currently in. The best ay how to get in touch with it is to setup your own environment and make few examples.Nevertheless, if you prefer to start with a bit of theory, go to http://behaviour-driven.org/.This article describes simple procedure to setup Rspec to work together with ruby on rails.

Installation of RSpec

The installation procedure described in documentation needed small improvement:

First of all, install the rspec gem

gem install -r rspec #mac users must sudo

Then install following gems:

  • rake # Runs the build script
  • rcov # Verifies that the code is 100% covered by specs
  • webgen # Generates the static HTML website
  • RedCloth # Required by webgen
  • syntax # Required by our own custom webgen extension to highlight ruby code
  • diff-lcs # Required if you use the -diff switch
  • win32console # Required by the -colour switch if you‘re on Windows
  • meta_project # Required in order to make releases at RubyForge
  • heckle # Required if you use the -heckle switch
  • hpricot # Used for parsing HTML from the HTML output formatter in RSpec’s own specs

Then continue with these steps:

svn co svn://rubyforge.org/var/svn/rspec/trunk rspec
cd rspec
rake install_dependencies
cd example_rails_app
export RSPEC_RAILS_VERSION=1.2.3
rake rspec:generate_mysql_config
mysql -u root -p &lt; db/mysql_setup.sql
cd ..change example_rails_app/config/database.yml to correspond to your configurationrake pre_commit

and…Make the first test.

Create new folder in your project called spec. Create a file named e.g. basic_test.rb and fill it with

describe "Sum computation" do  
  it "should return 2" do    
    (1+1).should == 2  
  end
end

run it with

spec spec/basic_test.rb

Your test should finish sucessfuly:

.Finished in 0.006001 seconds1 example, 0 failures

Working with rails

That’s great! Now, let’s make it running with your rails objects. Go to the root of your application and install rspec plugins:

ruby script/plugin install svn://rubyforge.org/var/svn/rspec/tags/CURRENT/rspec
ruby script/plugin install svn://rubyforge.org/var/svn/rspec/tags/CURRENT/rspec_on_rails

bootstrap your application with

ruby script/generate rspec

and start testing with the rails objects. Use rspec generator to create the first test:

ruby script/generate rspec_model user

It generates file spec/models/user_spec.rb file

require File.dirname(__FILE__) + '/../spec_helper'
describe User do  
  before(:each) do    
    @user = User.new  
  end  
 
  it "should be valid" do    
    @user.should be_valid  
  end
end

And now you can just extend the pre-generated file and enjoy it.

Rails upgrade from 1.1.6 to 1.2.3

I have just finished rails and gems upgrade and it was surprisingly smooth. There were just one minor issue.

The server was not able to start. It was teling me something like:

The solution was trivial. We are using an object Recconfig to store configuration of the web site. We do load and initialize the object in the environment.rb. In the 1.1.6 it was enough to write


in the new version it was necessary to require it.