Monitoring number of SQL queries in MySQL

I have created a new application in Grails and surprisingly, in production environment it was slower than on my laptop. Production administrator found that my application generate too many SQL queries. The delay was caused by network latency. So, I had to find a way how to monitor number of SQL queries on MySQL server.

For monitoring of the current session, there is a handy command:

SHOW STATUS WHERE variable_name='Com_select'

For monitoring of all the sessions, there is a switch:

SHOW global STATUS WHERE variable_name='Com_select'

All variables could be listed by

SHOW STATUS

From Mysql to Oracle: Grails application migration

Today I have finished migration of our Grails prototype. Originally we did develop it for MySQL, but the final system have to work with Oracle. This post summarize troubles and differences I was facing.

User creation

Since I am not Oracle administrator, it took me some time to put together script that creates user and appropriate rights for the Grails application. Use the script for development only.

CREATE user grails IDENTIFIED BY grassword;
GRANT connect  TO habilion;
GRANT CREATE TABLE TO habilion;
GRANT CREATE sequence TO habilion;
GRANT unlimited tablespace TO habilion;

Identifier length limitation

I usually do not use identifiers that are 30 characters long. But, there is a trick.

Imagine two classes: Author and Content. Author having multiple Contents.

Now, imagine there is several types of Content – Book, Newspaper, ElectronicNewspaper, ScientificArticle…
So, we can model the situation using following classes:

Class Content{
  static belongsTo = [ contentAuthor:Author ]
}
 
Class Book extends Content {}
Class Newspaper extends Content {}
Class ElectronicNewspaper extends Content {}
Class ScientificArticle extends Content {}
 
Class Author{
  static hasMany = [ contents: Content ]
}

And now comes the trick. The ‘content’ table in the database contains long columns:

content_author_id
newspaper_content_author_id
electronic_newspaper_content_author_id
book_content_authot_id
scientific_article_author_id

As you can see, it is easy to hit the 30 characters limit. If you look deeper, you will see the solution. The column names are composed of class name of the Content descendants and variable name pointing to the master entity.
So, I renamed the contentAuthor to ca, created getter and setter for contentAuthor and checked all “new Book…” snippets. God bless MVC!

Column size

The second major issue was a String column. I had a string column of 10000 characters.

Class Content {
  String abstract
  static constraints = { abstract(maxSize:10000) }
}

It was translated to Long data type. Unfortunately there is a bug in Oracle JDBC driver that causes nasty stack traces when processing Long data type. There is a workaround on the Internet, but it is working only for some versions of Oracle. Moreover, there is just one Long column allowed per table. So, changed the size to 4000 characters and the column was created as VARCHAR2.

And thats it!

Update: Empty strings

One thing to mention is a fact that Oracle does not distinguish between null and an empty string. So, if you try to store an empty string to Oracle DB, it actually stores it as a null value.

The problem arises, if you have not null column and you try to store empty string ‘’. For Grails it is a not null value, so it does not apply defaults. For Oracle it is null and thus constraint violation…

Improve performance of MySQL driver for RoR

Last week I was working on the performance tuning of a rails application. I ran a profiler and found something very interesting.

I found that there is a procedure that is called very often and takes a lot of time.

The procedure was Mysql#get_length:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# File src/rails-1.2.3/activerecord/lib/active_record/vendor/mysql.rb
  def get_length(data, longlong=nil)
    return if data.length == 0
    c = data.slice!(0)
    case c
    when 251
      return nil
    when 252
      a = data.slice!(0,2)
      return a[0]+a[1]*256
    when 253
      a = data.slice!(0,3)
      return a[0]+a[1]*256+a[2]*256**2
    when 254
      a = data.slice!(0,8)
      if longlong then
        return a[0]+a[1]*256+a[2]*256**2+a[3]*256**3+
          a[4]*256**4+a[5]*256**5+a[6]*256**6+a[7]*256**7
      else
        return a[0]+a[1]*256+a[2]*256**2+a[3]*256**3
      end
    else
      c
    end
  end

There is obviously space for improvement! Replace the multiplications by shifts

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
class Mysql
  def get_length(data, longlong=nil)
    return if data.length == 0
    c = data.slice!(0)
 
    case c
    when 251
      return nil
    when 252
      a = data.slice!(0,2)
      return a[0]+(a[1]<<8)
    when 253
      a = data.slice!(0,3)
      return a[0]+(a[1]<<8)+(a[2]<<16)
    when 254
      a = data.slice!(0,8)
      if longlong then
        return a[0]+(a[1]<<8)+(a[2]<<16) +(a[3]<<24)+(a[4]<<32)+(a[5]<<40)+(a[6]<<48)+(a[7]<<56)
      else
        return a[0]+(a[1]<<8)+(a[2]<<16)+(a[3]<<24)
      end
    else
      c
    end
  end
end

I ran the profiler again and, well… a wisdom of my university times popped on my mind: “There is an elegant, simple, nice and obvious solution for each problem. Unfortunately, it is wrong!”

Performance remained the same. So, deeper investigation is needed! It was not difficult to find out that most of the times the “else” branch is executed. I tried something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
  def get_length(data, longlong=nil)
    return if data.length == 0
    c = data.slice!(0)
 
    return c if c<251
 
    case c
    when 251
      return nil
    when 252
      a = data.slice!(0,2)
      return a[0]+(a[1]<<8)
    when 253
      a = data.slice!(0,3)
      return a[0]+(a[1]<<8)+(a[2]<<16)
    when 254
      a = data.slice!(0,8)
      if longlong then
        return a[0]+(a[1]<<8)+(a[2]<<16) +(a[3]<<24)+(a[4]<<32)+(a[5]<<40)+(a[6]<<48)+(a[7]<<56)
      else
        return a[0]+(a[1]<<8)+(a[2]<<16)+(a[3]<<24)
      end
    else
      c
    end
  end

And the performance?

Improved! The toplevel cumulative time is down by over 20 seconds.
Now, here’s how you can embed this hack into your application:

  • Create a file called e.g. mysql_fix.rb
  • Add there
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
require 'active_record/vendor/mysql'
 
class Mysql
  def get_length(data, longlong=nil)
    return if data.length == 0
    c = data.slice!(0)
 
    return c if c < 251
 
    case c
    when 251
      return nil
    when 252
      a = data.slice!(0,2)
      return a[0]+(a[1]<<8)
    when 253
      a = data.slice!(0,3)
      return a[0]+(a[1]<<8)+(a[2]<<16)
    when 254
      a = data.slice!(0,8)
      if longlong then
        return a[0]+(a[1]<<8)+(a[2]<<16) +(a[3]<<24)+(a[4]<<32)+(a[5]<<40)+(a[6]<<48)+(a[7]<<56)
      else
        return a[0]+(a[1]<<8)+(a[2]<<16)+(a[3]<<24)
      end
    else
      c
    end
  end
end
  • Put it into e.g. lib/zmok directory
  • Add into your environment.rb following line
1
require File.join(File.dirname(__FILE__), '../lib/zmok/mysql_fix')

It is funny how my performance tuning session ended. Instead of changing my Rails application, I ended up improving the MySQL driver.

Full text search in Ruby on Rails 2 - MySQL

My previous post compared MySQL and ferret full text search
engines. For our project, the ferret was the winner. Nevertheless, I
will try to show the beauty and simplicity of using MySQL indexes.


Create table and indices


First of all it is necessary to create table and the corresponding
index.

CREATE TABLE articles(
  id integer NOT NULL PRIMARY KEY AUTO_INCREMENT,
  title varchar(20),
  body varchar(100),
  fulltext(title, body)
) engine = MyISAM;<BR>

or create index after the table exists.

CREATE fulltext INDEX x_f_articles_body ON articles(body);

Please note the MyISAM engine. You cannot create full text index
on InnoDB tables.


And now, let’s insert some data

INSERT INTO articles(title, body)
  SELECT "Databases and IT", "Todays world ... database... MySQL, Sybase";
 
INSERT INTO articles(title, body) 
  SELECT "Sybase RS manual", "Sybase Replication server is a ...";
 
INSERT INTO articles(title, body) 
  SELECT "Sybase technology", "ASE, RS, IQ, PowerBuilder, all of them are...";
 
INSERT INTO articles(title, body) 
  SELECT "Databases and people", "People are using databases without knowing it...";
 
INSERT INTO articles(title, body) 
  SELECT "People everywhere", "Human population is increasing...";

Query syntax


Querying is simple. It is part of the MySQL dialect. Simple Boolean query searches for articles with “Databases”
keyword looks similar to this:

SELECT * 
FROM articles 
WHERE match(title,body) against ("Databases");






















id

title

body

4

Databases and people

People are using databases without knowing it…

1

Databases and IT

Today’s world … database… MySQL, Sybase

or you can create query looking for all database related articles

SELECT * 
FROM articles 
WHERE match(title,body) against ("Databases" WITH query expansion);



























id

title

body

1

Databases and IT

Todays world … database… MySQL, Sybase

4

Databases and people

People are using databases without knowing it…

5

People everywhere

Human population is increasing…

Note, that the “match” columns must be the same as they were in the
create index statement.


See also the record ID=5. There is nothing about databases, but it is
in the result set anyway. The query expansion means, that the MySQL
engine goes through the index twice. In first run it finds all
records with the searched keyword and builds a set of keywords that
appear together with the search string. In the second run it
searches for the expanded set of keywords. Since the “Databases”
are together with “People” in record ID=4, it was
returned as relevant.

This is useful. Unfortunately the result set is often too big.


Also note, that there are not records 2 and 3 containing the Sybase keyword (remember, record 1 contains both Databases and Sybase keywords). The reason is
simple. MySQL weights the words according to their frequency. If the
word is too often in the articles, the weight becomes 0 and thus not
relevant. You have always to remember it, because it’s a feature, not a
bug!


Search


So, let’s see it in action. The fulltext search in rails is as simple as
any other search.

Articles.find(:all, :conditions => ["match(title,body) against  (?)", "Databases"] )

naturally it is possible to combine it with other expressions:

Articles.find(:all, :conditions => ["match(title,body) against  (?) and id > ?",
"Databases", 2] )

Pagination


Pagination is as simple as it could be. You can use the same
pagination methods as for any other ActiveRecord query.

def list
  @article_pages, @articles = paginate :articles,
    :per_page => 10, 
    :conditions => ["match(title,body) against  (?)", "Databases"]
end

Scoping


Scoping is a feature you cannot do with ferret. How does it work?


First of all, imagine that you have a really “complex”
function looking for new articles (ID > 2).

def new_articles
  Articles.find( :all, :conditions => [ "id > ?", 2 ] )
end

Than your boss come and say… OK, but I would like to have
there a possibility to display only new articles about… for example
“Databases”.

Well, you can change the function, or you can scope it. Since you cannot learn now anything new by rewriting functions, we will try to
scope it.

def bosses_DB_articles
  Articles.with_scope(:find => {:conditions => ["match(title,body) against  (?)", "Databases"]} ) do
    new_articles
  end
end

And that’s the whole trick!


Final word


As I said. Searching with MySQL is really easy, but sometimes
it gives unexpected results.