Boolean Searches with Sphinx

Well, we now have a nice little search system for our entity data. But what if the user want to search for "a" OR "b". What we have so far doesn't handle that. So let's make the modifications we need to make that work.

All I have to say is it's a good thing that the RailsCast on Sphinx addressed this, because I couldn't find this in the thinking_sphinx online docs anywhere. Maybe I just missed it, but I've looked 20 times and just don't see it. (Don't think I'm knocking thinking_sphinx in any way. It's still awesome!)

Anyway, what we'll need to do is modify our search call in our home_controller.rb file. The magic word is :match_mode. So here's the new search line in the controller:

@entities = Entity.search params[:search],
   :page => params[:page],
   :per_page => 10,
   :order => "lastname ASC, firstname ASC",
   :match_mode => :boolean

Notice I've set the match mode to :boolean. This will allow us to do boolean searches. So what is a boolean search? Well, for one thing, it allows us to search for "a" OR "b". However, thinking_sphinx doesn't use the word OR, it uses a pipe (|) symbol. Hmm...I don't think my users will want to do this. I know I'm used to putting the word OR in most of the searches I do. So we'll have to do something about that later. Boolean searches also allow us to use AND, NOT and grouping operators. Those would be &, ! or -, and (). Again, not something most of my users will be able to/like to deal with. (Note that there are other match modes, but I'm not going to be using those. For further information on those, look here.)

So what do we do to allow our end user to use boolean syntax they are familiar with (assuming they're not programmers)? We have to do some string replacement in our code. For simplicity's sake right now, I'm just going to put this in my controller. I'll probably refactor it out later because I don't think that's really the appropriate place for it. Course, I'm a n00b, so what do I know?

Here's what I did. There may be better ways, and maybe I'll stumble on to those later, but for now, this should give us something to work with.

class HomeController < ApplicationController
  def index
    params[:search].gsub!(/ or /i," | ")
    params[:search].gsub!(/ and /i," & ")
    params[:search].gsub!(/\"(.*)\"/,'(\1)')
    params[:search].gsub!(/\'(.*)\'/,'(\1)')
    @entities = Entity.search params[:search],
      :page => params[:page],
      :per_page => 10,
      :order => "lastname ASC, firstname ASC",
      :match_mode => :boolean
  end
end

This makes the necessary replacements in the search string to allow the user to use OR, AND and single- or double-quoted strings in their search. Make sure you include the spaces around the "or" and "and" in your gsub calls! You don't want to be replacing "or" and "and" inside search terms! (Yes I made that mistake.) The /i makes the replacement case-insensitive. This is important because you don't know if the user will type "or" or "OR". Also, notice that we're using gsub! instead of just gsub. This is so that the params[:search] is modified directly. If we had used gsub instead, we'd have to assign each line to a variable, because gsub just returns the value, it doesn't modify the variable it's operating on directly.

So now I'm jazzed! The only thing that bugs me is that when I put a or b in my search box and click the "Search" button, when the search results come back, my search box shows a | b instead of the text I put in. While technically correct, I could see this confusing the tar out of end users. Not a big deal, but attention to detail is important! If I figure this one out, I'll let you know.

Now I'm going to eat lunch before I fall over on my keyboard!

See ya next post!

Chris

Sorting Sphinx

So we've gotten our results out and have them paginated nicely. That's spiffy, but I sure would like to see them sorted. So how do we do that?

The first thing we need to do is to tell Sphinx which fields are sortable. To do this, we'll add :sortable => true to the indexes in our entity model. So now the indexes look like this:

  define_index do
    indexes firstname, :sortable => true
    indexes lastname, :sortable => true
  end

Now we'll need to rebuild the indexes. Make sure that you shut down the Sphinx server if it's running or things won't work right (I think there's actually a way to rebuild the indexes on the fly without shutting the server down, but I'll have to investigate that later). To rebuild the indexes, we open the cmd window and type

rake ts:index

And then restart the server:

rake ts:start

Now, we'll just need to tell our controller to sort the results. In the home_controller.rb, the search now looks like this:

@entities = Entity.search params[:search],
     :page => params[:page],
     :per_page => 10,
     :order => "lastname ASC, firstname ASC"

You'll notice that in the :order, we've included the ASC for each field we want to index by. This is not normally necessary in a standard Rails .find command. However, Sphinx is a little quirky and requires this. If it's not there, Sphinx won't return any results. You can't imagine how long it took me to figure this out. Of course, if I'd read the thinking_sphinx instructions, I would have seen this and saved myself a lot of frustration. Stupid me!

Anyway, if we now reload our home page, we should see sorted results. Woot!

Next time, I'll be trying to figure out how to do some advanced searching with our simple search box.

See ya next post!

Chris

Working with Thinking_Sphinx

So now that Sphinx and Thinking_Sphinx are installed (right?), we can get on with actually using this stuff.

If you missed the post on installing Thinking_Sphinx, you can read it here. The post about installing Sphinx itself is here. And if you haven't yet watched the RailsCast about this, you need to do that!

Note that there are other Sphinx plugins for RoR, but based on all the stuff I've read on the web, I've chosen to use Thinking_Sphinx. We'll see how that decision pans out over the next few posts.

The first thing we need to do is generate the configuration file that Sphinx will use. To do that, I'll open the cmd windows, navigate to the root of my project and type

rake thinking_sphinx:configure

This should put out a file under our project\config directory named development.sphinx.conf. If that file isn't there, we've messed something up. The cmd output for the rake task should point to the issue. At this point, I'll assume that we've gotten this file created seccessfully.

Next, we'll need to specify some indexes in our models. Let's assume that we have an Entity model with the fields "firstname" and "lastname" (which I do). In our entity.rb model file, we'll add the following code:

entity.rb

class Entity < ActiveRecord::Base
  define_index do
    indexes firstname
    indexes lastname
  end
end

This should allow Sphinx to index the firstname and lastname fields of the entities table.

Now we'll need to generate the indexes. So we open the cmd window and type:

rake ts:index

This could take a few minutes if the dataset is very large. When it's done, it should have created some files in the project\db\sphinx\development directory. Mine are named entity_core.s*. Note that it would be a good idea to actually have the tables indexed in MySQL as Sphinx uses those indexes to retrieve data. From the docs it doesn't appear necessary, but will make the searching faster.

It's time to start the Sphinx service. In the cmd window we type:

rake ts:start

The Sphinx server should now be started. Now we can set up a search page and see if it works.

For simplicity, because we're just testing concepts here, I've set up a Home controller with its associated parts and pieces. The Home index page is my root page. I've put the search form in this page.

Here's a pastie of the relevent code. (yes, I'd like to show this code inline here, but I haven't figured that out yet. I'd appreciate any hints!)

You'll notice it's all very simple, basic stuff. I haven't even broken a sweat yet!

If we load the home page and type something into the search box and click the "Search" button, we're greeted with the search results. Rock on! This worked like a champ and wasn't very difficult at all. RoR kicks arse!

Next, we'll be dealing with pagination. Thinking_Sphinx automatically works with the will_paginate plugin. Here's the RailsCast on that one. Bone up and we'll muddle through that next time.

See ya next post!

Chris