Boolean Searches with Sphinx

Well, we now have a nice little search system for our entity data. But what if the user want to search for "a" OR "b". What we have so far doesn't handle that. So let's make the modifications we need to make that work.

All I have to say is it's a good thing that the RailsCast on Sphinx addressed this, because I couldn't find this in the thinking_sphinx online docs anywhere. Maybe I just missed it, but I've looked 20 times and just don't see it. (Don't think I'm knocking thinking_sphinx in any way. It's still awesome!)

Anyway, what we'll need to do is modify our search call in our home_controller.rb file. The magic word is :match_mode. So here's the new search line in the controller:

@entities = Entity.search params[:search],
   :page => params[:page],
   :per_page => 10,
   :order => "lastname ASC, firstname ASC",
   :match_mode => :boolean

Notice I've set the match mode to :boolean. This will allow us to do boolean searches. So what is a boolean search? Well, for one thing, it allows us to search for "a" OR "b". However, thinking_sphinx doesn't use the word OR, it uses a pipe (|) symbol. Hmm...I don't think my users will want to do this. I know I'm used to putting the word OR in most of the searches I do. So we'll have to do something about that later. Boolean searches also allow us to use AND, NOT and grouping operators. Those would be &, ! or -, and (). Again, not something most of my users will be able to/like to deal with. (Note that there are other match modes, but I'm not going to be using those. For further information on those, look here.)

So what do we do to allow our end user to use boolean syntax they are familiar with (assuming they're not programmers)? We have to do some string replacement in our code. For simplicity's sake right now, I'm just going to put this in my controller. I'll probably refactor it out later because I don't think that's really the appropriate place for it. Course, I'm a n00b, so what do I know?

Here's what I did. There may be better ways, and maybe I'll stumble on to those later, but for now, this should give us something to work with.

class HomeController < ApplicationController
  def index
    params[:search].gsub!(/ or /i," | ")
    params[:search].gsub!(/ and /i," & ")
    params[:search].gsub!(/\"(.*)\"/,'(\1)')
    params[:search].gsub!(/\'(.*)\'/,'(\1)')
    @entities = Entity.search params[:search],
      :page => params[:page],
      :per_page => 10,
      :order => "lastname ASC, firstname ASC",
      :match_mode => :boolean
  end
end

This makes the necessary replacements in the search string to allow the user to use OR, AND and single- or double-quoted strings in their search. Make sure you include the spaces around the "or" and "and" in your gsub calls! You don't want to be replacing "or" and "and" inside search terms! (Yes I made that mistake.) The /i makes the replacement case-insensitive. This is important because you don't know if the user will type "or" or "OR". Also, notice that we're using gsub! instead of just gsub. This is so that the params[:search] is modified directly. If we had used gsub instead, we'd have to assign each line to a variable, because gsub just returns the value, it doesn't modify the variable it's operating on directly.

So now I'm jazzed! The only thing that bugs me is that when I put a or b in my search box and click the "Search" button, when the search results come back, my search box shows a | b instead of the text I put in. While technically correct, I could see this confusing the tar out of end users. Not a big deal, but attention to detail is important! If I figure this one out, I'll let you know.

Now I'm going to eat lunch before I fall over on my keyboard!

See ya next post!

Chris

Sorting Sphinx

So we've gotten our results out and have them paginated nicely. That's spiffy, but I sure would like to see them sorted. So how do we do that?

The first thing we need to do is to tell Sphinx which fields are sortable. To do this, we'll add :sortable => true to the indexes in our entity model. So now the indexes look like this:

  define_index do
    indexes firstname, :sortable => true
    indexes lastname, :sortable => true
  end

Now we'll need to rebuild the indexes. Make sure that you shut down the Sphinx server if it's running or things won't work right (I think there's actually a way to rebuild the indexes on the fly without shutting the server down, but I'll have to investigate that later). To rebuild the indexes, we open the cmd window and type

rake ts:index

And then restart the server:

rake ts:start

Now, we'll just need to tell our controller to sort the results. In the home_controller.rb, the search now looks like this:

@entities = Entity.search params[:search],
     :page => params[:page],
     :per_page => 10,
     :order => "lastname ASC, firstname ASC"

You'll notice that in the :order, we've included the ASC for each field we want to index by. This is not normally necessary in a standard Rails .find command. However, Sphinx is a little quirky and requires this. If it's not there, Sphinx won't return any results. You can't imagine how long it took me to figure this out. Of course, if I'd read the thinking_sphinx instructions, I would have seen this and saved myself a lot of frustration. Stupid me!

Anyway, if we now reload our home page, we should see sorted results. Woot!

Next time, I'll be trying to figure out how to do some advanced searching with our simple search box.

See ya next post!

Chris

Installing Sphinx on Windows

So now we come to the part of installing Sphinx on Windows. It turns out that it was not terribly difficult, but I had a hard time finding instructions on the web, so I'll post my steps here.

1. I downloaded Sphinx from the official Sphinx download site (I downloaded Win32 release binaries with MySQL support: sphinx-0.9.8-win32.zip).

2. I unzipped the file into a temp folder.

3. I copied all of the files from the \sphinx-0.9.8-win32\bin directory into C:\Dev\Sphinx (you can put them in the directory of your choice).

4. I added C:\Dev\Sphinx to my Windows PATH

And Sphinx is installed. Nice, simple, easy.

In the next post, I'll see if I can actually set up some indexes and search with Thinking_Sphinx.

See ya next post!

Chris

Getting Git and Thinking Sphinx

As I was looking into what I needed to do to use Sphinx in our project, I noticed that the Thinking Sphinx plugin uses Git. I don't have Git. So I'll now be getting Git. (yeah, yeah. it sounds funny. Big laugh. Stop it.)

This poses a particular problem for me because I am using a (**gasp**) Windows machine (quit laughing). Hey, right now it's buy a Mac or feed my kids. The kids win.

It turns out that there is a project called msysgit that is basically Git for Windows. So I'm downloading the installer from here. I'm hoping this is what I need to be able to install the Thinking Sphinx plugin.

Stand by...

Stand by some more...

Ok. I executed the .exe file and installed Git. I chose the option to run from the Windows cmd window (seemed logical to me) so that Git would be in my path. Installation was a snap! I've been officially forked.

Now I've opened my cmd window, navigated to the root directory of my project (this is important!) and typed in the command to install the Thinking Sphinx plugin:

ruby script/plugin install git://github.com/freelancing-god/thinking-sphinx.git

Here comes the big moment...trepidation...anxiety...the shakes..."Enter"...Woohoo! I now have the Thinking Sphinx plugin in my vendors/plugins directory under my project. That wasn't too bad at all. Next time, I'll be downloading and installing Sphinx itself. Call the comedy club, cause that should be a riot!

See ya next post!

Chris

Here we go!

So I just started a large RoR project and thought I would set up this blog to share my frustrations and triumphs in case anyone else starting out in the Ruby world might benefit.

To be honest, this isn't completely magnanimous. I needed a place to log stuff that I've done so I don't forget it. But I figured a blog would be better than a notebook because someone else might learn something, too.

The project is essentially a Contact Management System (pretty dry stuff I know). But it's a large enough project with enough features that I thought it would be a good foray into this world and give me lots of material for a blog series.

Our first challenge is making sure that the system's search features are very fast. We're talking about large amounts of data here (millions of records across multiple tables). The users will need to be able to search for terms across multiple fields in those tables and it will have to be fast!

So my first step is to generate large amounts of realistic test data. Not a very pleasant prospect to be sure. Generating realistic names, addresses, phone numbers, etc. is not simple. Or so I thought. After starting to monkey around with creating a ruby library to generate this stuff for me (without a lot of progress), I stumbled on this railscast about setting up realistic test data. Wow! Not only is this my favorite rails site, but this railscast was posted the day after I started trying to do this! How did he know? Ryan Bates does an awesome job explaining this stuff. I would highly recommend that all RoR n00bs watch every episode!

I installed the two gems and followed the simple instructions and viola! my data is populated. It took me just a couple of hours to get a complex database set up with lots of realistic looking data. And that included setting up all of my models! (I know, RoR masters probably could have done it faster, but I was jazzed!) Very, Very Cool!

Next, I'm going to be experimenting with the search engines. Thinking Sphinx looks very promising for what we're doing. I'll post about my experience with that in the next blog. In the mean time, I'd recommend watching this railscast about Thinking Sphinx (RailsCasts again).

See ya next post!

Chris