Cucumber and Subdomains Revisited (the Capybara chronicles)

Well, I've been working steadily on RoR development for the last year.  I finally landed a job in November as a full time RoR developer, and it's AWESOME!

But that's not the point of this post.  In my last post, I had just figured out how to get Cucumber with Webrat to work with subdomains.  Then I needed to test JS.  But I don't want to run Selenium or Culerity or anything else for ALL my tests.  That would be waaaaay to slow.  Sooooo....over to Capybara I go.

And guess what?  Just about every test failed, because subdomains don't work the same.  Crap!  I'm not knocking Capybara.  I think it's awesome.  The benefits far outweigh this little foible for sure!  I just didn't anticipate this issue.

Just to be clear, the issue is not calling a 'visit' to a hard-coded url with a subdomain.  That's easy and seems to work just fine.  It's when you get redirected to other pages after that.  The subdomain isn't carried along and things start to break down.

So off to the research pits I go.  And I don't find much that helps.  Capybara's Github page does give a suggestion (https://gist.github.com/643a758320a2926bd2ed) about how to handle this with default_url_options.  I tried it.  I couldn't get it to work.  And I didn't really like it anyway.  Probably not a big deal, but it just offends my sensibilities to change code for tests.  Back to Google...still nothing.

So off to the Capybara code itself.  After some digging, I've come up with this solution using default_host and app_host that handles subdomains for the regular Capybara tests (Rack::Test) and for Capybara's Culerity driver.  Don't forget to add entries to your /etc/hosts file for all of the subdomains you'll be using with Culerity, or things will go boom.

One thing I'd like to improve about this is the port being a variable pulled from Capybara somehow (instead of hard-coded to 9887), but I couldn't figure out how to do that, and it isn't terribly important to me right now.  I'll be happy to update the gist if anyone figures this out.  Just let me know.

I didn't test this with any of the other drivers (like Selenium) because I'm only using Culerity so far.  Haven't had any reason to do any others yet.  If anyone else tests this with any of the other drivers, let me know, and I'll edit the post with that info.

Edit
Got a question about what relevant gems I'm using and realized that was important information!  So here they are:

Cucumber - 0.7.2
Capybara - 0.3.8
Langalex's Culerity - 0.2.10

Cucumber and Subdomains

So cucumber has become my new best buddy. How can you not like this for integration testing? A brain dead monkey could get simple tests going. And that must be true, because I got some working.

In case you missed em, there's some RailsCasts (yeah, Ryan Bates is my hero) on Cucumber and Webrat here, here and here

So there I am plugging along on some tests in my latest project and just giggling like a little school girl at the simplicity of it all. Then I needed to test my subdomain routing. Brick wall. So now instead of giggling, it was more like stuck-pig-squeal. Crap.

So Google to the rescue, right? Not so fast. The problem is that cucumber is, relatively speaking, rather new, so there's not a ton of stuff up on the web yet. The other little sticky-widget I ran into was the fact that I was using 0.3.1 (which had been released like the day before) and it had changed the way some things work. So the only post I was able to find about this that contained a solution no longer worked (not to mention the fact that it was way to complicated for my lazy butt).

After struggling with this for a couple of days, I finally ran across a post that pointed me in the right direction (sorry I can't remember the site to give credit where credit is due). So I worked the kinks out and ended up with a super-simple way to test subdomains in Cucumber. So far this has worked for everything I've thrown at it.

I'd love some feedback if anyone can think of ways to improve this or sees potential problems I haven't thought out.

Boolean Searches with Sphinx

Well, we now have a nice little search system for our entity data. But what if the user want to search for "a" OR "b". What we have so far doesn't handle that. So let's make the modifications we need to make that work.

All I have to say is it's a good thing that the RailsCast on Sphinx addressed this, because I couldn't find this in the thinking_sphinx online docs anywhere. Maybe I just missed it, but I've looked 20 times and just don't see it. (Don't think I'm knocking thinking_sphinx in any way. It's still awesome!)

Anyway, what we'll need to do is modify our search call in our home_controller.rb file. The magic word is :match_mode. So here's the new search line in the controller:

@entities = Entity.search params[:search],
   :page => params[:page],
   :per_page => 10,
   :order => "lastname ASC, firstname ASC",
   :match_mode => :boolean

Notice I've set the match mode to :boolean. This will allow us to do boolean searches. So what is a boolean search? Well, for one thing, it allows us to search for "a" OR "b". However, thinking_sphinx doesn't use the word OR, it uses a pipe (|) symbol. Hmm...I don't think my users will want to do this. I know I'm used to putting the word OR in most of the searches I do. So we'll have to do something about that later. Boolean searches also allow us to use AND, NOT and grouping operators. Those would be &, ! or -, and (). Again, not something most of my users will be able to/like to deal with. (Note that there are other match modes, but I'm not going to be using those. For further information on those, look here.)

So what do we do to allow our end user to use boolean syntax they are familiar with (assuming they're not programmers)? We have to do some string replacement in our code. For simplicity's sake right now, I'm just going to put this in my controller. I'll probably refactor it out later because I don't think that's really the appropriate place for it. Course, I'm a n00b, so what do I know?

Here's what I did. There may be better ways, and maybe I'll stumble on to those later, but for now, this should give us something to work with.

class HomeController < ApplicationController
  def index
    params[:search].gsub!(/ or /i," | ")
    params[:search].gsub!(/ and /i," & ")
    params[:search].gsub!(/\"(.*)\"/,'(\1)')
    params[:search].gsub!(/\'(.*)\'/,'(\1)')
    @entities = Entity.search params[:search],
      :page => params[:page],
      :per_page => 10,
      :order => "lastname ASC, firstname ASC",
      :match_mode => :boolean
  end
end

This makes the necessary replacements in the search string to allow the user to use OR, AND and single- or double-quoted strings in their search. Make sure you include the spaces around the "or" and "and" in your gsub calls! You don't want to be replacing "or" and "and" inside search terms! (Yes I made that mistake.) The /i makes the replacement case-insensitive. This is important because you don't know if the user will type "or" or "OR". Also, notice that we're using gsub! instead of just gsub. This is so that the params[:search] is modified directly. If we had used gsub instead, we'd have to assign each line to a variable, because gsub just returns the value, it doesn't modify the variable it's operating on directly.

So now I'm jazzed! The only thing that bugs me is that when I put a or b in my search box and click the "Search" button, when the search results come back, my search box shows a | b instead of the text I put in. While technically correct, I could see this confusing the tar out of end users. Not a big deal, but attention to detail is important! If I figure this one out, I'll let you know.

Now I'm going to eat lunch before I fall over on my keyboard!

See ya next post!

Chris

Sorting Sphinx

So we've gotten our results out and have them paginated nicely. That's spiffy, but I sure would like to see them sorted. So how do we do that?

The first thing we need to do is to tell Sphinx which fields are sortable. To do this, we'll add :sortable => true to the indexes in our entity model. So now the indexes look like this:

  define_index do
    indexes firstname, :sortable => true
    indexes lastname, :sortable => true
  end

Now we'll need to rebuild the indexes. Make sure that you shut down the Sphinx server if it's running or things won't work right (I think there's actually a way to rebuild the indexes on the fly without shutting the server down, but I'll have to investigate that later). To rebuild the indexes, we open the cmd window and type

rake ts:index

And then restart the server:

rake ts:start

Now, we'll just need to tell our controller to sort the results. In the home_controller.rb, the search now looks like this:

@entities = Entity.search params[:search],
     :page => params[:page],
     :per_page => 10,
     :order => "lastname ASC, firstname ASC"

You'll notice that in the :order, we've included the ASC for each field we want to index by. This is not normally necessary in a standard Rails .find command. However, Sphinx is a little quirky and requires this. If it's not there, Sphinx won't return any results. You can't imagine how long it took me to figure this out. Of course, if I'd read the thinking_sphinx instructions, I would have seen this and saved myself a lot of frustration. Stupid me!

Anyway, if we now reload our home page, we should see sorted results. Woot!

Next time, I'll be trying to figure out how to do some advanced searching with our simple search box.

See ya next post!

Chris

Paginating Sphinx

In the last post, you'll remember that I mentioned the fact that Thinking_Sphinx automatically works with will_paginate. So this time, we're going to explore how to get pagination working with our spiffy new Sphinx searching.

The first thing we'll need to do is to get will_paginate. Will_paginate has been around for a while, but has recently (relatively speaking) been moved to GitHub. Brilliant move IMHO. However, it's name also changed to mislav-will_paginate. If you don't already have the old will_paginate gem installed, that's not relevant to you. If you do, you'll want to remove the old will_paginate before installing this one (unless, of course, removing it will dork up your other projects, in which case you're on your own. Sorry).

Let's head over to the (very nicely done) will_paginate installation instructions page. You'll notice there are several options for installing it. I'm going to use "Installing the Gem manually". You go with whatever floats your boat.

So I open my cmd window and type:

gem sources -a http://gems.github.com

because I don't have github in my gem sources. I shouldn't ever have to do this for this machine again. And good thing, because that was just brutal...

Anyway, now I type:

gem install mislav-will_paginate

Now the gem is installed. Simple goodness...

Open the project\config\environment.rb file and add the following line AFTER the end of the initializer block (if you put it inside the block, it won't work!)

require "will_paginate"

I'm pretty sure at this point, we need to restart whatever rails server we're using (I'm using WEbrick for testing). If not, just call me stupid and move on!

Now we're all set up to use pagination. This should be pretty simple (even for me)...

The pagination stuff will be added in my views/home/index.html.erb. Just put the following line wherever you want the pagination links to show up (I'm putting mine at the bottom, so there):

<%=will_paginate @entities%>

Now, if we reload the page (and we have multiple pages of data in the result), we'll see the pagination links. If you don't see any, you probably don't have enough data. I'd recommend watching the RailsCast I mentioned in my first post to get some data in your tables. Or you can be masochistic and put it in by hand. Go ahead, we'll wait...

Ok, so now we see the pagination links, right? Just click on one of those links and...WTH???? I keep getting the same data back regardless of which page link I select! Well, that's because I haven't had my coffee today and forgot an important little change. We've got to tell the controller to give us the next page, duh. So in the controllers/home_controller.rb, in the index method, I'm going to make the search command look like this:

@entities = Entity.search params[:search],
  :page => params[:page]

Now if we click on the pagination links, they should work.

The one thing I'm not liking on this is that it's showing me 20 entities per page. I only want to see 10. So one more addition to the controllers/home_controller.rb:

@entities = Entity.search params[:search],
  :page => params[:page],
  :per_page => 10

Time for a celebratory beer...

See ya next post!

Chris

Working with Thinking_Sphinx

So now that Sphinx and Thinking_Sphinx are installed (right?), we can get on with actually using this stuff.

If you missed the post on installing Thinking_Sphinx, you can read it here. The post about installing Sphinx itself is here. And if you haven't yet watched the RailsCast about this, you need to do that!

Note that there are other Sphinx plugins for RoR, but based on all the stuff I've read on the web, I've chosen to use Thinking_Sphinx. We'll see how that decision pans out over the next few posts.

The first thing we need to do is generate the configuration file that Sphinx will use. To do that, I'll open the cmd windows, navigate to the root of my project and type

rake thinking_sphinx:configure

This should put out a file under our project\config directory named development.sphinx.conf. If that file isn't there, we've messed something up. The cmd output for the rake task should point to the issue. At this point, I'll assume that we've gotten this file created seccessfully.

Next, we'll need to specify some indexes in our models. Let's assume that we have an Entity model with the fields "firstname" and "lastname" (which I do). In our entity.rb model file, we'll add the following code:

entity.rb

class Entity < ActiveRecord::Base
  define_index do
    indexes firstname
    indexes lastname
  end
end

This should allow Sphinx to index the firstname and lastname fields of the entities table.

Now we'll need to generate the indexes. So we open the cmd window and type:

rake ts:index

This could take a few minutes if the dataset is very large. When it's done, it should have created some files in the project\db\sphinx\development directory. Mine are named entity_core.s*. Note that it would be a good idea to actually have the tables indexed in MySQL as Sphinx uses those indexes to retrieve data. From the docs it doesn't appear necessary, but will make the searching faster.

It's time to start the Sphinx service. In the cmd window we type:

rake ts:start

The Sphinx server should now be started. Now we can set up a search page and see if it works.

For simplicity, because we're just testing concepts here, I've set up a Home controller with its associated parts and pieces. The Home index page is my root page. I've put the search form in this page.

Here's a pastie of the relevent code. (yes, I'd like to show this code inline here, but I haven't figured that out yet. I'd appreciate any hints!)

You'll notice it's all very simple, basic stuff. I haven't even broken a sweat yet!

If we load the home page and type something into the search box and click the "Search" button, we're greeted with the search results. Rock on! This worked like a champ and wasn't very difficult at all. RoR kicks arse!

Next, we'll be dealing with pagination. Thinking_Sphinx automatically works with the will_paginate plugin. Here's the RailsCast on that one. Bone up and we'll muddle through that next time.

See ya next post!

Chris

Installing Sphinx on Windows

So now we come to the part of installing Sphinx on Windows. It turns out that it was not terribly difficult, but I had a hard time finding instructions on the web, so I'll post my steps here.

1. I downloaded Sphinx from the official Sphinx download site (I downloaded Win32 release binaries with MySQL support: sphinx-0.9.8-win32.zip).

2. I unzipped the file into a temp folder.

3. I copied all of the files from the \sphinx-0.9.8-win32\bin directory into C:\Dev\Sphinx (you can put them in the directory of your choice).

4. I added C:\Dev\Sphinx to my Windows PATH

And Sphinx is installed. Nice, simple, easy.

In the next post, I'll see if I can actually set up some indexes and search with Thinking_Sphinx.

See ya next post!

Chris