Real Estate on Rails (Working with RETS Data)

In the past, I have worked on a lot of real estate websites. After having taken a few year break from doing any realty-related web development, which was a welcome break, I have recently started working on another real estate site. My hope going into this was that working with Rails would make the job much easier, and that things had improved somewhat since my last experiences. I was right, and the combination of Rails and RETS data has actually made the process almost fun, relative to the past.

IDX or RETS

The first order of business was to figure out the best way to get at the MLS (Multiple Listing Service) data that I wanted. In the past I had used IDX feeds, which translates to FTP access to a whole bunch of property data and images. Each day, someone or something would need to access this FTP server, download all the new files and update the website. It always seemed like there was a better way, and it turns out that there is.

Rather than an IDX feed, I have been working with RETS data, or Real Estate Transaction Standard data, which basically means that (finally) the MLS data is available as XML data via a web service. To me this is much more attractive, and seems to be a much more efficient way of updating MLS data.

RETS for Ruby Gem

Of course, the first thing I did was check to see if anyone had created a Ruby gem or Rails plugin for accessing and handling RETS data, and it turns out that someone has. The rets4r gem is what I have been experimenting with, and so far it has been extremely useful. I would be lying if I said I wasn’t tempted to rewrite the thing, and probably will in the future sometime, however it provides me with the functionality that I need to access the RETS data that we have access to and deal with that data as Ruby objects, which is nice. Of course, a large part of working with RETS is understanding how the data is formated and developing a strategy for keeping your site updated. We’ll start by requiring the rets4r gem in our Rails application:

config.gem "rets4r", :version => '>=0.8.5'

This will go in your config/environment.rb file.

The Daily Download

Even though RETS provides a web service that could probably be queried directly, it is still a much better idea to pull the MLS data down into a local database and go from there. That being said, I needed to come up with a strategy where all the properties that need to be displayed are pulled down originally (only once) and then each day check the RETS server for updates and deletions. These are essentially the same operations, in that the first time we pull down data we will be asking the RETS server for all properties that we have access to, and the daily updates will just ask for properties that have been updated since our last request. Let’s take a look at how we do that using the rets4r gem:

client = RETS4R::Client.new('http://retsprovider.com/rets/')
client.login('username', 'password') do |success|
  if success
    client.search('Property', 'A', "(FIELD=#{24.hours.ago.strftime('%Y-%m-%dT%T')}+)", {'Format' => 'COMPACT-DECODED'}) do |result|
      result.data.each do |row|
         property = Property.find_or_create_by_mls(row["MLS"])
         property.price = row["PRICE"]
         property.sqft = row["SQFT"]
         ...
         property.save
      end
   end
  client.logout
end

In this (very simplified) example, we are logging in to the RETS server, and running a search for all properties that have been modified in the last 24 hours. You will need to check your own MLS data provider to find out exactly what the various fields are, but I am using examples here. One thing to note is that I am searching the Property resource on the RETS server, and within that resource I am specifically searching within the A class of properties. In my case this happens to be residential properties, but you will want to do this for each resource class that you want to update (perfect place for a loop). Additionally, you will want to figure out what to replace FIELD with so that you are querying the last modified date field for your particular RETS server. Also, notice that I am adding the Format option to my search, and asking for the results to be returned in COMPACT-DECODED format. For my particular provider, this means that all lookup values for particular fields will be shown, rather than the obscure lookup codes that might otherwise be there. This is nice since dealing with lookup values is not really central to what I am trying to do, so that is handy. The rest is just about assigning the correct values to the correct attributes in my Property model, and we are off.

One thing that is worth mentioning is that not all the resource classes have the same fields. For example the field denoting the street surface type for a residential property might be called FEAT00001. However, the same street surface type field for a commercial property might be FEAT00003. It’s important to track down each field that you want for each resource class. It is done this way to account for different values and flexibility between property types, but can be a pain. In my case, there are about 15 fields that I want data for in my Property model that have to be mapped differently depending on the resource class. It’s a fact of life.

Getting at Images

So the script above, if run daily, will check the RETS server for any properties that have been modified in the last 24 hours and pull them down to our local database. Not a bad start. The next thing we need this script to do is to download the images for each of the properties. Typically there will be a field for photos_count or something along those lines so that you can tell for each property if it has associated photos. Using that field, we can add the following to our script, just before the property.save line:

if row["PHOTOS_COUNT"].to_i > 0
  photo_index = 1
  row["PHOTOS_COUNT"].to_i.times do
    client.get_object('Property', 'Photo', "#{row["ID"]}:#{row_index}, 0) do |photo|
      handle_object(photo)
    end
    photo_index = photo_index + 1
  end
end

This one is pretty basic as well, and you will need to write your own handle_object method depending on what you want to do with the images once they are downloaded. There is a way of pulling down all the images in a single request, although I could not coax the rets4r gem to play nicely with that one.

Deleting properties

One thing that was interesting with the RETS feed I have access to is that the property status is not really used. In some cases it is there, but it cannot be used to see if a property has been deleted. Since our feed only spits out properties that we have access to (which does not include deleted records), we have to come up with some way to delete properties that have been removed.

Interestingly enough, the company that provides my RETS feed has a solution, although I can’t help but wonder if there isn’t a cleaner way. They suggest that each day as part of the daily update, we query the RETS server for all the properties that we should be displaying, grabbing only one field (like the id or mls number) to make the query more efficient. We then compare that to the contents of our local database, and any records that we have but do not appear in the daily list need to be deleted. Backwards, huh? Here’s what it looks like:

client.search('Property', 'A', "(FIELD=1950-01-01T00:00:00+)", {"Select" => "MLS_NO"}) do |result|
  local_properties = Property.find(:all, :select => 'mls')
  local_properties.each do |property|
    property.destroy unless result.data.index(property.mls)
    # Don't forget to delete all images too.
  end
end

Again, depending on how you are storing or managing your images, you will want to be sure and delete them when you delete an associated property. However, our loop simply grabs a list of all the MLS numbers that we should have in our local database. Then we loop through all of our property records, deleting any that have MLS numbers that aren’t in our list. Easy enough, although something about it seems ugly to me.

Development As Usual

Now that the challenging part is over, we can run our script and watch as properties fill our database and images come down as well. From here, developing a real estate site becomes like developing any other data-driven site. Rather than spending our time worrying about how to deal with MLS data, we can spend our time developing a great application.

From my standpoint, the hassle of dealing with MLS providers, and the intimidation of daily MLS updates are now removed from the equation, which means that real estate sites are no longer something to be feared. Although, that is from a technical point of view. You still need to deal with realtors to build their sites, and I’m not sure I’ve overcome that one yet.

Published by Brian Getting on Wednesday, April 15, 2009.
Subscribe to our blog RSS feed.