Spatial Search

Did they spell special wrong, what is spatial?

A spatial query is unique because it utilizes points, lines, and polygons to make a relationship between geometries. There are many queries that can be applied to these data types but for this blog post we are going to focus on distance. Our goal in implementing a spatial query was to input coordinates (latitude/longitude) and a desired distance (10 miles for example) and get a set of results back that were within the desired distance.  How did we do this, keep reading 🙂

How are we utilizing spatial?

Currently on our site you are able to open a modal and search for travel agent. It initially does a spatial query using the visitor’s latitude and longitude back to our SQL database, this query gives us all the travel agents that are in the visitor’s area. It works but not as quickly as we would like it to be. SQL can be slow and that is where Solr comes in. We have rewritten the backend code of this functionality to use a new Solr index, this index is different from a typical Sitecore Solr index as it does not map back to items but rather database records in one of our in-house databases. The Solr Spatial code is just about to enter our QA process and it is not live just yet, should be another week depending on QA.

Radius Query

Very similar to a bounding box query although more strict, more precise.  If a coordinate is outside of the drawn radius than it will be excluded from your results.  Putting together the query with SolrNet is very similar, there is a method for a bounding box query and another method for a radius query, both taking the same input.  Which one to use?  It depends on the business need, and how strict do you need to be?
radius

Bounding Box Query

We utilize the bounding box query which is similar to a radius query but it connects the four extremes of the radius (north, west, east and south) and draws a box. The reason for using the bounding box is it has some give to it. It encompasses all of the points of interest but it may also include other points that are slightly outside of the required distance. You can see in the radius screenshot (above) there is a green marker that looks to be outside of the circle in the top left, this would be excluded in a radius query but included in a bounding box query.

boundingBox2

How to build a spatial index

Building a spatial index for your Sitecore solution is going to be completely different than what you are used to.  You don’t need Sitecore to generate a schema file for you but you do need to create one manually.  This schema file will outline what fields will make up a document in your index.  These fields will definitely include coordinates but may also include Title, Address, City, Zipcode, Phone Number or any other type of meta-data you will need.  I’m going to assume you have Solr up and running with at least 1 index.

  • Stop tomcat as we need to modify Solr.
  • Find an existing index, copy it and rename it to SpatialIndex.
  • Within the index you should find a core.xml, open that up and set the name equal to SpatialIndex.
  • Delete the data folder
  • In the root of your Solr directory you should see a solr.xml file.  You will want to open that file and add a core for your new index.
  • You will need to get a default schema.xml for your version of Solr.
  • Start tomcat and make sure Solr loads.

Defining your data and your crawler

  • In your global.asax file we need to create an IOC container.  We are using castle.windsor.
  • We need to define the object that we are going to be committing to Solr.  Here is an example of our class that is backed by a simple interface. Notice we are decorating the properties with “SolrUniqueKey”.
    public class AgentsIndexMapper: IAgentsIndexMapper
    {
    	public AgentsIndexMapper()
    	{
    			
    	}
    
    	public AgentsIndexMapper(IAgentsIndexMapper agentsIndex)
    	{
    		this.Id = agentsIndex.Id;
    		this.Name = agentsIndex.Name;
    		this.AgencyLocation = agentsIndex.AgencyLocation;
    		this.Address = agentsIndex.Address;
    		this.AddressPart2 = agentsIndex.AddressPart2;
    		this.CityName = agentsIndex.CityName;
    		this.State = agentsIndex.State;
    		this.StateCode = agentsIndex.StateCode;
    		this.Countrycode = agentsIndex.Countrycode;
    		this.Zipcode = agentsIndex.Zipcode;
    		this.PhoneNumber = agentsIndex.PhoneNumber;
    		this.EmailId = agentsIndex.EmailId;
    		this.IsPreferred = agentsIndex.IsPreferred;
    		this.Rank = agentsIndex.Rank;
    		this.WebsiteName = agentsIndex.WebsiteName;
    	}
    
    	///
    <summary>
    	/// AgencyId
    	/// </summary>
    
    	[SolrUniqueKey("agency_id")]
    	public string Id { get; set; }
    
    	///
    <summary>
    	/// Name of the Travel Agent
    	/// </summary>
    
    	[SolrField("agency_name")]
    	public string Name { get; set; }
    
    	///
    <summary>
    	/// Agency Location
    	/// </summary>
    
    	[SolrField("agency_loc")]
    	public SolrNet.Location AgencyLocation { get; set; }
    
    	///
    <summary>
    	/// Address
    	/// </summary>
    
    	[SolrField("address")]
    	public string Address { get;set; }
    
    	///
    <summary>
    	/// Address
    	/// </summary>
    
    	[SolrField("address2")]
    	public string AddressPart2 { get; set; }
    
    	///
    <summary>
    	/// City
    	/// </summary>
    
    	[SolrField("city")]
    	public string CityName { get;set; }
    
    	///
    <summary>
    	/// Two digit state code
    	/// </summary>
    
    	[SolrField("state_code")]
    	public string StateCode { get;set; }
    
    	///
    <summary>
    	/// State
    	/// </summary>
    
    	[SolrField("state")]
    	public string State { get;set; }
    
    	///
    <summary>phone
    	/// Zipcode
    	/// </summary>
    
    	[SolrField("country_code")]
    	public string Countrycode { get;set; }
    
    	///
    <summary>
    	/// Zipcode
    	/// </summary>
    
    	[SolrField("zipcode")]
    	public string Zipcode { get;set; }
    
    	///
    <summary>
    	/// Phone Number
    	/// </summary>
    
    	[SolrField("phone")]
    	public string PhoneNumber { get; set; }
    
    	///
    <summary>
    	/// Email address
    	/// </summary>
    
    	[SolrField("email")]
    	public string EmailId { get; set; }
    
    	///
    <summary>
    	/// IsPreferred
    	/// </summary>
    
    	[SolrField("ispreferred")]
    	public bool IsPreferred { get; set; }
    
    	///
    <summary>
    	/// Ranking
    	/// </summary>
    
    	[SolrField("rank")]
    	public Int32 Rank { get; set; }
    
    	///
    <summary>
    	/// Website name
    	/// </summary>
    
    	[SolrUniqueKey("website_name")]
    	public string WebsiteName { get; set; }
    }
    
  • Now we need a crawler class that will know how to gather a list of these objects. Here is our crawler class. It inherits from an abstract (first code block).  The AgentIndexCrawler is pretty easy to comprehend. We are calling the RebuildIndex method, gathering the objects from SQL then calling another RebuildIndex signature and passing in a list of those objects.  You can see the abstract class is calling our SolrUpdateContext class which is responsible for updating and commiting the documents.
    public abstract class Crawler<T> : ICrawler<T>
    {
    	private SolrUpdateContext<T> _solrUpdateContext;
    
    	public void Initialize(ISolrOperations<T> indexOperations, string indexName)
    	{
    		_solrUpdateContext = new SolrUpdateContext<T>(indexOperations, indexName, Logger);
    		Logger.Debug(string.Format("Crawler<T> - Initialize: Initialized the solrUpdateContext for index - {0}", indexName));
    	}
    
    	public virtual void Initialize(ISolrOperations<T> indexOperations, string indexName, int batchSize, bool batchingRequired)
    	{
    		_solrUpdateContext = new SolrUpdateContext<T>(indexOperations, indexName, Logger, batchSize, batchingRequired);
    		Logger.Debug(string.Format("Crawler<T> - Initialize: Initialized the solrUpdateContext for index - {0} with batching enabled.", indexName));
    	}
    
    	protected ILog Logger
    	{
    		get { return LogManager.GetLogger("Sitecore.Diagnostics.Crawling"); }
    	}
    
    	public void RebuildIndex(IEnumerable<T> docs)
    	{
    		if (_solrUpdateContext == null)
    		{
    			Logger.Debug(string.Format("Crawler<T> - RebuildIndex : Initialize is not called for {0}. So cannot rebuildIndex.", this));
    			return;
    		}
    		_solrUpdateContext.AddDocs(docs);
    	}
    }
    
    public class AgentsIndexCrawler : Crawler<AgentsIndexMapper>
    {
    	public void Initialize()
    	{
    		Initialize(IOC.Container.Resolve<ISolrOperations<AgentsIndexMapper>>(), AgentsIndex.IndexName);
    	}
    
    	public void RebuildIndex()
    	{
    		using (PivotalDataContext context = new PivotalDataContext())
    		{
    			List<AgentsIndexMapper> agentListing = new List<AgentsIndexMapper>();
    			IEnumerable<IAgentsIndexMapper> nonTCTAgents = context.p_Search_Travel_Agents_Having_LatLong().ToList();
    			if (!nonTCTAgents.Any())
    			{
    				Logger.Debug(
    					"AgentsIndexCrawler - RebuildIndex : No agents returned from the database using procedure p_Search_Travel_Agents_Having_LatLong()");
    			}
    			else
    			{
    				agentListing.AddRange(GetAgentListing(nonTCTAgents));
    			}
    
    			IEnumerable<IAgentsIndexMapper> TCTAgents = context.p_Search_TCT_Travel_Agents_Having_LatLong().ToList();
    			if (!TCTAgents.Any())
    			{
    				Logger.Debug("AgentsIndexCrawler - RebuildIndex : No agents returned from the database using procedure p_Search_TCT_Travel_Agents_Having_LatLong()");
    			}
    			else
    			{
    				agentListing.AddRange(GetAgentListing(TCTAgents));
    			}
    
    			if (!agentListing.Any())
    			{
    				Logger.Debug("AgentsIndexCrawler - RebuildIndex : No agents so not rebuilding");
    				return;
    			}
    
    			RebuildIndex(agentListing);
    		}
    	}
    
    	private List&amp;lt;AgentsIndexMapper&amp;gt; GetAgentListing(IEnumerable<IAgentsIndexMapper> agents)
    	{
    		List<AgentsIndexMapper> agentListing = new List<AgentsIndexMapper>();
    		try
    		{
    			foreach (IAgentsIndexMapper agent in agents)
    			{
    				agentListing.Add(new AgentsIndexMapper(agent));
    			}
    			if (!agentListing.Any())
    			{
    				Logger.Debug(string.Format("AgentsIndexCrawler - GetAgentListing : No agents in list after converting to a IAgentsIndexMapper"));
    			}
    			return agentListing;
    		}
    		catch (Exception e)
    		{
    			Logger.Debug("AgentsIndexCrawler - GetAgentListing : Exception " + e);
    			return agentListing;
    		}
    	}
    }
    
  • Create a class with a property of ISolrOperations.  This interface comes from the SolrNet library.  Take a look at our SolrUpdateContext class in the code block below.  This class is solely responsible for updating Solr. As you can see we have built it using generics making it extremely flexible so it can be used across indexes that take in different object types for documents.
    public class SolrUpdateContext<T> : ISolrUpdateContext<T>
    {
    		private readonly ISolrOperations<T> _indexOperations;
    		private readonly string _indexName;
    		private readonly ILog Logger;
    		private readonly int _batchsize = 0;
    		private readonly bool _batchingRequired = false;
    
    		public SolrUpdateContext() {}
    
    		public SolrUpdateContext(ISolrOperations<T> indexOperations, string indexName, ILog logger)
    		{
    			this._indexOperations = indexOperations;
    			this._indexName = indexName;
    			this.Logger = logger;
    		}
    
    		public SolrUpdateContext(ISolrOperations<T> indexOperations, string indexName, ILog logger, int batchsize, bool batchingRequired)
    		{
    			this._indexOperations = indexOperations;
    			this._indexName = indexName;
    			this.Logger = logger;
    			this._batchingRequired = batchingRequired;
    			this._batchsize = batchsize;
    		}
    
    		///
    <summary>
    		/// Add a doc to solr
    		/// </summary>
    
    		/// <param name="indexMapper"</param>
    		public void AddDoc(T indexMapper)
    		{
    			_indexOperations.Add(indexMapper);
    			_indexOperations.Commit();
    			Logger.Debug(string.Format("SolrUpdateContext&amp;amp;amp;lt;T&amp;amp;amp;gt; - AddDoc: Added a doc to index - {0}", _indexName));
    		}
    
    		///
    
    <summary>
    		/// Add docs to solr
    		/// </summary>
    
    		/// <param name="indexMappers"</param>
    		public void AddDocs(IEnumerable<T> indexMappers)
    		{
    			Logger.Debug(string.Format("SolrUpdateContext<T> - AddDocs: Indexing Process started for index - {0}. Time is {1}", _indexName, DateTime.Now));
    			if (_batchingRequired)
    			{
    				foreach (var group in indexMappers.Partition(1000))
    					_indexOperations.AddRange(group);
    			}
    			else
    			{
    				_indexOperations.AddRange(indexMappers);
    			}
    			
    			_indexOperations.Commit();
    			Logger.Debug(string.Format("SolrUpdateContext<T>- AddDocs: Indexing Process completed for index - {0}. Time is {1}", _indexName, DateTime.Now));
    		}
    	}
    

Querying the Index

This is the easy part. We need to use our IOC container which is added to the application to get to the SolrOperations object (this comes from SolrNet). By having this we can begin building our query. As you can see it is pretty straight forward.

ISolrOperations<AgentsIndexMapper> indexOperations = IOC.Container.Resolve<ISolrOperations<AgentsIndexMapper>>();

SolrQueryResults<AgentsIndexMapper> agents = indexOperations.Query(new SolrQueryByDistance("agency_loc", new Location(latitude, longitude), distance.Value, CalculationAccuracy.BoundingBox));
if (!agents.Any())
{
	Log.Debug(string.Format("AColletteAgentProvider- GetAgents(double latitude = {0}, double longitude= {1}, int? distance= {2}) : " +
		" No results returned from agentsindex.", latitude, longitude, (distance.HasValue && distance.Value > 0 ? distance: Distance)));
	return new List<IAgentsIndexMapper>();
}
This entry was posted in Solr and tagged on by .

About tbraga1983

Digital Professional, Sitecore MVP, started working with Sitecore back in 2008 when I worked for Velir. I joined Collette in 2013 to lead up their new website from the ground up. We started fresh on Sitecore 7, Solr, EXM and xDB.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s