How to Setup Production-Ready Solr Instance on Linux for Sitecore

Vasiliy Fomichev

In How To, Installation Posted June 26, 2014

Words Linux and Solr seem to have some strange effect to them that makes .NET developers run for their lives. Recently Sitecore 7 came out with a new Solr search provider, allowing to supercharge the search functionality of the CMS. Since Solr runs on Java, it is therefore platform independent, and can be easily installed on Windows, however, Jetty, the default Java web browser that comes prepackaged with Solr, run best in its native habitat – Linux. If you are a developer that was tasked with setting up Solr on a development or production server – fear not, and read on for the exact steps and best practices!

In my past life I was doing a lot of Linux development for Dell, testing and developing packages for Red Hat Enterprise Linux, right after they split off into a commercial edition (Yes, I was once one of the Windows hating and .NET mocking developers, until, I realized that .NET developers actually had real girlfriends 🙂  ). In short, anything with a “Tux badge” is near and dear to my heart.

Recently on one of the Sitecore projects we had to setup a Solr environment for our client on RHEL 6.5, and after a brief research it turned out there was not a single post on this topic. There is a great post on how to setup Solr in a local Windows environment, but nothing on how to setup Solr for Sitecore on a Linux machine. So after speaking to some of my “girlfriendless” Linux guru friends and swallowing a  666-page “Solr in Action” PDF (that page count really threw me off), here comes the step by step tutorial on how to setup Solr for Sitecore on remote RHEL servers (steps are very similar for other Linux flavors.)

Sitecore Solr Setup Requirements Overview

Sitecore Solr production setup diagram.

In this post, we will be following these requirements:

• Setup two load-balanced Solr instances
• Enable index replication
• Ensure the Solr starts automatically upon the machine reboot
• Setup some initial performance enhancements

This type of setup might seem a bit overwhelming at first, however, one of the ideas behind the development of Solr was to make the setup and integration of the search engine into any environment as easy as possible – oh, and did they succeed!

Lastly before we begin, here are the assumptions:

• Both servers have RHEL 6 and Java already installed
• Both servers are on the same internal network
• We have access to the server via SSH with elevated privileges (never work as a root for security and safety purposes)

Tools Used for the Tutorial

In this tutorial we will be using the following tools:

• Putty  – in this tutorial it will be used for accessing the RHEL servers remotely using SSH
• WinSCP – although not required, it may come in handy for file transfers using SFTP from your local machine to the remote servers, or vice versa.

Setting Up Solr Core for Sitecore

1. Download the desired version of the Solr server
   For this step if the Linux server has access to the internet you can simply run the following command (relace VERSION with the desired Solr version) to download the Solr archive (otherwise, download it to your local machine and upload to the remote servers using WinSCP).

   wget  http://apache.claz.org/lucene/solr/4.9.0/solr-VERSION.tgz

2. Extract the content of the downloaded archive into the /opt directory

   tar zxf /path/to/the/solr.archive.tgz /opt/solr
   

   If you get an error regarding the lack of proper access rights, use the sudo prefix to elevate your privileges.

3. Rename the Solr directory for convenience (there are many other ways to find out the version of Solr instance installed)

   mv /opt/solr-{version} /opt/solr
   

4. Rename the Jetty instance folder to sitecore-master 

   mv /opt/solr/example /opt/solr/sitecore-master

5. Rename the example core directory to sitecore, the new core name

   mv /opt/solr/sitecore-master/solr/collection1 /opt/solr/sitecore-master/solr/sitecore

6. Edit the content of core.properties file to have the name of the core – sitecore

   vi /opt/solr/sitecore-master/solr/sitecore/core.properties
   

 Quick Tip: Solr uses an autodiscovery mechanism for finding cores, thus, to create a new core, simply copy one of the existing core directories and modify the core.properties file with a new core name.

I am using vi – the “OG” of the text editors in the Linux world. Some of its commands are a bit outdated ad may seem confusing (somehow they made sense back in the day). For instance, once you open a file using the vi command, you are placed in the read-only mode, to switch to insert mode, hit Shift+I. Here you can use your arrow key to move around and backspace or delete keys to remove characters. To exit the Insert Mode, hit Esc, and finally, to save changes – Shift+: and then wq (stands for write and quit) and hit Enter (weren’t these steps obvious?).

This is a good stopping point to verify that our core installation of Solr is properly setup. Let’s start our Solr server for the first time:

java –jar /opt/solr/sitecore-master/start.jar

Now let’s make sure that that Solr server dashboard is accessible by navigating to http://{SERVER IP}:8983/solr using your local browser (we’ve had some issues with accessing the dashboard using Lynx). If the server dashboard comes up without any errors – repeat steps 1-6 on the other server (change sitecore-master to sitecore-slave for the instance name on the second server), otherwise, start troubleshooting the error received.

Solr Dashboard.

Script for Restarting Solr on Server Reboot

Unfortunately, although both servers are currently up and running, with the current configuration we would have to run the command above to restart it after each server reboot. Of course, as good developers we want to automate that, and here is how –

First, let’s stop the currently running server in the terminal by hitting Ctrl+C. Although the server seems to be stopped and if you were to go to the dashboard URL you would get a 404 error, the port 8983 (default Solr port) did not get released, because the service is running on the background. So at this point if you were to try and run “java –jar /opt/solr/sitecore-master/start.jar”, you would get an error about the port not being available. In order to release the port we need to terminate the process completely, but first, we need to get the process Id:

lsof –i :8983

We now have a list of records with process(es)  using the port. Knowing the process Id we can now terminate it and free up the port

kill -9  999

Where 999 should be replaced by the actual process id.

Now that we have the manual start and stop process figured out, let’s proceed with creating the Solr autostart script to automate it.

1. Create a file /etc/init.d/solr

   vi /etc/init.d/solr
   

2. Add the following content to the newly created file:

   #!/bin/sh
   
    
   
   # Starts, stops, and restarts Apache Solr.
   
   #
   
   # chkconfig: 35 92 08
   
   # description: Starts and stops Apache Solr
   
    
   
   SOLR_DIR="/opt/solr/solr-vp01"
   
   JAVA_OPTIONS="-Xmx2g -Xms2g -DSTOP.PORT=8079 -DSTOP.KEY=sitecore -jar start.jar"
   
   LOG_FILE="/var/log/solr.log"
   
   JAVA="/usr/bin/java"
   
    
   
   case $1 in
   
   start)
   
   echo "Starting solr..."
   
   cd $SOLR_DIR
   
   find solr/*/data/ -type f -exec cat {} \; > /dev/null
   
   $JAVA $JAVA_OPTIONS 2> $LOG_FILE &
   
   ;;
   
   stop)
   
   echo "Stopping solr..."
   
   cd $SOLR_DIR
   
   $JAVA $JAVA_OPTIONS --stop
   
   ;;
   
   restart)
   
   $0 stop
   
   sleep 1
   
   $0 start
   
   ;;
   
   *)
   
   echo "Usage: $0 {start|stop|restart}" >&2
   
   exit 1
   
   ;;
   
   esac
   

This script supports commands start, stop and restart. Notice the -Xmx2g -Xms2g options specified in the startup command; these parameters set the minimum and maximum size of system memory dedicated to Solr to 2GB eliminating any room for scaling. Wait, but scaling is good! In some cases – yes, however, here it’s done to avoid performance loss during the memory scaling process, which can be a bit taxing in some cases.

Something smart this script also does is it loads indexes for all cores into memory right before starting Solr. This allows avoiding those first several slow queries that happen on a freshly started Solr index. By default to process these initial requests Solr would first have to read the index file from disk and then put it into memory.  With our script queries will perform lighting fast from the get-go.

3. Let’s make the file executable

   chmod a+rx /etc/init.d/solr

4. Create a service instance from the newly created file

   chkconfig  --add  /etc/init.d/solr

That’s it! Now we can test our script by running the following command checking that the Solr dashboard is back up

service solr start

If everything starts fine – repeat steps 1-4 on the second server, otherwise, start troubleshooting the error message.

How to Setup Solr Instance Replication

One last step in our setup is to configure Solr index replication. We will setup the sitecore-master and sitecore-slave instances accordingly.

Solr has a great feature called SolrCloud that came out recently and allows to easily manage complex replication and sharding setups (environment digrams, centralized configuration, ZooKeeper…etc.). Although it might seem as the tool for the job, using it in our scenario would be similar to driving up in a tank to a fist fight. SolrCloud would bring a lot of maintenance and setup costs when setup properly, which would outweigh any benefits imaginable in our case.

In order to configure the replication in a two-server scenario it is recommended to simply update the solrconfig.xml inside each of the cores on the master and slave as follows:

 <requestHandler name="/replication" class="solr.ReplicationHandler" > 
       <lst name="slave">
         <str name="enable">true</str>
	 <str name="masterUrl">http://{SERVER IP}:8983/solr/sitecore</str>
         <str name="pollInterval">00:00:60</str>
       </lst>
  </requestHandler>

<requestHandler name="/replication" class="solr.ReplicationHandler" > 
       <lst name="master">
         <str name="replicateAfter">commit</str>
         <str name="replicateAfter">startup</str>
         <str name="replicateAfter">optimize</str>
         <str name="confFiles">schema.xml,stopwords.txt</str>
       </lst>
  </requestHandler>

Voila! We are done! Now to test the environment, I would recommend to restart both servers and ensure that both Solr instances get restarted automatically and the replication configuration has been retained. The replication can be tested by submitting a test record using the Solr web interface on the master and running a *:* query on the slave instance.

It is important to note that the replication is uni-directional, meaning updates always propagate from the master to the slave, and never from the slave to the master. In other words, if we were to create an extra record on the slave instance, it would be removed during the next sleeping operation.

At this point our environment is ready for Sitecore and production, all that’s left is to drop in the schema.xml file, setup the load balancer, and point Sitecore to the virtual IP in the Solr configuration file \Website\App_Config\Include\Sitecore.ContentSearch.Solr.Indexes.config.

Load Balancer Setup

By this point you might be thinking, “well, querying makes sense, now that we have two servers with identical indexes, how do we update the index?”  Since the replication process is uni-directional, we need to only update the master index and it will update the slave during the sync operation. How do we only update the master index?  – load balancer!(or any proxy server with similar functionality)

Hopefully you are lucky enough to have a hardware server, otherwise, one of the software alternatives would do the job. On our load balancer we simply need to configure all read traffic (GET HTTP requests) to go to both servers, while update (POST HTTP requests) only to the master.

(Since there is a large variety of possible proxies, I will leave the exact setup steps to setup a load balancer outside of the scope of this article).

Caveats and Tips on Deploying Solr to Production

As always there a few things to consider with this setup.

First and foremost – although this replication setup is fault tolerant, if the master instance in the replication goes down, Sitecore will not be able to successfully process index updates.

Secondly, system administrators bus monitor both servers closely monitored for performance and resource use and adjust configuration and hardware accordingly. Remember that Solr hogs RAM by loading index files into memory, and to add to that it can get very CPU intensive. If there is shortage of RAM, or a large number of documents indexed, Hard drive speed will play a vital role in the search server performance, thus, if it is a mission-critical server, and it is vital for it to start asap after a restart spend some extra cash on the SSD drives and your users will thank you!

Lastly, Solr being a really good team player, so it has provided a URL that can be used by load balancers to check whether a Solr instance is up and running: http://{SERVER IP}:8983/solr/sitecore/admin/ping When the load balancer sends a GET request to this URL, Solr will reply with an error code in the HTTP header, if the core is experiencing problems, letting the load balancer know to redirect traffic to another instance….and if that wasn’t enough, Solr also provides a way to manually disable a Solr instance. If the <str name=”healthcheckFile”>server-enabled.txt</str> is enabled, Solr would check first for the existence of the file specified in the solrconfig.xml and if the file is not present, it will not even check on the heath of the server and immediately return an error response back to the load balancer. This allows system administrators to safely take Solr instances offline for any maintenance tasks, which gets Solr a big star in my book.

What About Solr Security?

Solr doesn’t need any. This is not to say that Solr comes secured out of the box, the search engine team just rather conveniently decided to not include any security settings at all. This is why it is advised to keep Solr servers behind a proxy or a firewall (remember Solr has an extensive REST-like API, which, if left exposed, might lead to some unexpected results being returned on your website at best).

That’s it, the big scary Solr monster has been conquered! Please feel free to comment below with any questions or suggestions and always follow the best practices!

UPDATE: Solr 4.8 Schema.xml Patch

It appears that the schema format has changed in version 4.8 of Solr. @stephenpope has posted a good article describing the updates needed to the schema.xml generated by Sitecore.