Wednesday, July 8, 2015

Basic Tips to Prevent Solr Downtime

If you've followed my series on installing Solr for Sitecore then you should have a shiny, new Solr instance somewhere in your environment happily indexing Sitecore data and returning results to queries. Hopefully, that never changes, but we all know that hiccups can happen. This post suggests a few things you can do to mitigate or prevent down-time.

Logging

If you find yourself troubleshooting, you'll be very glad to have Solr-specific logs to refer to. Given how easy this is to configure, you owe it to yourself to do so. Assuming you have the downloaded .zip from Solr:
  1. Copy the Jar files from  solr/example/lib/ext to Tomcat's lib/ folder.
  2. Copy properties file from solr/example/resources to Tomcat's lib/ folder.
All done! You will find your new Solr logs in the install path of Tomcat in the logs/ folder.

RAM

When dealing with Solr there are two kinds of RAM to consider. One is the amount of RAM dedicated to the Java heap and the second is OS disk cache. While I can't give specific guidance on how much RAM you should devote and where, I will provide some general advice and guidance.

Java Heap

To set the Java heap size is pretty straightforward matter once you understand the implementation details of Tomcat for your machine. Mainly, this means what version of Tomcat are you running and which OS do you use. I'll be covering Tomcat 8 as a Windows service. If you differ from me in one or more regards, don't despair. Most of what I say still applies to you, you'll probably need to look a little to find the equivalent spots to make your setting changes.

First, let's review the four different memory management parameters you may control
  • Xms - The minimum size of your heap
  • Xmx - The maximum heap size
  • XX:PermSize - Specifies the initial size allocated to the JVM at startup
  • XX:MaxPermSize - If necessary, up this maximum will be allocated to the JVM during startup

Most likely, you won't need to worry about XX:PermSize and XX:MaxPermSize unless you see errors like Java.lang.OutOfMemoryError: PermGen Space. Much more likely, you will want to control the bounds on your already-running heap through Xms and Xmx. If you are running Tomcat as a Windows service then this is as simple as filling in a text box. For example:


The above screenshot shows the equivalent of setting -Xms=256m and -Xmx=512m. Additionally, I elected to specify the XX:PermSize as 128MB.

As a final note on heap size, be aware that for heap sizes greater than 2GB, garbage collection can cause performance problems. Symptoms are occasional pauses in program execution during a full GC. This can be mitigated through GC tuning of your JVM or electing to use a commercial JVM.

Disk Cache

For disk cache, you would ideally have enough RAM to hold the entire index in memory. Whatever memory remains unused once the OS, running programs, and the Java heap have been satisfied is fair game for disk cache. Thus, if 12GB of RAM is unused, you could potentially fit 12GB of index data into memory before the OS is forced to start paging. In practice, you must use trial and error to find the right memory fit for your data and usage patterns.

Secondary Cores

Given that you have elected to use Solr, you probably treat search as a first-class citizen in your environment. If you aren't using secondary cores to provide data continuity during an index rebuild, you're simply doing it wrong. It helps that the process for configuring secondary cores is easy to follow.

Note: every time a rebuild occurs, the name values in the core.properties files for the two related cores will swap. This is normal behavior, of course, but can be horribly confusing if you aren't aware of it. I.e. don't just assume that the name of the core you are viewing matches the core's folder name in your Solr home directory!

Replication

This topic is actually quite broad and probably deserves a blog post or several all of its own. Nevertheless, we can at least imagine the base case of wishing to provision a second Solr instance that is slaved to a master instance. Fail-over will not be automatic although you could script it.
  1. Modify the core.properties file in your cores to set whether the core is a master or a slave.
    • On Master
      • enable.master=true
      • enable.slave=false
    • On Slave
      • enable.master=false
      • enable.slave=true
  2. Modify conf/solrconfig.xml file in each core to include a request handler for replication. Below is a snippet of XML you can use. Simply replace the "remote_host" and "core_name" in the snippet's XML with your environment's values. Note: the way I have constructed this snippet means you can apply it "as is" to any core on your master OR your slave instance. The trick I used was to associate the state of the "enable" property for the master and slave elements with the value of the enable.master and enable.slave properties from the core's core.properties file which you should have set in step 1. This makes your bookkeeping duties a little less painful, especially if you ever find yourself swapping the master and slave around.
What to do in the event your master goes down? Edit the master/slave properties in the core.properties file and change the ServiceBaseAddress used by Solr in the Sitecore.ContentSearch.Solr.DefaultIndexConfiguration.config file. You should also (as soon as time allows) edit the Replication handler XML appropriately: either change the URL or comment it out entirely.

Further Reading

Monday, July 6, 2015

PowerShell Sitecore Install Script

Update (4-17-2016): Here I discuss enhancements I've made to the script since last year as well as a video tour of the config file used by the script.



Download from GitHub.

I've been searching for a solution to automating Sitecore installations in any environment higher than my personal development VM--for that we already have SIM. I can be stubborn and exacting, sometimes to a fault, and while a manual install affords me complete control over an environment, it also is horribly time-consuming. Also, if I'm being honest with myself (I'm sure this plainly obvious to you) this process is mistake-prone.

Search for a Solution

The following are the three most prominent examples of existing solutions I looked at, but I looked at many more.

Sitecore Instance Manager

I've been using SIM for a while now to manage my Sitecore instances on my development box. It's a wonderful solution, but it's not suitable for a production environment without a lot of post-install intervention. I also looked at the console app for SIM. Alas, while it seems to extend SIM to the command-line, it does not allow for greater flexibility in how SIM installs a Sitecore instance.

Sitecore's Installer

Jeremy Davis had the very clever idea of deconstructing Sitecore's .exe installer to get at the underlying .msi file. He successfully identified all of the command-line switches the .msi accepts. I very nearly settled on this approach. After all, one would expect Sitecore's own installer to follow Sitecore's installation guideline's recommendations. It does a better job than SIM, but you are also rather constrained in some of your options and that was a deal breaker for me.

PowerShell Script

All-star Alex Shyba wrote a PowerShell script to automate his installs. His use case is the same as mine for SIM, however: he built the script to install development instances. Like the previous solutions discussed, concerns such as file system permissions, user mappings in SQL, using a domain account for the application pool identity, and CD-hardening are left as post-install exercises for a human.

Inspiration

Alex's script gave me the push I needed to write my own. My goal is to completely automate a production-ready Sitecore CM or CD server install. Once you run my script the only thing left to do is install your desired modules. Actually, let's take a minute to unpack that, because buried inside that sentence is a subtle point on my deployment philosophy, and it impacts the way I designed my Sitecore installer. I believe that any Sitecore change that can be managed through a .update package (or .zip module) should. For me, this includes managing changes like SwitchMasterToWeb, scalability, and web.config amongst many others. Thus, my over-arching design philosophy for an automated install is to do everything I would normally in SQL, IIS, the file system, and (yes...) the .config files but only enough to create a working instance and no more. Once the installer is done, the instance should be 100% ready for management via .update packages. That is my goal.

The Solution

I decided to make my script available on GitHub for a couple reasons.
  • I suspect and hope other will want to make use of it
  • Community feedback will help me improve it

Major Features

  • Install Sitecore with or without the databases.
  • Script sanity checks SQL and input validation prior to making any changes
  • Write output to the screen and to a log file.
  • Fine-grained control of the application pool identity (built-in or domain account)
  • Assign recommended file system permissions on web server.
  • Add application pool identity to recommended local groups on web server.
  • Create user mappings for login in SQL.
  • Install database files on any valid path or UNC
  • SQL Login used during install doesn't have to be the same account executing the script.
  • May specifiy a host name and port used for MongoDB
  • May supply a Solr base address
  • Choose to use SQL as a session state server
  • Many CD-hardening options

One limitation of the script today is I do not support choosing MongoDB as a session state server. My suspicion is that this would be a very easy change to make, and I will be including it soon. The script is strictly limited to automating the Sitecore install itself, not MongoDB or Solr. While it's not necessary, it would be a good idea to provisions those applications first if you plan to use them. Speaking of Solr, if you do plan to use it, then be sure to check out my other PowerShell script to change the search provider from Lucene to Solr.

Finally, I built this script to install Sitecore 8.0. I've briefly tested it with Sitecore 7.5 and it mostly works, but breaks on some assumptions about the existence of .config files like SwitchMasterToWeb.config.example and Sitecore.ContentSearch.Solr.DefaultIndexConfiguration.config.example. Even earlier versions of Sitecore would need some more adjustments (example: deal with with differences in databases.) Depending upon the level of interest expressed I will consider making the script compatible with prior Sitecore releases.