FAST Search 2010 – Crawl\Indexing Baseline Performance Testing

In preparation for making your FAST Search for SharePoint environment live in your production environment, it is imperative that you have tested your crawl performance to get your baseline statistics. Once you have your baseline stats then you will be able to re-test your production environment at monthly\six monthly intervals to ensure your FAST Search environment is performing consistently and there is no service degradation.

In order to get your baseline results you need to know which perfmon counters you will need to monitor during a full crawl and what is the acceptable performance range. The FAST Servers have different roles so there will be an overlap of components during the testing. I have detailed below the main perfmon counters to monitor during a full crawl and what to look for in your results. The performance ranges that I have detailed in the results below were the outcome of working with a Microsoft FAST Search specialist so they are accurate:

Processor Utilization – (System\Processor Queue Length): This is the number of threads queued and waiting for time on the CPU. Results: Divide this by the number of CPU’s in the system; if less than 10 then the system is running well.

Useful Additional Counter – (Processor\%Processor Time\_Total): Useful to show how busy the CPU is during the crawl process.

Memory Utilization – (Memory\Pages Input/Sec): This shows the rate at which pages are read from disk to resolve hard page faults. To go into more depth, this is the number of times the system was forced to retrieve something from disk that should have been from RAM. Results: Occasional spikes are acceptable but this should generally be very low, if not zero.

Useful Additional Counter – (Process\Working Set\_Total): This will show how much memory is in the working set.

Disk Utilization – (PhysicalDisk\Current Disk Queue Length\driveletter): Highly valuable counter to monitor. This shows how many read or write requests are waiting to execute to the disk. Results: Single disks it should be idle at 2-3 or below with occasional spikes being fine. For RAID arrays, divide by the number of active spindles in the array and aim for 2-3 or below. If the queue lengths are high then you will need to look at the memory\pages input/sec counter and potentially add more RAM. Aim for zero for this counter. The disk utilization will only be apparent on the Indexing server as the processing servers will mainly be using CPU and memory to process documents.

Useful Additional Counter – (Physical Disk\Bytes/sec\_Total) shows the number of bytes per second being written to or read from disk.

Network Utilization – (Network Interface\Output Queue Length\nic name): This is the number of packets in queue waiting to be sent. Results: If there is a persistent average of more than two packets in queue then you should look to resolve the network for a bottleneck. Aim for zero packets in the queue during a crawl. (Network Interface\Packets Received Errors\nic name): This is packet errors that kept the TCP/IP stack from delivering packets to higher layers. Results: This value should be low.

Useful Additional Counter – (Network Interface\Bytes Total/sec\nic name): This monitors the number of bytes sent or received.

Once you have completed your full crawl and evaluated your results, you will have a good idea of how your FAST Search environment has coped with the volume of content you are crawling and will provide you with vital information to determine if your FAST Search topology is adequate or whether you need to add any more hardware to your servers. I would recommend Microsoft’s article on performance and capacity planning for preparing your FAST Search topology.

In addition, I can also recommend using the Performance and Analysis of Logs (PAL) tool from Codeplex. You can extract a threshold xml file which will have a number of pre-defined counters for FAST Search, which you can then use with your perfmon Data Collector Set to gather and analyse your results. The beauty of this tool is that you can add your results file (.blg file) to the application and it will analyse the results against the recommended thresholds for each counter. Very useful and an excellent way to assist in presenting your results!