FAST Search – Repair or Rebuild your Corrupt Index from FIXML?

The FAST Search environment is a robust search solution for your SharePoint environment and once it is configured correctly and optimized then it will purr in the background surfacing rich search results for your users. However, on the rare occasion that your index gets corrupted, you as a SharePoint administrator will need to be aware of the tools and methods you can use to get it working again. If anything else, this should be part of your recovery procedure for your SharePoint environment.

So what options do you have when your index gets corrupted? Well, firstly you could completely delete the Index from SharePoint in the Search administration and via powershell on your FAST Search admin server. Or if you’re Index will take too long to re-build and will not meet your SLA then you can recover your Index using the FIXML. When FAST Search for SharePoint is indexing items it not only stores the physical index itself in the location ‘%drive%\FASTSeach\data\data_index’ but it also stores each indexed item in the FIXML in the location ‘%drive%\FASTSearch\data\data_fixml’. The FIXML contains all the information which is to become that item in the index.

Now that we know we can use the FIXML, there are two options available to you that are detailed below.

Option A – Repair a corrupt Index from FIXML

This process can be performed from any FS4SP server and on any column or row. The index reset does not rebuild the column from scratch, the indexer validates each item within the FS4SP column against the original FIXML. Any item not in sync or corrupt will be updated in the column / index.

  1. Ensure all crawls are stopped in Search Administration and the FS4SP column is idle. Use the powershell command below to show the status of FAST Search.

</p><p>Indexerinfo --row=0 --column=0 status
</p><p>

  1. Stop the web analyser and relevancy admin processes.

</p><p>Waadin abort processing
</p><p>Spreladmin abortprocessing
</p><p>

  1. Issue an Index Reset. You can re-run the second command to monitor the status or check the indexer.txt file in the logs directory (FAST Search\var\log\indexer).

</p><p>Indexeradmin --row=0 --column=0 resetindex
</p><p>Indexeradmin --row=0 --column=0 status
</p><p>

  1. Once the repair is complete, you can then resume the web analyser and relevancy admin.

</p><p>Waadin enqueueview
</p><p>Spreladmin enqueue
</p><p>

Your FAST Search Index will now be repaired and will be operational.

Option B – Rebuild an Index from XML

Ok, so you attempted Option A and this didn’t resolve your issue and your index is still corrupted. The next step is to rebuild the index from your fixml. Rebuilding the index requires a lot more disk space than option A as temporary files are created and released (within FAST Search directory) which means you are likely to consume twice as much disk space. If the disk space gets to 2GB of free space then the rebuild will fail so you will need to manage your disk space. Follow the steps below to complete this task.

  1. Ensure all crawls are stopped in Search Administration and the FS4SP column is idle. Use the powershell command below to show the status of FAST Search and keep a note of the entries “document size” and the “indexed=0”.

</p><p>Indexerinfo --row=0 --column=0 status
</p><p>

  1. Stop the web analyser and relevancy admin

</p><p>Waadin abort processing
</p><p>Spreladmin abortprocessing
</p><p>

  1. Rebuild the primary index column. Run the command from the primary indexer server to stop the processes.

</p><p>Nctrl stop
</p><p>

  1. Delete the folder ‘data_index’ within the directory ‘FASTSearch\data’ and start the services again using the nctrl command. When you start the processes the ‘data_index’ folder will be re-created and will be populated with a rebuild of the index.

</p><p>Nctrl start
</p><p>Indexerinfo --row=0 --column=0 status
</p><p>

Notice the “document size” entry, and check that the “indexed=0” is displayed as this comes from the fixml and means the index is empty. Keep re-running the status query until indexed items are at their original value. This is when it is complete.

  1. Once the rebuild is complete, you can then resume the web analyser and relevancy admin.

</p><p>Waadin enqueueview
</p><p>Spreladmin enqueue
</p><p>

Option B is now in place and complete. This will bring your FAST Search Index back online and ready for use.

SP2010 April CU stops incoming emails to a list or calendar

If you have followed my earlier post “Setting Up Incoming Mail to a List or Calendar” then you will most likely have a working incoming email configuration for your SharePoint 2010 environment. However, the laws that be at Microsoft have inadvertently released a bug in the April (2012) CU that will affect the incoming mail process.

The symptoms of this bug are that your email request will have been successfully sent through the Exchange process and arrived at your drop folder on your SharePoint web front end servers in the form of an ‘.EML’ file ready to be picked up by the timer job called ‘Microsoft SharePoint Foundation Incoming mail’. However, since the April CU was applied the .EML file will remain in the drop folder and is not picked up by the timer job. This is a frustrating bug and if you were to analyse the ULS logs a little deeper you are likely to get entries similar to the screenshot below and in particular the EventID of 6871.

The error is down to the Incoming Mail timer job somehow now being dependent on your site collection having a quota set. If you leave the site collection with the default quota of ‘0’ then the emails will fail to be picked up by SharePoint. However, this is not always consistent as I have a working SharePoint farm that is on the same farm level version with no quotas set for the site collection and incoming mail is working perfectly (very weird). So if you are unlucky to encounter this issue, then help is at hand as there are two options that you can run with which are detailed below:

Option 1: Configure your site collection with a site quota to your required specification. Once you make the changes the incoming emails that are sitting in your Drop folder will be picked up the next time the timer job has run.

Go to Central Administration > Application Management > Configure quotas and locks > Select the relevant site collection from the drop down menu > on the Site Quota Information section, set the limit to your desired number (e.g 10000 MB). Note: This be the new storage limit of your site collection so make sure have catered for future growth of the site collection.

Option 2: Download and apply the Microsoft hotfix KB2598348 which includes the fix for this bug. One thing to take into consideration for this resolution is that it will likely to bring your farm version level to July 2012 so you will need to test your new farm version in your development or staging environment to ensure it is stable for your site collections.

Web Analyzer Failing for FAST Search

The FAST Search implementation for SharePoint 2010 is a convoluted process and mistakes in the deployment.xml can cause performance and functionality headaches further down the line during your testing and implementation process. As part of the initial installation phase of FAST search you are required to pre-configure the deployment.xml file. The deployment.xml file is used to determine what components are to be utilized and for which server in your FAST Search topology. As part of all FAST Search topologies, you will have an admin server which will host the web analyzer component. This component is used to analyse search clickthrough logs and hyperlink structures which both contribute to better ranked search results and relevancy.

An issue that we discovered in testing was that the web analyzer was not functioning correctly as we had numerous error messages in the webanalyzer.log file stating that the web analyzer could not connect to the config server. Typical error messages we discovered are detailed below:

“Module (Web Analyzer) at “fast1.domain:13300″ is not responding”

“Couldn’t connect to the Config Server”    

So what caused this error? The error relates to the deployment.xml, the file is case sensitive and if you name your admin server with a case mismatch (e.g “Server1.domain.com”) then this will cause the web analyzer to fail as it will not be able to identify the config server due to the upper case ‘S’ even if you use the FQDN. So in order to avoid this issue then I recommend using a lower case naming convention for all server names when configuring your deployment.xml.

If you encounter this issue in your existing environment then there is a process that can be followed to update your FAST Search configuration. The steps are detailed below:

1.Stop the services on all FAST Search Servers using “nctrl stop” in FAST powershell.

2. Edit the deployment.xml on the FAST Search Admin server and change the FQDN for the admin component to be all lower case.

3. On the admin server, edit ‘etc\waconfig.xml’ and ‘etc\node.xml’ and update the hostname to be all lowercase.

4. In FAST Powershell, run “Set-FASTSearchConfiguration” on the admin server and start the services by running “nctrl start”.

5. Now repeat steps 3 and 4 for all your other FAST servers in your topology.

Your FAST Search environment will now have a working web analyzer. For further reading on the deployment.xml then please refer to the msdn article.

FAST Search 2010 (FS4SP) – Configuring the Document Converter for Legacy Office Documents

As with most organisations who have implemented a new SharePoint 2010 farm, there is likely to be a need to import/migrate office documents from shared file servers or other types of document management systems into SharePoint. This will mean that there might well be documents that were created using Office 97, XP or 2000 onwards that will be migrated into SharePoint 2010. On a recent exercise while using FAST Search for SharePoint 2010 as the search component, we found that the content of legacy office documents were failing to be indexed and thus not appearing in the users search queries. After further investigation and delving through logs, it was discovered that the Microsoft iFilter 2.0 was not able to convert the legacy documents to HTML to enable them to be indexed. However, this only related to the contents of the documents as the metadata were being crawled and made available in search results.

So how do we spot if this is occurring in your FAST environment? Well, you will need to go into FAST Search administration and check for the warnings in the crawl log for your content source. You will need to be looking for the warning message stated below:

“The FAST Search backend reported warnings when processing the item. ( Document conversion failed: LoadIFilter() failed: Bad extension for file (0x800401e6) )”

Unfortunately, there is not a fix for the Microsoft iFilter 2.0 but a workaround is possible with a configuration change in the FAST environment which will ensure that the legacy file types such as “.doc”, “.ppt” and “.xls” are converted using the SearchExportConverter and not the iFilterConverter. The SearchExportConverter still supports legacy formats and handles their conversion better than the iFilterConverter. The iFilterConverter will still be used to convert new versions of office documents such as “.docx” and “.xlsx” etc.

In order to make this change, I would strongly recommend implementing the process on your test environment to ensure you are happy that the legacy documents are being indexed and are available in search queries for your users. Please follow the steps below on each of your FAST Search servers to implement the SearchExportConverter into your FAST environment:

1) Go to your FAST Search root directory and set the Search Export Converter to “active=Yes” in the file \etc\config_data\documentprocessor\optionalprocessing.xml.

2) Open the converter_rules.xml file in the location \etc\formatdetector\ and comment out the legacy document types.

3) Open the pipelineconfig.xml and swap the order of the “iFilterConverter” and the “SearchExportConverter” in both pipeline sections.

4) In FAST Powershell, you now need to run the following cmdlet “ntrl reloadcfg” which reloads the configuration.

5) Again in FAST Powershell, you will need to re-start all your procservers by using the cmdlet “ntrl restart procserver_X”. X applies to the number of the procserver which you will need to restart as there will be more than one., “nctrl status” will list all the active procservers on your FAST Server.

I would also recommend a reboot of the FAST Search Servers and then you are free to run search queries against the contents of legacy office documents after running a full crawl.

Important Note: If you apply this fix before Service Pack 1, then you will need to re-apply step 3 after deploying Service Pack 1 as the update deploys a new version of the pipelineconfig.xml which includes a new entry called ‘CustomExtractor’ in the Elevated Linguistics pipeline section of the file.

FAST Search 2010 (FS4SP) – Disabling TCP/IP Offloading Engine (TOE)

Installing and configuring FAST Search for SharePoint 2010 is a complex process which is mainly due to FAST Search being bought by Microsoft and then bolted onto the SharePoint 2010 product. If you are deploying your FAST Search Server to virtualised servers, then you may well encounter an issue where FAST servers fail to communicate with each other correctly with the default settings of the network adapter. In addition to the communication issues, the feeding of content to a multi-node FAST environment will not be indexing properly and the full crawls will take drastically longer than they should to complete. This will be due to a known issue where IPSEC does not work correctly with TCP/IP offloading.

The types of errors you will be getting in your FAST Search logs are:

WARNING systemmsg Module (Search Dispatcher) at fast.yourdomain.com:15102 is not responding
VERBOSE systemmsg The ping call resulted in the following exception: socket.error: [Errno 10061] Connection refused.
WARNING systemmsg Module (NodeControl) at fast.yourdomain.com:16015 is not responding
VERBOSE systemmsg The ping call resulted in the following exception: Timeout

TCP/IP Offloading (or TOE for short) is a technology system used for the purpose of optimizing the throughput of Ethernet systems. The communication speed in Ethernet systems has increased faster than computer processor speed. This means that the speed of the data from the network is too much for the processor to handle and thus will cause a bottleneck. TOE would solve this problem by removing offloading from the microprocessor and I/O subsystem. However, TOE does not work with IPSEC which is used by FAST Search for SharePoint so TOE is required to be disabled on your virtual machine. In doing this, you will see a performance increase in FAST Search (and your SharePoint servers) and the communication errors during crawls will be resolved.

To disable TCP/IP offloading on your virtual machine, follow the steps below:

1) Open a Cmd prompt and run the following:

netsh int ip set global taskoffload=disabled

netsh int ip show global (This will show that it is disabled Task Offload’)

Netsh int tcp set global chimney=disabled

Netsh int tcp show global (This will show the ‘Chimney Offload State’ as disabled)

2) Now go to the ‘Advanced’ tab of the Network Adapter device properties and change all the properties with ‘Offload’ in the property name to ‘Disabled’ and reboot the machine. Repeat this process for all your FAST Search servers.

This process is now complete and you will no longer have the communication errors during crawls and the warnings will no long appear in your logs. If you wish to read more about this TCP/IP offloading issue, then I would recommend Microsoft’s KB article.