Full Crawl keeps running rather than an Incremental Crawl

I discovered a strange issue with my SharePoint 2010 Farm that uses FAST Search 2010 as its search infrastructure. The issue related to my content source continually running a full crawl rather than an incremental even though the crawl schedules were set correctly. There was nothing obvious to say there was a problem as the full crawl completed but took longer than expected and all other content sources looked to run correctly using an incremental.

I started out troubleshooting the environment and going through the main health checks detailed below:

  • SQL: Check to see if there are any locks on the crawler and content dbs
  • SharePoint Crawl Servers: Check to see if the CPU or memory is maxing out of running consistently high.
  • FAST Search: Check to see if the index is healthy by running “Indexerinfo status -a” on the FAST Servers” and see if the number documents that are active match the total. Ensure to check both the primary and backup indexer in your results from running the command in powershell for fast search.
  • FAST Search: Check through the logs in %FAST SEARCH%\var\log and in particular the ‘configserver.log’ and ‘all.log’ on the FAST admin server.

After checking through the main troubleshooting points, you can run a perfmon trace to get more information, but instead I ran through my install notes and also checked through the details of Kristopher Loranger’s blog ‘FAST Search for SharePoint 2010 Crawler Troubleshooting’ to  uncover the issue.

From the ‘configserver.log’ file I found numerous entries relating to the error below.

“the ping call resulted in the following exception: socket.error: [Errno 10060] The operation timed out.”

This namely pointed to a communication error between the fast search admin server and the indexing servers. From my install notes and the blog from Kristopher Loranger, a communication error would usually relate to a TCP offloading issue on the FAST Search Servers. TCP Offloading should be disabled on your FAST Search and SharePoint servers as it doesn’t work effectively with IPSEC.

I checked the TCP offloading settings using the command “netsh int ip show global, and the settings were correct. Chimney Offload State = Disabled.

Image1

Then I checked the offloading settings on the network card layer… This is where the problem lay. All the entries with ‘Offload’ were enabled. This was changed when the VM was moved to a different host so definitely one to watch for when migrating servers.

Image2

So to resolve the issue, change all entries with ‘Offload’ in the property name and reboot all the fast servers. Ensure you disable your crawls before following through with this task.

The Microsoft Blog on TCP Offloading is really useful going through the detailed steps in TCP offloading for your FAST Search environment.

 

 

 

 

 

Advertisements

SharePoint – ‘Modified By’ does not get updated for an item

An issue was reported to me relating to a site collection whose Modified By information for their item in their lists would not update leaving the original name of the person who uploaded the file as the “Modified By” person. This was a consistent issue across all lists within only one particular Site Collection. All other site collections were working as expected.

After searching for the solution, it was discovered that the issue I was having was the same as discovered by Marc D Anderson (Link) and Victor Butuza ((Link).

Marc D Anderson – Item ‘Modified By’ Value Doesn’t change: Fixing a Damaged Column Schema

Victor Batuza – Modified By Column does not get updated

To diagnose the issue from the articles above to confirm I was having the same problem, I used SharePoint Manager 2010 to inspect the schema.xml of the site collection and a particular site collection. Please air on the side of caution on using SharePoint Manager as you don’t want to make changes to properties using this tool without knowing exactly what you are doing. Please use it on a copy\backup of the web application you are looking to diagnose.

What the schema should look like:

<Field ID=”{d31655d1-1d5b-4511-95a1-7a09e9b75bf2}” ColName=”tp_Editor” RowOrdinal=”0” ReadOnly=”TRUE” Type=”User” List=”UserInfo” Name=”Editor” DisplayName=”Modified By” SourceID=”http://schemas.microsoft.com/sharepoint/v3” StaticName=”Editor” FromBaseType=”TRUE” />

What my schema.xml actually looked like:

<Field ID=”{d31655d1-1d5b-4511-95a1-7a09e9b75bf2}” Name=”Editor” SourceID=”http://schemas.microsoft.com/sharepoint/v3” StaticName=”Editor” Group=”_Hidden” ColName=”tp_Editor” RowOrdinal=”0” Type=”User” List=”UserInfo” DisplayName=”Modified By” ReadOnly=”TRUE” Version=”1” />

As you can see from the above FromBaseType=”TRUE” was missing from my schema.xml. Next, I used the excellent powershell scripts from Marc D Anderson blog to diagnose how many lists were affected.

  1. Diagnose how many Lists are affected:
$siteURL = "http://siteurl"
$site = Get-SPSite($siteURL)
$errors = 0
$thisWebNum = 0
foreach($web in $site.AllWebs) {
$thisWebNum = $thisWebNum + 1
write-host $thisWebNum " Web " $web.Url  " Created on "  $web.Created
$listCounter = $web.Lists.Count
for($i=0;$i -lt $listCounter;$i++) {
$list = $web.Lists[$i]
$thisListNum = $i + 1
write-host "(" $thisListNum "/" $listCounter ") [" $list.Title "] Created on "  $list.Created
$f = $list.Fields.GetFieldByInternalName("Editor")
if ($f.SchemaXML -NotMatch 'FromBaseType="TRUE"')
{
$errors = $errors + 1
write-host "  Issue in schema " $f.schemaxml
}
}
$web.Dispose();
}
$site.Dispose();
write-host "TOTAL ISSUES: " $errors
  1. Fix one Document Library to test the fix

Once you have identified the number of document libraries that are affected affect, you will now need to update the document libraries schema.xml and add the FromBaseType=”True” to each one. To do this manually is an arduous task, so powershell is your friend here. Try it for one document library first and test it. This is explained is much more detail in Marc D Andersons blog so I won’t go over the details. But you will need to update the webappurl, sitename and list name before you run it.

IMPORTANT: Before you do any work on the list to fix it, it is important to have a backup of your content databases and run the fix on a copy of your web application so you are happy with the process. Then schedule downtime to put the fix in place on production. This is standard operation procedure, but still has to be mentioned.

$s = get-spsite http://webappurl
$w = $s.OpenWeb("SiteName")
$l = $w.Lists["ListName"]
$f = $l.Fields.GetFieldByInternalName("Editor")
write-host "BEFORE field at " $w.Url  " List "  $l.Title  "  is " $f.schemaxml
#add at the end of the schema the needed string and update the field and list
$f.SchemaXML = $f.SchemaXML -replace '/>',' FromBaseType="TRUE" />'
$f.Update()
$l.Update()
write-host "FIXED field at " $w.Url  " List "  $l.Title  "  is " $f.schemaxml
$w.Dispose();
$s.Dispose();
  1. Update all Lists with the Fix

Now that you have tested the fix, you now need to update the fix on all your lists. Run the powershell below on your site collection:

$siteURL = "http://siteurl"
$site = Get-SPSite($siteURL)
$errors = 0
$thisWebNum = 0
foreach($web in $site.AllWebs) {
$thisWebNum = $thisWebNum + 1
$listCounter = $web.Lists.Count
for($i=0;$i -lt $listCounter;$i++) {
$list = $web.Lists[$i]
$thisListNum = $i + 1
$f = $list.Fields.GetFieldByInternalName("Editor")
if ($f.SchemaXML -NotMatch 'FromBaseType="TRUE"')
{
$errors = $errors + 1
# fix the schema and update the field and list
$f.SchemaXML = $f.SchemaXML -replace '/>',' FromBaseType="TRUE" />'
$f.Update()
$list.Update()
write-host "FIXED field at " $w.Url  " List "  $l.Title  "  is " $f.schemaxml
}
if ($errors -gt 0)
{
write-host $thisWebNum " Web " $web.Url  " Created on "  $web.Created  " had " $errors " errors"
}
$errors = 0;
$web.Dispose();
}
$site.Dispose();
}

You can re-run step 2 to check to see if the fix has updated the schema.xml for all the lists.

  1. Test to Ensure New Document Libraries don’t have the same issue

A very good test is to check that when you create a new document library in the same site collection, that the original issue does not re-appear for the new list. If it does, then it is likely to be an issue with the site collection’s schema.xml which will need to be updated separately.

To update the schema.xml for the Site Collection, I found it easier to use SharePoint Manager 2010 as you can navigate to your troublesome site collection, go to the ‘Fields’ and find ‘Modified By’ which will be the second of the two entries and then add FromBaseType=”TRUE” to the schema.xml.

SharePoint Manager 2010 – https://social.technet.microsoft.com/wiki/contents/articles/8553.sharepoint-2010-manager.aspx

Download Link – http://spm.codeplex.com/releases/view/51438

I hope this helps people who had the same issue.

Storage Metrics in SP2010 Service Pack 1

Quite a nice feature in SharePoint 2010 that has gone relatively unnoticed in SharePoint 2010 service pack 1, is the Storage Metrics feature. It is available under Site Collection Administration and it enables you to get useful information regarding how content is being used within your site collection. The report will give you a list breakdown of the content and the physical size along with the percentage of the site quota that has been used. This is particularly useful for Site Collection Administrators and IT Pro’s who will want to monitor the size of the content for particular sites. You can take this a step further with using third party tools such as Axceler’s Control Point application which will give you trend analysis on your content.

You are able to go granular with the results to the individual file level, so you can troubleshoot a growing content database by finding out what sort of file types are taking up space in a particular document library or list.

Please note that this feature is only available in Service Pack 1 of SharePoint 2010.

Copenhagen for the European SharePoint Conference 2013

This February it was the turn of the European SharePoint community to get a piece of the SharePoint pie. Four days of keynote speeches, workshops, seminars and a bustling array of third party vendors offering knowledgeable expertise, products and eye-catching prizes. The event was hosted at the Bella Center in Copenhagen which was a great venue and well organised but with a dreary wi-fi connection. So yes, I was one of the special people constantly manoeuvring to try and find a better signal (just like one of many lemmings each day!). I did raise a cheeky grin to see the speakers struggle with stuttering demos down to the wi-fi connection too. Steve Jobs would be smiling down with joy!

After the four days, I have gathered some useful information and discussion points that I will be blogging in the next few days. They will mainly be focused around the IT Pro arena so I will try and delve into as much detail as I can for you that will hopefully help you in your SharePoint journey.

On a separate note, the event was well organised and there was good banter around the booths from the energetic 3rd party vendors and creative stands (Well done to Axceler for the green smurfs..). The AvePoint and Axceler parties were very enjoyable and provided a chance to chat and see the experts let loose. There was even a rumour of projectile vomiting from a delegate on the way to Wednesday’s keynote speech, that would have been a long day for that poor chap!

Anyhow, keep an eye out for my next few blogs relating to the SharePoint 2013 Conference.

FAST Search 2010 – Crawl\Indexing Baseline Performance Testing

In preparation for making your FAST Search for SharePoint environment live in your production environment, it is imperative that you have tested your crawl performance to get your baseline statistics. Once you have your baseline stats then you will be able to re-test your production environment at monthly\six monthly intervals to ensure your FAST Search environment is performing consistently and there is no service degradation.

In order to get your baseline results you need to know which perfmon counters you will need to monitor during a full crawl and what is the acceptable performance range. The FAST Servers have different roles so there will be an overlap of components during the testing. I have detailed below the main perfmon counters to monitor during a full crawl and what to look for in your results. The performance ranges that I have detailed in the results below were the outcome of working with a Microsoft FAST Search specialist so they are accurate:

Processor Utilization – (System\Processor Queue Length): This is the number of threads queued and waiting for time on the CPU. Results: Divide this by the number of CPU’s in the system; if less than 10 then the system is running well.

Useful Additional Counter – (Processor\%Processor Time\_Total): Useful to show how busy the CPU is during the crawl process.

Memory Utilization – (Memory\Pages Input/Sec): This shows the rate at which pages are read from disk to resolve hard page faults. To go into more depth, this is the number of times the system was forced to retrieve something from disk that should have been from RAM. Results: Occasional spikes are acceptable but this should generally be very low, if not zero.

Useful Additional Counter – (Process\Working Set\_Total): This will show how much memory is in the working set.

Disk Utilization – (PhysicalDisk\Current Disk Queue Length\driveletter): Highly valuable counter to monitor. This shows how many read or write requests are waiting to execute to the disk. Results: Single disks it should be idle at 2-3 or below with occasional spikes being fine. For RAID arrays, divide by the number of active spindles in the array and aim for 2-3 or below. If the queue lengths are high then you will need to look at the memory\pages input/sec counter and potentially add more RAM. Aim for zero for this counter. The disk utilization will only be apparent on the Indexing server as the processing servers will mainly be using CPU and memory to process documents.

Useful Additional Counter – (Physical Disk\Bytes/sec\_Total) shows the number of bytes per second being written to or read from disk.

Network Utilization – (Network Interface\Output Queue Length\nic name): This is the number of packets in queue waiting to be sent. Results: If there is a persistent average of more than two packets in queue then you should look to resolve the network for a bottleneck. Aim for zero packets in the queue during a crawl. (Network Interface\Packets Received Errors\nic name): This is packet errors that kept the TCP/IP stack from delivering packets to higher layers. Results: This value should be low.

Useful Additional Counter – (Network Interface\Bytes Total/sec\nic name): This monitors the number of bytes sent or received.

Once you have completed your full crawl and evaluated your results, you will have a good idea of how your FAST Search environment has coped with the volume of content you are crawling and will provide you with vital information to determine if your FAST Search topology is adequate or whether you need to add any more hardware to your servers. I would recommend Microsoft’s article on performance and capacity planning for preparing your FAST Search topology.

In addition, I can also recommend using the Performance and Analysis of Logs (PAL) tool from Codeplex. You can extract a threshold xml file which will have a number of pre-defined counters for FAST Search, which you can then use with your perfmon Data Collector Set to gather and analyse your results. The beauty of this tool is that you can add your results file (.blg file) to the application and it will analyse the results against the recommended thresholds for each counter. Very useful and an excellent way to assist in presenting your results!