Troubleshooting a Memory Leak on your SharePoint Web Server

You may encounter an issue where your SharePoint web application continually falls over and you need to ascertain why! One of the questions you may need to ask is whether you have a memory leak on your web front end servers? So how do we find this out? In this instance, performance monitor is your friend and you can set up a Data Collector Set that will include the relevant counters specific to discovering a memory leak.

I have detailed below the XML code that you can use to create your Data Collector Set:

</p><p>&lt;?xml version="1.0" encoding="UTF-8"?&gt;
</p><p>&lt;?Copyright (c) Microsoft Corporation. All rights reserved.?&gt;
</p><p>&lt;DataCollectorSet&gt;
</p><p>&lt;Name&gt;SharePoint_Server_2010_Memory&lt;/Name&gt;
</p><p>&lt;DisplayName&gt;@%systemroot%\system32\wdc.dll,#10026&lt;/DisplayName&gt;
</p><p>&lt;Description&gt;@%systemroot%\system32\wdc.dll,#10027&lt;/Description&gt;
</p><p>&lt;Keyword&gt;Memory&lt;/Keyword&gt;
</p><p>&lt;Keyword&gt;Disk&lt;/Keyword&gt;
</p><p>&lt;Keyword&gt;Performance&lt;/Keyword&gt;
</p><p>&lt;RootPath&gt;%systemdrive%\perflogs\System\Performance&lt;/RootPath&gt;
</p><p>&lt;SubdirectoryFormat&gt;3&lt;/SubdirectoryFormat&gt;
</p><p>&lt;SubdirectoryFormatPattern&gt;yyyyMMdd\-NNNNNN&lt;/SubdirectoryFormatPattern&gt;
</p><p>&lt;PerformanceCounterDataCollector&gt;
</p><p>    &lt;Name&gt;SharePoint_Server_2010_Memory&lt;/Name&gt;
</p><p>    &lt;SampleInterval&gt;15&lt;/SampleInterval&gt;
</p><p>    &lt;Counter&gt;\LogicalDisk(*)\% Idle Time&lt;/Counter&gt;
</p><p>    &lt;Counter&gt;\LogicalDisk(*)\Split I/O /sec&lt;/Counter&gt;
</p><p>    &lt;Counter&gt;\Memory\% Committed Bytes In Use&lt;/Counter&gt;
</p><p>    &lt;Counter&gt;\Memory\Available MBytes&lt;/Counter&gt;
</p><p>    &lt;Counter&gt;\Memory\Committed Bytes&lt;/Counter&gt;
</p><p>    &lt;Counter&gt;\Memory\Pages Input/sec&lt;/Counter&gt;
</p><p>    &lt;Counter&gt;\Memory\Pages/sec&lt;/Counter&gt;
</p><p>    &lt;Counter&gt;\Memory\Pool Nonpaged Bytes&lt;/Counter&gt;
</p><p>    &lt;Counter&gt;\Memory\Pool Paged Bytes&lt;/Counter&gt;
</p><p>    &lt;Counter&gt;\Process(_Total)\Private Bytes&lt;/Counter&gt;
</p><p>&lt;/PerformanceCounterDataCollector&gt;
</p><p>&lt;DataManager&gt;
</p><p>    &lt;Enabled&gt;-1&lt;/Enabled&gt;
</p><p>    &lt;CheckBeforeRunning&gt;-1&lt;/CheckBeforeRunning&gt;
</p><p>    &lt;MinFreeDisk&gt;200&lt;/MinFreeDisk&gt;
</p><p>    &lt;MaxSize&gt;1024&lt;/MaxSize&gt;
</p><p>    &lt;MaxFolderCount&gt;100&lt;/MaxFolderCount&gt;
</p><p>    &lt;ResourcePolicy&gt;0&lt;/ResourcePolicy&gt;
</p><p>    &lt;FolderAction&gt;
</p><p>        &lt;Size&gt;0&lt;/Size&gt;
</p><p>        &lt;Age&gt;1&lt;/Age&gt;
</p><p>        &lt;Actions&gt;3&lt;/Actions&gt;
</p><p>    &lt;/FolderAction&gt;
</p><p>    &lt;FolderAction&gt;
</p><p>        &lt;Size&gt;0&lt;/Size&gt;
</p><p>        &lt;Age&gt;56&lt;/Age&gt;
</p><p>        &lt;Actions&gt;8&lt;/Actions&gt;
</p><p>    &lt;/FolderAction&gt;
</p><p>    &lt;FolderAction&gt;
</p><p>        &lt;Size&gt;0&lt;/Size&gt;
</p><p>        &lt;Age&gt;168&lt;/Age&gt;
</p><p>        &lt;Actions&gt;26&lt;/Actions&gt;
</p><p>    &lt;/FolderAction&gt;
</p><p>    &lt;ReportSchema&gt;
</p><p>        &lt;Report name="PAL Report" version="1" threshold="100"&gt;
</p><p>            &lt;Import file="%systemroot%\pla\reports\Report.System.Common.xml"/&gt;
</p><p>            &lt;Import file="%systemroot%\pla\reports\Report.System.Summary.xml"/&gt;
</p><p>            &lt;Import file="%systemroot%\pla\reports\Report.System.Performance.xml"/&gt;
</p><p>            &lt;Import file="%systemroot%\pla\reports\Report.System.CPU.xml"/&gt;
</p><p>            &lt;Import file="%systemroot%\pla\reports\Report.System.Network.xml"/&gt;
</p><p>            &lt;Import file="%systemroot%\pla\reports\Report.System.Disk.xml"/&gt;
</p><p>            &lt;Import file="%systemroot%\pla\reports\Report.System.Memory.xml"/&gt;
</p><p>        &lt;/Report&gt;
</p><p>    &lt;/ReportSchema&gt;
</p><p>    &lt;Rules&gt;
</p><p>        &lt;Logging level="15" file="rules.log"/&gt;
</p><p>        &lt;Import file="%systemroot%\pla\rules\Rules.System.Common.xml"/&gt;
</p><p>        &lt;Import file="%systemroot%\pla\rules\Rules.System.Summary.xml"/&gt;
</p><p>        &lt;Import file="%systemroot%\pla\rules\Rules.System.Performance.xml"/&gt;
</p><p>        &lt;Import file="%systemroot%\pla\rules\Rules.System.CPU.xml"/&gt;
</p><p>        &lt;Import file="%systemroot%\pla\rules\Rules.System.Network.xml"/&gt;
</p><p>        &lt;Import file="%systemroot%\pla\rules\Rules.System.Disk.xml"/&gt;
</p><p>        &lt;Import file="%systemroot%\pla\rules\Rules.System.Memory.xml"/&gt;
</p><p>    &lt;/Rules&gt;
</p><p>&lt;/DataManager&gt;
</p><p>&lt;/DataCollectorSet&gt;
</p><p>

 

So now that we have run the Data Collector Set and got our results, how do we discover if there is an issues with the memory? The information below will help you troubleshoot your results. The source information has come from an excellent blog CC Hamed who is a Microsoft Support Engineer. The blog goes into more detail so it is well worth a read.

Memory \ %Committed Bytes in Use:

If this value is consistently over 80% then your page file may be too small.

Memory \ Available Bytes:

If this value falls below 5% of installed RAM on a consistent basis, then you should investigate.  If the value drops below 1% of installed RAM on a consistent basis, there is a definite problem!

Memory \ Committed Bytes:

Keep an eye on the trend of this value – if the value is constantly increasing without levelling off, you should investigate.

Memory \ Pages / sec:

This will depend on the speed of the disk on which the page file is stored.  If there are consistently more than 40 per second on a slower disk or 300 per second on fast disks you should investigate.

Memory \ Pages Input / sec:

This will vary – based on the disk hardware and overall system performance.  On a slow disk, if this value is consistently over 20 you might have an issue.  A faster disk can handle more.

Memory \ Pool Nonpaged Bytes:

If Nonpaged pool is running at greater than 80%, on a consistent basis, you may be headed for a Nonpaged Pool Depletion issue (Event ID 2019).

Memory \ Pool Paged Bytes:

Paged Pool is a larger resource than Nonpaged pool – however, if this value is consistently greater than 70% of the maximum configured pool size, you may be at risk of a Paged Pool depletion (Event ID 2020). 

Process (_Total) \ Private Bytes:

Similar to the Committed Bytes counter for memory, keep an eye on the trending of this value.  A consistently increasing value may be indicative of a memory leak

LogicalDisk (pagefile drive) \ % idle time:

If the drive(s) hosting the page file are idle less than 50% of the time, you may have an issue with high disk I/O

LogicalDisk (pagefile drive) \ Split I/O / sec:

Issues relating to Split I/O depend on the disk drive type and configuration.

Making SharePoint Search Available in Windows Explorer

In SharePoint 2010 there is a nice little feature in a search site called a ‘search connector’ that enables you to run search queries from Windows Explorer. The search connector is displayed in your favourites drop down. Also, you can have more that one search connector if you have a number of different search sites configured in your SharePoint environment and name them accordingly e.g Intranet Search, Team Site Search etc.

Below are the details of how to set up a search connector:

1) Run a search query from your search site (Any OOTB search site) and click on the third icon below which will be at the top of your search results and slightly to the right.

2) Select ‘Add’ to the Add Search Connector prompt’.

Your Search icon will now appear in your Favourites menu in Windows Explorer. The icon can be renamed accordingly.

Setting up Incoming Mail to a List or Calendar in SharePoint 2010

In SharePoint 2010 there is the capability to enable incoming mail directly to a list or a calendar. This is a very useful feature for your SharePoint 2010 environment as it can enable your users to send documents directly to a list and send meeting requests to a shared SharePoint calendar. The benefits of sending meeting requests to a shared SharePoint calendar is that your users may have a requirement to show the meetings that are taking place for a certain project or office depending on how your site is being used.

Before I go through the steps in how to set this up, you need to be aware that the shared Calendar in SharePoint is not stored in Exchange. It is stored in SharePoint along with are any attachments that are sent to the shared Calendar.

Please note that I am not going to include the steps of creating the Directory Management Service which will enable SharePoint to be able to write to Active Directory. I will be covering creating the SMTP Service on the SharePoint servers, configuring the SMTP Send Connector in Exchange, configuring Incoming Mail in Central Administration and enabling incoming mail in your list/calendar. This will include ensuring that your mail address will be using a friendly name e.g ‘@mail.contoso.com’. If you wish to find out more about how to set up Directory Management Service with SharePoint then I can recommend SharePoint George’s blog.

Step 1: Configuring the SMTP Service on the SharePoint Server(s)

The first step is to set up the SMTP Service. This needs to be set up on your SharePoint servers that you have designated to handle the mail (eg web front end servers).

SharePoint 2010 requires the SMTP Service to be running, so you need to go to Features in Server Manager and install the SMTP Service on your Windows Server 2008 R2 server.

Once installed you will get the results screen to show a successful install.

Now open the IIS 6.0 Management Tools from Administrative Tools and configure the SMTP Service.

Go to the properties of the ‘SMTP Virtual Server’, click on the General tab and enable logging for troubleshooting. Then click on the access tab and enable “Anonymous access”. On the same tab, go to “Relay” and select the settings in the screenshot below.

Now go to messages and set the relevant settings for your organisation, an example is in the screenshot below. Then go to the Delivery tab and you can make the settings as you desire. I have gone for the default settings.

On the Security tab add the service account this is running the SharePoint 2010 timer service. This is the SharePoint service account that will be accessing the SMTP service to collect the mail. Ensure the same service account has read/write permissions to the mailroot folder (C:\InetPub\Mailroot). This will enable mail to be deleted once the timer service has collected the mail item.

Your SMTP Service will now look like the image below.

The domain name is automatically set as the ‘servername.domain’. The domain name is what is specified in the incoming email settings in Central Administration, so we need to ensure this is a friendly name for the users. In IIS 6.0 change the domain name to a friendly name e.g ‘mail.contoso.com’. This is important as the same name will need to be specified as the name of the Address Space in the SMTP Send Connector and the Incoming Email settings in Central Administration. So it needs to be a functional and user friendly name. Nb: The Domain Name does not need to be registered in DNS as a host record or MX record as Exchange will handle the address space.

Step 2: Configuring the SMTP connector to send mail to SharePoint.

The next stage is to enable mail to be routed to your SharePoint farm from Exchange. This is configured in the Exchange Management Console using the SMTP Send Connector. The SMTP Send Connector effectively routes mail to the relevant SharePoint Servers hosting the SMTP service (configured in step 1). Before you create the SMTP Send Connector, in DNS create an MX record for each of your SharePoint servers hosting the SMTP Service. The MX records needs to have the same naming convention for each SharePoint Server. An example is below:

MX Record ‘sharepointmail.contoso.com’ mapped to ‘sharepointwfe01.contoso.com’, with cost of ’10’

MX Record ‘sharepointmail.contoso.com’ mapped to ‘sharepointwfe02.contoso.com’, with cost of ’10’.

In Exchange Management Console, create a SMTP Send Connector with a useful name e.g ‘SharePoint 2010 Incoming Mail’.

In the address space, specify the address space name that you wish to use. E.g ‘mail.contoso.com’ with a cost of ’50’.

In the Network Settings, specify to use ‘Route Mail through the following smart hosts’ and add the name of the smart host which is the name of the MX Records. E.g ‘sharepointmail.contoso.com’.

In the Source Server section, add the relevant Mail servers that you wish to associate to the Send Connector.

Nb: If you have two or more servers and want to send mail to one server first, then specify the server1 with a cost of 10 and the server2 with a cost of 20. Mail will go to server1 first and then to server2 second if server1 is unavailable. This will need experience of the Exchange and AD as the cost is associated to the MX Record.

Step 3: Configuring SharePoint to use Incoming Mail

The penultimate stage is to configure SharePoint with the relevant incoming email settings and enable incoming mail on your document library or calendar. To do this, go to Central Administration, System Settings and Configure Incoming Mail. Turn the feature on and specify’ Advanced’ rather than ‘automatic’.

Specify ‘No’ for the Directory Management Service

Specify the mail server display name as ‘mail.contoso.com’. This is the same name as the SMTP Send Connector and the Domain Name in your SMTP Service so it is important to keep the naming convention consistent. This will be the name space for all your email address that you create for your document library or calendar.

Specify the drop folder as ‘C:\inetpub\mailroot\drop’. This is the folder where Exchange will drop the email\calendar item. The item will then be picked up from the SharePoint 2010 Timer Service and inserted into the relevant document library or calendar.

Step 4: Enable Incoming Mail on your Calendar or Document Library

The final stage is to now enable incoming mail on your calendar or document library. This can be done by going to the ‘List Settings’ of your doc library\calendar and under communications select ‘Incoming Email Settings’ and in the incoming email section type the name of your email address. For this example I have used ‘calendar@mail.contoso.com’. This will ensure that all meeting requests that are sent to this address will be added to the calendar.

If you would like to monitor the mail progress then I would recommend monitoring the drop folder for the first couple of test emails\appointments to ensure the item is dropped in that directory and collected by the timer service. Please note that the timer job ‘Microsoft SharePoint Foundation Incoming E-Mail’ checks the folder every couple of minutes to collect mail.

Storage Metrics in SP2010 Service Pack 1

Quite a nice feature in SharePoint 2010 that has gone relatively unnoticed in SharePoint 2010 service pack 1, is the Storage Metrics feature. It is available under Site Collection Administration and it enables you to get useful information regarding how content is being used within your site collection. The report will give you a list breakdown of the content and the physical size along with the percentage of the site quota that has been used. This is particularly useful for Site Collection Administrators and IT Pro’s who will want to monitor the size of the content for particular sites. You can take this a step further with using third party tools such as Axceler’s Control Point application which will give you trend analysis on your content.

You are able to go granular with the results to the individual file level, so you can troubleshoot a growing content database by finding out what sort of file types are taking up space in a particular document library or list.

Please note that this feature is only available in Service Pack 1 of SharePoint 2010.

Copenhagen for the European SharePoint Conference 2013

This February it was the turn of the European SharePoint community to get a piece of the SharePoint pie. Four days of keynote speeches, workshops, seminars and a bustling array of third party vendors offering knowledgeable expertise, products and eye-catching prizes. The event was hosted at the Bella Center in Copenhagen which was a great venue and well organised but with a dreary wi-fi connection. So yes, I was one of the special people constantly manoeuvring to try and find a better signal (just like one of many lemmings each day!). I did raise a cheeky grin to see the speakers struggle with stuttering demos down to the wi-fi connection too. Steve Jobs would be smiling down with joy!

After the four days, I have gathered some useful information and discussion points that I will be blogging in the next few days. They will mainly be focused around the IT Pro arena so I will try and delve into as much detail as I can for you that will hopefully help you in your SharePoint journey.

On a separate note, the event was well organised and there was good banter around the booths from the energetic 3rd party vendors and creative stands (Well done to Axceler for the green smurfs..). The AvePoint and Axceler parties were very enjoyable and provided a chance to chat and see the experts let loose. There was even a rumour of projectile vomiting from a delegate on the way to Wednesday’s keynote speech, that would have been a long day for that poor chap!

Anyhow, keep an eye out for my next few blogs relating to the SharePoint 2013 Conference.

FAST Search 2010 – Crawl\Indexing Baseline Performance Testing

In preparation for making your FAST Search for SharePoint environment live in your production environment, it is imperative that you have tested your crawl performance to get your baseline statistics. Once you have your baseline stats then you will be able to re-test your production environment at monthly\six monthly intervals to ensure your FAST Search environment is performing consistently and there is no service degradation.

In order to get your baseline results you need to know which perfmon counters you will need to monitor during a full crawl and what is the acceptable performance range. The FAST Servers have different roles so there will be an overlap of components during the testing. I have detailed below the main perfmon counters to monitor during a full crawl and what to look for in your results. The performance ranges that I have detailed in the results below were the outcome of working with a Microsoft FAST Search specialist so they are accurate:

Processor Utilization – (System\Processor Queue Length): This is the number of threads queued and waiting for time on the CPU. Results: Divide this by the number of CPU’s in the system; if less than 10 then the system is running well.

Useful Additional Counter – (Processor\%Processor Time\_Total): Useful to show how busy the CPU is during the crawl process.

Memory Utilization – (Memory\Pages Input/Sec): This shows the rate at which pages are read from disk to resolve hard page faults. To go into more depth, this is the number of times the system was forced to retrieve something from disk that should have been from RAM. Results: Occasional spikes are acceptable but this should generally be very low, if not zero.

Useful Additional Counter – (Process\Working Set\_Total): This will show how much memory is in the working set.

Disk Utilization – (PhysicalDisk\Current Disk Queue Length\driveletter): Highly valuable counter to monitor. This shows how many read or write requests are waiting to execute to the disk. Results: Single disks it should be idle at 2-3 or below with occasional spikes being fine. For RAID arrays, divide by the number of active spindles in the array and aim for 2-3 or below. If the queue lengths are high then you will need to look at the memory\pages input/sec counter and potentially add more RAM. Aim for zero for this counter. The disk utilization will only be apparent on the Indexing server as the processing servers will mainly be using CPU and memory to process documents.

Useful Additional Counter – (Physical Disk\Bytes/sec\_Total) shows the number of bytes per second being written to or read from disk.

Network Utilization – (Network Interface\Output Queue Length\nic name): This is the number of packets in queue waiting to be sent. Results: If there is a persistent average of more than two packets in queue then you should look to resolve the network for a bottleneck. Aim for zero packets in the queue during a crawl. (Network Interface\Packets Received Errors\nic name): This is packet errors that kept the TCP/IP stack from delivering packets to higher layers. Results: This value should be low.

Useful Additional Counter – (Network Interface\Bytes Total/sec\nic name): This monitors the number of bytes sent or received.

Once you have completed your full crawl and evaluated your results, you will have a good idea of how your FAST Search environment has coped with the volume of content you are crawling and will provide you with vital information to determine if your FAST Search topology is adequate or whether you need to add any more hardware to your servers. I would recommend Microsoft’s article on performance and capacity planning for preparing your FAST Search topology.

In addition, I can also recommend using the Performance and Analysis of Logs (PAL) tool from Codeplex. You can extract a threshold xml file which will have a number of pre-defined counters for FAST Search, which you can then use with your perfmon Data Collector Set to gather and analyse your results. The beauty of this tool is that you can add your results file (.blg file) to the application and it will analyse the results against the recommended thresholds for each counter. Very useful and an excellent way to assist in presenting your results!

FBA Users automatically being made inactive from a Site Collection

An issue that had been causing no end of trouble in our MOSS 2007 extranet environment had been when FBA users appear to have been mysteriously deleted from a Site collection. This had been happening with no administration interference from the SharePoint site administrators and at times outside of when the Profile import synchronisation timer job had been run. This proved to be a challenge to further investigate and identify the cause of the issue.

The first step was to identify what was going on under the covers of SharePoint (i.e.in the content database). On a new occurrence of the issue I identified the users FBA login and ran the SQL query below against the userinfo table of the content database. The userinfo table contains the information on the SharePoint profiles that have been created in the database. A SharePoint profile is created the first time a user logs on to the site. To show the information for this instance I will use the name ‘d.newton’ as an example FBA login name.

select * from [ContentDatabaseName].[dbo].[UserInfo] where tp_login like ‘%d.newton%’

The SQL query above returned more than one user with the same tp_login name and more importantly the column tp_lsActive was set to ‘0’ for all the ‘d.newton’ logins. This effectively means that the user profile for ‘d.newton’ is inactive in SharePoint and no access is granted to the site collection. In addition, the column tp_deleted was populated with the same value as the users profile number. After comparing the duplicated tp_login names, I found that the names appeared to be the same but when you copy the login name from SQL to notepad I found that the duplicate tp_login names had a whitespace at the end. An example is below:

Tp_id = 551    tp_login = ‘membershipprovider:d.newton’    

Tp_id = 552    tp_login = ‘membershipprovider:d.newton ‘    (Single white space)

Tp_id = 553    tp_login= ‘membershipprovider:d.newton ‘    (double white space)

So how, when and why are the duplicated profiles created?? After running a number of custom reports I found that the duplicated profiles were being created by the SharePoint system account and after further testing against the custom extranet login.aspx it was discovered that the login page would accept a username with a whitespace at the end of the username ‘d.newton’ (Note: the password will need to be the same as the original username’s password). So what is happening in the userinfo table when you add the username with the whitespace? The original FBA user is made inactive with the tp_IsActive column being set to ‘0’ and a new profile is created with the same tp_login name with a whitespace and a new profile id but with no access to the relevant SharePoint site. From the user’s experience, they will not be able to see the relevant site. When the user logs in with the incorrect login, SharePoint detects that there is conflict in the users tp_login name where it is required to be distinct, so SharePoint automatically makes the original SharePoint profile inactive to resolve the conflict.

Armed with this new information, we now have the knowledge that the profile 551 with the username ‘d.newton’ with no whitespace is the correct profile and we can make this user active again from within site permissions of the relevant site.

It is also useful to know that we can now replicate the issue, which is good news! However, we need to find a resolution to stop this from re-occurring. The first task I would recommend undertaking is to change the extranet’s forms loginurl to the out of the box Microsoft login.aspx and re-test the same issue. It is very likely that you will find that the Microsoft’s default login.aspx page doesn’t have the same behaviour. The reason for this is that the Microsoft login.aspx page trims whitespace at the end of the username, which means that a username conflict or duplication never occurs. The steps to change the forms login page back to the OOTB login.aspx is detailed below:

1. On your web front end server, open the web.config of your extranet site and navigate to the ‘authentication mode’ section.

2. Change the forms loginurl to “/_layouts/login.aspx” (Details below)

<authentication mode=”Forms”>

<forms loginUrl=”/_layouts/Login.aspx” />

</authentication>

In our environment it turned out to be the custom login.aspx page that was causing the issue. Our custom login page was referencing code that was not trimming the username correctly and thus allowing the duplication of profiles to be created in the userinfo table. What is annoying is that the userinfo table accepts whitespaces in the tp_login column! If you encounter the same issue as us then I would recommend either resolving your code or create a new login.aspx page based on the Microsoft OOTB login.aspx.

Removing the Duplicated Users

As we now had a number of duplicated SharePoint profiles in the userinfo table, it would be useful if the profile synchronisation timer job would clean up the duplicates or inactive users. However, I have been advised by Microsoft that the profile synchronisation timer job will only delete Active Directory inactive users and not FBA inactive users. The only supported way to remove an FBA user from the userinfo table is via powershell and the article by Niklas Goude explains it very well.