How we deployed SharePoint WSP Solutions without downtime


 

Introduction

As your SharePoint environment grows and more solutions are built on the platform it becomes critical to the business. Often it gets to the level that the business is very reluctant to allow SharePoint to be unavailable. This makes it difficult to plan application deployments which generally cause downtime.

Recently, I have been working on an environment which was in use globally and therefore the window for taking down the SharePoint farm is very small.

With that in mind I thought I’d share the process that I now use to deploy SharePoint WSP solutions without any downtime to the SharePoint farm. There are going to be times where you cannot avoid downtime, certain SharePoint object model calls cause all the server to recycle their application pools.

In order for deployments to be executed with no downtime there are a few requirements. There are infrastructure requirements as well as following a process.

 

Farm Configuration Requirements

In order to deploy SharePoint solutions there must be at least two SharePoint Servers configured as Web Front Ends. These web front ends are running the Web Application Service instance.

The next step is to ensure that these servers are being load balanced, either using software load balancing such as Windows Network Load Balancing (NLB) or using hardware load balancers such as the F5 Network’s BIG IP.

The F5 Network’s BIG IP configuration guide can be found here.

Without either the load balancers or the two web front ends then you are not going to be able to prevent downtime.

Regarding the load balancers, they must be configured so that when a server is down then no traffic is directed to that server.

At one client this required a bit more thought. The client used the F5 Network BIG IP load balancers unfortunately the load was not balanced as Round Robin balancing was being used.

Also the method to detect whether a server was down did not work. The way that the load balancers tested that a server was available was by using a low level call into IIS looking for an HTTP status of 200. This meant that as long as IIS up then the server was up and it would receive requests,  even if SharePoint was failing for some reason.

This was fixed by using a cURL script which allows Windows Authentication to be used to call into SharePoint. The script would then look for a particular text string in the page. As long as this was found then the server is up and available.

The last tweak to the load balancers were configured to use Observed dynamic balancing. The following article by Don MacVitte gives a great overview of the different types of load balancing.

Observed load balancing, balances the load by using a number of metrics, this snippet from the article explains in more detail:-

Observed: The Observed method uses a combination of the logic used in the Least Connections and Fastest algorithms to load balance connections to servers being load-balanced. With this method, servers are ranked based on a combination of the number of current connections and the response time. Servers that have a better balance of fewest connections and fastest response time receive a greater proportion of the connections. This Application Delivery Controller method is rarely available in a simple load balancer., including the response time from each server.

 

Deployment Process

SharePoint Deployment Without Downtime Diagram Process

With these infrastructure requirements in place it is possible to deploy solutions without downtime.

For SharePoint 2010 deployments, PowerShell is the tool of choice when deploying SharePoint Solutions.

Once you have the farm configured correctly then you will need to deploy the solutions through the SharePoint 2010 Management Shell.

So generally I login onto the server through Remote Desktop, copy over the solution file(s) into a folder such as c:\install\[solutionname]\[solutionversion].

Once the files are copied over:

  • Start SharePoint 2010 Management Shell
  • Change Directory to where the SharePoint solution (.wsp) are found.
    • cd c:\install\[solutionname]\1.0\

    The approach that I take is to use PowerShell variables where I can. This helps reduce the amount of typing and there are less mistakes made.

  • Create a variable to the current folder
  • $curdir = gl;
  • Create a variable for the solution’s filename
  • $solutionname = "MySolution.wsp”
  • Add the solution
  • Add-SPSolution –LiteralPath $curdir\$solutionname

Next is the important part, the solution has been added to the Configuration database and the next step is to deploy the solution to the servers. To ensure that there is no downtime, we will ensure that the deployment only occurs on one machine at a time.

Depending on the type of solution being deployed there are a few additional attributes that you may have to specify to the Install-SPSolution command.

However the most important attribute we need to remember is the –Local attribute this will only ensure that the solution deployment will only occur on the local server.

Other attributes include:-

  • WebApplication – this will deploy the solution to a specific Sharepoint Web Application
  • GacDeployment – this option allows a solution which contains .NET assemblies to be installed in to the GAC to be installed.
  • CasPolicies – this option allows a solution which contains code access security policies for the assemblies / web parts to be deployed.

Once the Install-SPSolution command has been run then the solution deployment success needs to be checked. This can also be achieved using PowerShell.

   $checkSolution = Get-SPSolution $solutionname;   $checkSolution.LastOperationalDetails;  

When this has been executed an output such as the following will be displayed for the local server.

sharepoint_solution_lastoperationdetails

The process now has to wait until the SPSolution.LastOperationDetails call returns back that the deployment is successful.

Once the deployment has completed, I now restart IIS and the SharePoint Timer Job Service using the following PowerShell.

 Restart-Service sptimerv4;   iisreset;   

The installation of the SharePoint solutions will have caused IIS to restart and therefore the server will not have responded to the load balancer. Provided that the load balancer has been configured correctly the server should no longer be processing requests to clients and therefore there is no loss of service.

Depending on configuration of the load balancers, the time it takes for a server to start responding to requests will based on a setting called ramp up time.

The slow ramp up time is the time the load balancer will wait after the server has come back online before it starts sending all requests to the server. This gives the server time to get back on its feet and processing those requests.

Once the server is back online, the SharePoint solution process can be repeated for each of the other SharePoint Servers.

 

Deployment Gotchas

Although this approach will for most cases work there are a few gotchas that you need to watch out for. In some of these occasions all SharePoint WFEs will have their application pools restarted.

Currently these are:-

  • SPWebConfigModification – its a good practice to apply changes to the web.config files for an IIS Web Application using the SharePoint SPWebConfigModification object. This will ensure that any settings required in the web.config for your solution will be applied. This reduces the likelihood of an issue with the solution due to down to missing configuration. Also if new servers are added to the farm then their web.config files wil be setup correctly. This will save you lots of time if you ever have to do a disaster recovery exercise!
  • SharePoint Field Controls – if your solution includes a new custom SharePoint field. Then this will not be available until all SharePoint servers have been updated. This can be more of a problem when for example you are using something like Telerik’s RadEditor control and are performing an upgrade. When these type of deployments are being made then its best to inform the users that the application will not be available. At least the rest of the SharePoint farm is up and running!

Note: As more issues are found then this section will be updated.

Deployment Script

The final deployment script is as follows:-

  cd c:\install\[solutionname]\[versionnumber];   $curdir=gl;   $solutionname=”SolutionName.wsp”;   Add-SPSolution –LiteralPath $curdir\$solutionname;   Install-SPSolution –Local –Identity $solutionname –GACDeployment –CASPolicies;   $checksolution = Get-SPSolution $solutionname;   $checksolution.LastOperationalDetails;   

 

Finally

I hope that you find this useful and would love to hear about your experience and approach to deployment of SharePoint solutions.

PowerShell to Detect Web Configuration Modification jobs


Introduction

Over the past few months I have been working on some automated SharePoint deployment scripts using Team Foundation Server and the TFS build service. As the development became more complex so did the scripts. One of the problems we had to work around was activating a number of features at the same time.

The deployment reinstalls five SharePoint solution packages, uninstalling and reinstalling all of the Features which are packaged in the solutions. Now some of these features perform custom tasks using feature receivers. For example making changes to the web.config using SPWebConfigModfications. However, one of the problems with modifying web.config using feature receivers is that if they are run on a multi server SharePoint farm then the modifications are performed using a timer job.

When you try and activate two features, one after the other, which perform the web configuration modifications you get the following error message:-

"A web configuration modification is already running"

This kills the deployment script!

Thus we needed to find a way to detect whether a web configuration modification job is in progress.

Understanding the process was made more dificult as the development machine was part of a single server SharePoint farm. With single server farms, when aa SPWebConfigModification is applied to a SPWebApplication then the modification is executed in a different manner and occurs immediately.

Anyway, after some digging about using Reflector I found an SPJobDefinition which looked like the web config modification timer job and say down and wrote the following PowerShell script.

Solution

To be able to detect whether a web config modification is running we need to see if a timer job is active.

The timer job that performs the web config modification has the name “job-webconfig-modification”.

To detect whether a web configuration is pending or running.  As web configuration operations occur against a web application. Therefore using the SPWebApplication’s SPWebService property which has two collections relating to the timer service:-

  • JobDefinitions – this contains the jobs that are pending to be run
  • RunningJobs – this contains the jobs that are currently running

Detecting whether a web configuration modification is in progress, we need to check those two collections for our “job-webconfig-modification” timer job.

The following PowerShell function, IsWebConfigModificationJobPendingOrRunning, is used to check whether a Web Configuration operation is taking place. The function takes a SPWebApplication as a parameter and returns true if the web configuration modification operation is running.

 


################################################################################################
#Function: IsWebConfigModificationJobPendingOrRunning
#Description: Checks to see if a web configuration job is pending or in progress
################################################################################################
function IsWebConfigModificationJobPendingOrRunning([Microsoft.SharePoint.Administration.SPWebApplication] $webApp)
{
Write-Host "IsWebConfigModificationJobPendingOrRunning() Enter";

$bRunning=$false;
try
{
$webConfigPendingJobCount = 0;
$webConfigRunningJobCount = 0;

#if there is only one job returned then array is not created
$webConfigPendingJobs = $webApp.WebService.JobDefinitions |
Where-Object {$_.Name -like "*job-webconfig*"};
$webConfigRunningJobs = $webApp.WebService.RunningJobs
| Where-Object {$_.Name -like "*Web.Config*"};

#if there is only one job returned then array is not created - check to see if an object has been returned
#check to see if its an array if not then presume its a job.
if($webConfigPendingJobs -ne $null)
{
if($webConfigPendingJobs.GetType().BaseType.Name -eq "Array")
{
$webConfigPendingJobCount = webConfigPendingJobs.Count;
}
else
{
$webConfigPendingJobCount = 1;
}
}

#if there is only one job returned then array is not created - check to see if an object has been returned
#check to see if its an array if not then presume its a job.
if($webConfigRunningJobs -ne $null)
{
if($webConfigRunningJobs.GetType().BaseType.Name -eq "Array")
{
$webConfigRunningJobCount = webConfigRunningJobs.Count;
}
else
{
$webConfigRunningJobCount = 1;
}
}

Write-Host "Found " $webConfigPendingJobCount " Pending Jobs and " $webConfigRunningJobCount " Running Jobs";
if(($webConfigPendingJobCount -gt 0) -Or ($webConfigRunningJobCount -gt 0))
{
$bRunning=$true;
}
else
{
$bRunning=$false;
}
}
catch
{
Write-Error "IsWebConfigModificationJobPendingOrRunning Exception: $_ ";
}

Write-Host "IsWebConfigModificationJobPendingOrRunning() Exit : $bRunning";
return $bRunning;
}

 

Example Script

The script below shows part of the deployment script. This would be used  to loop through an array of feature names, check their scope and activate them. The script waits until the feature has activated and checks that there are no web configuration modification jobs running.

$SharePointWebApp = Get-SPWebApplication "http://sharepoint";
$spfeature = "Feature1";

$checkspfeatureactivation = Get-SPFeature -WebApplication $SharePointWebApp
-Identity $spfeature -ErrorAction SilentlyContinue;

if($checkspfeatureactivation -eq $null)
{
Write-Host "Activating Feature: " $spfeature.Name " Scope: " $spfeature.Scope;
Enable-SPFeature -Identity $spfeature -Url $SharePointWebApp.Url -Confirm:$false;

#check that web config job mod not running, wait if it is
do{
Start-Sleep -Seconds 2;
Write-Host "Waiting for Web Config Job to be Completed";
$configJobRunning = IsWebConfigModificationJobPendingOrRunning $SharePointWebApp;
}while ($configJobRunning);

}
else
{
Write-Host "Feature Already Activated: " $spfeature.Name " Scope: " $spfeature.Scope;
}

 

I hope that helps someone who is having similar problems, I hope to add a bit more about the deployment process that I have been developing in later blog posts.

Please leave comments if you need any help or there are any problems.

Here are the actual PowerShell scripts as txt files.

Save typing when Manipulating SharePoint Solution Files in PowerShell


 

Introduction

 

One of the things that annoyed me with PowerShell and SharePoint are the following commands:-

  • Add-SPSolution
  • Update-SPSolution
  • Actually anything that requires a –LiteralPath parameter

Don’t get me wrong I think that using PowerShell to admin SharePoint is fantastic but these commands bug me because they require a full path to work correctly. Unfortunately you don’t seem to be able to use the full stop (or period) to represent the current directory.

So you cannot do something like this:-

Add-SPSolution .\mysharepoint.wsp;

Instead you need to do this:-

Add-SPSolution c:\thisisareallylongpath\release1\section2\mysharepoint.wsp

Solution

Anyway, that got me thinking surely there is a better way and I can save myself some typing.

The solution is to use the command Get-Location.

So instead you can do this:-

$curdir = Get-Location;
Add-SPSolution–LiteralPath $curdir”\mysharepoint.wsp”;

Actually there is a PowerShell Alias called gl which is a shortcut for the Get-Location command.

So the shortest version in terms of saving your typing fingers is this:-

$curdir = gl;
Add-SPSolution –LiteralPath $curdir”\mysharepoint.wsp”;

Really hope that helps save you some time, also love to hear any other alternatives, there is bound to be an even better way of doing this.

SharePoint Utility to manage Content Type and List Event Receivers


Introduction

Recently I have been building a solution which makes use of Content Type Event Receivers. The event receivers were being deployed using declarative xml for example:-

<ContentType ID=”[Removed]” Name=”My Content Type” Group=”My Content Types” Version=”0″>
<DocumentTemplate TargetName=”/_layouts/CreatePage.aspx” />
<XmlDocuments>
<XmlDocument NamespaceURI=”http://schemas.microsoft.com/sharepoint/events“>
<Receivers xmlns:spe=”http://schemas.microsoft.com/sharepoint/events“>
<Receiver>
<Name>MyEventHandler</Name>
<Type>ItemUpdated</Type>
<SequenceNumber>10000</SequenceNumber>
<Assembly>MyAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=000000000000</Assembly>
<Class>MyAssembly.EventReceivers.ContentTypeEventReceiver</Class>
<Data></Data>
<Filter></Filter>
</Receiver>
……

Now this normally works great, except when you make a mistake. Its quite well known that upgrading Content Types that have been defined using declarative xml can be tricky.

This is one of the reasons why Microsoft released Feature Upgrades in SharePoint 2010 (if you haven’t seen this then please check out Chris O’Brien’s set of blog posts (http://www.sharepointnutsandbolts.com/2010/06/feature-upgrade-part-1-fundamentals.html) on the subject).

In this case the development was on a SharePoint 2007 environment and I found myself in a situation with deploying Content Type Event Receivers.

It seems that once the Content Type had been deployed and the feature switched on its quite difficult to make certain changes to the content type. This includes changes like the following:-

  • changing a field’s default value
  • changing whether a field is required
  • changing the event receiver definition
    However, unfortunately Gary LaPointe’s stsadm commands would not working correctly for my content type and Gaetan Bouveret’s utility only manages list event receivers but does look very good at that!

Solution

Therefore I started to write my own command line utility. The first version was quickly written to get what I needed to do, done.

I have updated it so that at least its a bit more intuitive but it still only allows you to modify event receivers at a content type or list level.

Please be careful because if you ask it to delete an event receiver then guess what it will delete the event receiver!

As with all these utilities please do not try using this on your production environment without performing some testing on a development environment beforehand.

If you have any comments, problems with the utility please leave a comment and I’ll take a look..

Utility System Requirements:-

  • SharePoint 2007 SP2 (Tested on April 2010 and April 2011 Cumulative Update)
  • Utility must be run from machine which is connected to the SharePoint farm that is being modified.

Usage:-

Console application name: ITSP.EventReceiverConfig.exe

Command Line Arguments:-

  • -url : full url of site that hosts content type or list
  • -mode : [list | add | remove]
  • -targettype : [list | contenttype | assembly]
  • -targetname : [name of list | name of content type | name of assembly]
  • -assembly : [full assembly name including version number etc]
  • -class : [fully qualified class name that is implementing the event receiver]
  • -type : [itemadding | itemadded | itemupdating | itemupdated]
  • -sequence : [number bigger than 1]

The utility with releases, document and source code can be found on Codeplex.

Codeplex Project: http://itspeventreceiver.codeplex.com

Documentation can be found here: http://itspeventreceiver.codeplex.com/documentation

Issue with Updating SharePoint Publishing Page Layouts and Ghosting


Introduction

This post has come about due to some challenges when upgrading a publishing site page layout.

Before we delve into the issues I best explain how the Page Layouts are setup. The page layouts are all contained within a feature called “Company.Intranet.Publishing” which are deployed to the folder:

SharePointRoot\Template\Features\Company.Intranet.Publishing\PageLayouts.

Each of the page layouts are deployed into the master page gallery (/_catalogs/masterpages) using a Module. The page layouts are deployed as uncustomised or ghosted files.

The fact these resources are deployed as ghosted files is very important as you will see in the next section.

Finally the feature is deployed using the SharePoint Solution framework as a single SharePoint WSP using WSP Builder.

What do you mean by Ghosted / UnCustomised files?

There has been quite a lot of information published about this and the following are good resources:-

My understanding is that a ghosted file is where the binary content is actually stored on the file system of the server and a pointer to that binary content is held in the content database. If a user modifies the content of a page layout in a site collection by a tool such as SharePoint Designer then a copy of that binary content is taken from the file system and merged along with the changes made by the user and stored in the content database.

Hence when you retrieve an uncustomised file its binary content is being pulled from the content database.

Why are Ghosted / Uncustomised files preferred?

Recently at the SharePoint Best Practices conference, Eric Shupps and Rob Foster did a great presentation on Fine Tuning and Optimising SharePoint Performance. One of the key points in improving performance is to ensure that files are ghosted/uncustomised for the following reasons:-

  • Ghosted files all point to a single file and hence they all point to the same binary content. This means only one file needs to be cached and the load on the database is significantly reduced.
  • Actually Unghosted files cannot be cached within a BlobCache. Unghosted files are therefore slower to load and render (10% slower on average) than a ghosted file.

Upgrading Page Layouts

The method used to upgrade page layouts is simply by redeploying the files through an updated wsp solution file. We have also been careful through training and guidance with users to ensure that page layouts are not edited through SharePoint Designer.

Therefore if a change to a page layout is required, then the page layout is updated within the feature directory using Visual Studio and the WSP solution file is rebuilt.

The solution is then ready to be updated using the command:-

stsadm –o upgradesolution –filename [solutionname.wsp] –name [solutionname.wsp] –allowgacdeployment –allowcaspolicies –immediate

The change to the page layout is then reflected within the SharePoint site.

Why are my changes are not seen?

More history, our last deployment required that we tidied up the page layout’s title and description.

One of the problems with using a Module to deploy files within a feature is that with SharePoint 2007 it is difficult to make changes to existing files deployed by a module without resorting to code. Any changes to the filenames or properties that are deployed by the Module are not reflected until the file is deleted from the site collection.

As there were not that many site collections, we took the decision not to write any code and we modified the page layout titles and descriptions through the SharePoint UI.

Unfortunately when we did our next release into our testing environment the changes to the page layout markup were not seen within the site.

After checking that the page layouts were being deployed correctly to the the WFE’s file system we started thinking of the reason. Finally I thought it must be due to unghosting of the files.

Normally you can use SharePoint Designer to revert page layouts to their site definition. So I fired up SharePoint Designer and browsed to the /_catalogs/masterpages document library but could see that the Page Layouts were not customised.

I started to search for tools to find out which files are customised. The legendary Gary LaPointe (@glapointe) came to the rescue.

If you don’t deploy Gary LaPointe STSADM tools on your Production/UAT environments then I plead with you that you start to do so as soon as possible. Time and time again I have been in a bind and Gary’s STSADM commands have got me out of trouble.

The commands that saved the day this time are:-

The commands give a lot of flexibility, though I will warn you that you need to be a bit careful as you could quickly reghost all the files within your site collection if you are not careful.

I performed the following command:-

stsadm –o gl-enumghostedfiles –url http://sharepointsite.domain.com/

This displayed a list of files that were unghosted. Funnily enough all the page layouts that we had modified and changed their Title and Description fields had become unghosted!

Once we had a list of the unghosted page layouts we could reghost each of them in turn using the command:-

stsadm –o gl-reghostfile –url http://sharepointsite.domain.com/_catalogs/masterpage/RandomPageLayout.aspx –Scope File

This command would reghost each of the files, there were occasions where the page layout reghosting would fail, using the –force option for the gl-reghostfile would ensure that the page was ghosted.

Now looking at the site the page layouts had been updated and reflected the changes that had been deployed by the update solution.

Hope that helps!

Corrupted List Url Fix: Link to SharePoint List points to /_layouts/listedit.aspx?List=[ListGuid]


Issue

Had an issue at a client where a SharePoint task list got corrupted.

The issue had come about due to the list being copied using Metalogix SharePoint Migration Manager within the same site. The copy of the list had been corrupted so that if you clicked on “View All Site Content”, then clicked on the list name you were taken to the Edit List Settings page (/_layouts/listaspx.aspx?List=[GuidOfList]). Once in the list settings page, clicking on the breadcrumb to take you back to the list would take you to the web application homepage.

Solution

The fix was actually quite straightforward, if you typed in the url of the list correctly, http://sharepointsiteurl/sites/site/lists/My%20List%20Name, you could access the default list view with no problem. So I used a similar method to fix that I have used in this blog post, Manage Content and Structure error.

To fix:-

  • Delete the list using List Settings->Delete this list.
  • Restore the list from the recycle bin

Voila! The link within “View All Site content” is now working as expected.

My Bookshelf 2011


Andrew Woodward (@andrewwoody) from 21apps posted a picture of his bookshelf on his company’s site.

His bookshelf is completely different to mine which I am going to take on board and look into in more detail. However, its interesting to hear about what people are reading.

So here is mine (well the ones that I am allowed to have out in the open!) …

simondoy_geek_bookshelf