Before we get started this is not a fix to the problem (at the moment) we are still awaiting a diagnosis from Microsoft. However, in my experience if you have a similar problem and can google the issue and find out that someone else has the problem and they can give you some sort of advice then that does help!
Last week we had an issue with our production servers. I received a phone call saying that at 17:25 a number of web pages missed page content. The homepage was kinda normal with the web parts all there but the web parts all in the same web part zone on the left hand side.
Although there didn’t seem to be any data loss we were not sure so we ended up rolling the database back to 15:00. After the event this now seemed a bit hasty but at the time where its more important to get the systems up and running.
So thanks to the good DR procedures and database admin we were able to get SharePoint back up and running, losing an hours worth of data.
Because we took a backup of the corrupted database we could do a post mortem and also look at the data that had changed to work out the documents/list items that were missing. Not a simple process but doable.
So the following day my plan was to understand what happened.
Firstly we got Microsoft involved and set up a call. Secondly I took copies of the database and then attached them to my development SQL box.
Using a very cool bit of software by ApexSQL I was able to look through each of the transaction logs from the time of restore to what I am now going to call D-day. A hint: using a filter on EventData table you can quickly see which items were modified.
After a few hours of analysis I didn’t turn up much so I decided to actually mount the database onto SharePoint. Adding the database as an additional content database through Central Administration->Application Management I was able to view the websites again.
Still nothing had changed the web parts were all over the place. Now with more time on my hands I started to look into things a bit more. I went on to a page which had no content and then edited the page.
I added a web part, clicked the Advanced Web Part link and looked through the Web Part collections. Clicking on the Closed Web Parts collection revealed all the web parts that were previously on the page!!!
After more investigation it seemed that there had been no data loss at all apart from the web parts disappearing. Dragging these web parts back on brought the pages as they were.
So for some reason these web parts closed automatically, now I don’t have an idea why that happened. There was no development going on at that time which would explain the issue… however with MS on the case we should have some answers. When we do I will update this post.
I am in the process of writing an application which will help find any webparts which are closed on a site, this will get published via CodePlex. I’ll add a link on here when thats finished.
If you want any more information then just leave a comment and I’ll get back to you.
Thanks for reading.