NTFS event ID 55 after disk extend on Server 2008

This isn't the first time I've ran into this, but hopefully it will be the last.  The first time was with a volume containing SQL data.  This time it was one with Exchange mailbox stores.  A little bit of thought upfront will go a long way in preventing this so hopefully the following helps.  (Chances are though, if you are reading this, you probably searched Google after it was already done, but hopefully it still helps and keeps you from jumping off the cliff!)

My Scenario:  An SBS 2008 server was running low on space.  It was running as a VM inside vSphere with the appropriate the licenses to grow disks without downtime.  It was 2:30AM on a Thursday and I was just trying to knock an item off my to do list before bed.

My Typical Steps:  I increased the disk size on the VM, then went into Windows, Disk Management, rescan disks, right clicked the volume and selected Extend.  Since it was 2008 (non R2), there is a bug that seems to set the size 1MB larger than possible, so adjust it down and then back up (if you have ever done it, you will know what I'm referring to) and finished.

The Problem:  After running the extend, I got an error message saying "Invalid parameter".  In Disk Management, the volume appeared to be resized.  However, I noticed that in Windows Explorer, the drive was not showing the additional space.

What I Did Next (not quite right):  In the past when I have seen that, I found that running "diskpart", selecting the appropriate volume, then running "extend filesystem" typically fixes Explorer.  I tried it and received an error (wish I would have done a screen capture on it) but it also said the command was successful.  Explorer still didn't show the free space, and then I noticed a Windows error reporting window stating that "store.exe" crashed:

I checked Event Viewer and found NTFS event ID 55's:

Since this was the Exchange volume, I then went into Exchange Management Console and noticed the mailbox store was down and when I tried to mount it I received an error:

Now panic sets in.  Corrupt disk/volume, database that will not mount, it was 3AM and I was fighting something that should have been point, click, done.  Backups are good, but this server is several hundreds of GB so a restore would take way to long and potentially lose hours of data.

What I Did Next (better):  Since it is a VM on VMware, I had the luxury of creating a quiesced snapshot before proceeding so I did that.  Next, I stopped all services that might be interacting with the volume in question.  In this case, it was Exchange only so I stopped those services (and the Microsoft Search service).  Next, I dropped to a command line and ran chkdsk (no /F yet) on the drive.  It found an error:

Attribute record (128, "") from file record segment 6)

Did a little research on this and it looked like doomsday based on what I read.  Most people said they had to format.

Since I had my snapshot (and backups as worst case), I went ahead and tried running "chkdsk E: /F", allowed it to dismount the volume, and it ran through and said it fixed the problem.  I was then able to restart all the services and the mailbox store mounted!  Mailboxes were available and mail started flowing again.  Yay!

Hindsight:  I might have just been able to restart the Information Store service but that wouldn't have fixed the NTFS corruption.  After the first time we went through this, our team documented a pretty failsafe procedure for extending disks like this on Windows Server 2008 and up that I failed to follow.  (Windows 2003 is a different beast but can be similar using diskpart and/or a tool called ExtPart.exe (32-bit only) if you need to extend a critical volume.)  Basically, for 2008 and up, we have found that the following seems to work well:  extend the disk in VMware, fresh reboot, stop the appropriate services in the guest, take a snapshot with VMware, then try to extend the disk using diskpart extend, then diskpart extend filesystem.  So far, we have never had an issue doing it that way.  I decided to be a bit of a renegade because of lack of sleep and I have had such good luck lately extending disks in 2008 R2 on the fly through the GUI that I didn't feel the need to abide by the safe method.  Err... I mean... I just needed something good to write about so I decided to do this on purpose... yeah... 

No more renegade warrior for me, at least, not tonight :)  Special thanks to JK for somehow sensing I was in trouble, waking up and replying to an email, and being a calm voice with several good ideas (and for not rubbing it in my face that I didn't follow the safe method).

Note: This really only seems to be an issue on 2008 (non R2) that I have seen.

VCAP-DCA: The wait is over...

Finally, after 17 days of anxious anticipation, the exam results are in...  and... wait for it... I PASSED!!!  w00t w00t!  I needed a 300 to pass and I scored 358 out of 500.  Not the best score in the world but considering I decided to just "try it" and only gave myself two weeks to study, I couldn't be happier!  Now, if they would just hurry up and send my ID so I know what number I am...

Update:  Number 474!

Poor performance after upgrading from vCenter 4.1 to 5

I recently upgraded a VMware vCenter installation from 4.1 to 5.0.  Everything went well during install from what I remember but afterward I noticed that things kept timing out when managing the vSphere environment.  Upon further investigation I found that java.exe seemed to be the culprit.  I tried updating it, tried to remove VMware Update Manager, tried simply shutting down the vCenter and SQL services, but none of that helped.  The CPU was still pegged at 100%.  At that point I started shutting down services one at a time, starting with anything VMware related.  Within a few minutes I found that the Converter services, which were installed in the 4.1 environment but not updated, were the most likely culprits.  I removed the programs related to Converter and CPU dropped to 0%.  At this point I decided to reboot and ensure things were sane.

Unfortunately, after the reboot, the java.exe CPU issue had returned.  I then started the troubleshooting process again but this time with a little help.  I downloaded and ran an old favorite of mine:  ProcessExplorer from Microsoft/Sysinternals.  This let me see more information about each process such as what the running environment looks like and which process is considered the parent.  There were three separate java.exe's running but the most interesting one was launched by the vSphere Web Client services.

I went ahead and removed the Web Client.  While I was at it, I also removed the new Dump Collector and the Syslog Collector services.  They each prompted for a restart so I did and checked CPU usage again when it came back up.  Still at 100% with another java.exe fighting sqlserver.exe for highest allocation.  This time it was the Inventory Service.  Even though it was pegged, it was still more responsive than usual so I feel like I am making some progress at least.  Back to Process Explorer...

While I was looking at some articles about this to understand what it did, I found that the CPU returned to a sane state.  This must have been the Inventory Service catching up after the reboot.  I'm still seeing sqlserver.exe running at a pretty constant 50% (essentially the equivalent of 1 CPU core) but I would half way expect that.  I also see Tomcat spike up occasionally and together with SQL consume 100% CPU.  At this point I am going to continue to monitor the situation.  If I find anything else I will try to post.

Just for reference, the vCenter server in question is a VM with 2 vCPU and 8GB RAM running on FC storage managing 6 hosts with approximately 60 VM's so I would assume based on the recommended specs that this would be sufficient.  I may increase the memory to 12GB to test further though.  Maybe the next step is to finally separate out the SQL instance to a dedicated server since we are technically over the supported limit of 5 hosts and 50 VM's (although I think that is more of a size restriction but who am I to argue with VMware best practices).

*UPDATE:  Well, I feel like an uber douche.  I was getting ready to call it quits when I saw some alerts coming in from one of our monitoring tools saying that our newest host, the one where the vCenter server ran, was swapping memory in and out frequently.  I started looking and found a fair amount of memory ballooned and swapped.  This was totally unexpected because that host has 120GB of RAM and only about 60GB was actually in use.  Upon further investigation I found that several VM's had limits set on their memory, including (you guessed it) the vCenter server.  It was actually capped at 2GB!  Not sure why this would have ever been set in this environment but it has been addressed and resources look great again.  Java hates being low on physical RAM.  CPU is still getting hammered on by SQL but total commit on memory is back down to less than 4GB.

References:

Installing vCenter Server 5.0 best practices
http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2003790

Upgrading to vCenter Server 5.0 best practices
http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&popup=true&languageId=&externalID=2003866

Minimum requirements for the VMware vCenter Server 5.x Appliance
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2005086

vSphere 5 ‘s new services in vCenter
http://geeksilver.wordpress.com/2011/08/31/vsphere-5-s-new-services-in-vcenter/

Update management service at 100% CPu after patching to 4.1.0 build 345043
http://communities.vmware.com/thread/306584

VCAP-DCA: now I wait...

And now I wait.  I think the test went better than I had expected, but I would still be surprised if I passed.  Overall it was a great test and I actually had fun doing it.  I am very much a hands on learner and I work best when I am told to just get it done, and that is 100% the nature of this test.  Everything was very relavent to real life with no real abstract scenarios or trick questions that I could see.  My test was 34 questions/scenarios and in the end I counted about 6 total that I knew I had missed, didn't finish, or didn't even start.  Even though 3.5 hours seems like a lot of time, it really isn't.  You either know it or you don't and if you don't, move on and come back if time permits.  Having VMware documents available for reference was nice, but you cannot rely on them due to the time limitations.

Regardless of if I passed or not, I learned a lot in studying for this and will be able to apply everything to my job role.

I wanted to give a shoutout to the following people who's blogs about their experiences, their study notes, or just generally good articles helped get me to a point where I felt more confident and ready to try:

Sean Crookston:  http://www.seancrookston.com/vcap-dca/
David Davis:  http://www.trainsignal.com/blog/vcap-dca-exam
Edward Grigson:  http://www.vexperienced.co.uk/tag/vcap-dca/
Kendrick Coleman:  http://www.kendrickcoleman.com/index.php?/Tech-Blog/vcap-datacenter-administration-exam-landing-page-vdca410.html
Eric Sloof:  http://www.ntpro.nl/blog/
"Virtual Vargi":  http://virtual-vargi.blogspot.com/p/vcap-dca.html
Duncan Epping:  http://www.yellow-bricks.com/2010/06/28/vcap-dca-exam/
Ray Heffer:  http://www.rayheffer.com/1861/vcap-dca-results-and-exam-experience-i-passed/

And everyone who has posted their experiences on the VMware Community Forums:  http://www.google.com/search?hl=en&biw=1304&bih=656&q=+site:communities.vmware.com+vcap-dca+passed

Hopefully within the next couple of weeks I will be able to post that I passed!  And if not, I will post that I am scheduling my re-take!

Also wanted to say thanks to Brad Butler and Acropolis for rolling the dice with me :)

VCDA410 Tomorrow! A LOT nervous!

From what I am reading, very smart people are saying this is the hardest test they have ever had to take for an IT certification.  There are apparently about 40 exam questions, all of which require configuration in a real lab environment, and there are ONLY 4 hours to do it.  4 hours seems like a lot right?  I am starting to sweat bullets here because doing the math on that, it is about 6 minutes per question, which means I will be expected to just get it done and move on.  This is not my style.  I tend to analyze, configure, re-analyze, and then re-analyze again.  And the worst part, I won't know my results for up to 2 weeks after.  Again, not my style!  I like instant gratification!  Agh!!  Back to studying.

Exam prep for VCAP-DCA

I really wish that I had an unlimited budget of time and money.  Unfortunately, I have neither so I study when I can and work with what I have.

I've been preparing the VMware Certified Advanced Professional Data Center Administrator certification exam and wow, it is daunting.  I've been working with VMware for a few years now and I got my VCP back in March of last year.  I manage a small data center and have been fortunate enough to get my hands on a lot of really cool technology that I thought I knew quite a bit about.  But in studying for this exam, I realize just how much I still need to learn.  I will be taking it on Friday, fully expecting at this point to fail, but it has opened my eyes so much during my studies that it will only make me better at what I do.

I have to applaud VMware for putting together a fantastic "exam blueprint" for this exam.  It is loaded with great references.  Also, I found several people who have done a great job at posting their study notes and taking deep dives into specific areas.  I am planning on doing the same once I get done with the actual test.

I'll post again on Friday after I take it but I will not know the results until up to 10 days after since it is a lab test and the results will be reviewed by a human being.  If you have any interest in specific objectives, please let me know and I'll do what I can to help.  Wish me luck!

Sorry Microsoft but I am falling for Google again (part 2)

I started looking at Google's services a little more after I found that the two big complaints I had back in the day seem resolved at this point and things are looking better than ever.  I found a way to get OneNote synced up to Google Docs using a tool called DAV-pocket and I can now access all the services that I use through my Google Apps account!  Life is great!  I will still have two separate accounts, but they can now be linked so I can easily switch between the two.  This is fine for now and I will eventually be consolidating them, but I am happier than a tornado in a trailer park (yes, I've seen Cars too many times :)).  Anyway, I'm going to slowly start the journey back to Google Apps tonight for at least a few things.  I may leave it split between Live and Apps for a few reasons:

#1 SkyDrive:  Hey, it's 25GB of free storage!  I can't pass that up.

#2 Office Web Apps:  I still use this quite a bit at work so I don't see this going away.  The whole Microsoft Gold Partner thing you know.

#3 My Wife:  If I tell her she is going to have to change her email settings again she will cut off my .... dinner.  What?!  She can be ruthless but come on, what did you think?

Anyway, I may be having this same debate again in a couple of months when Apple officially releases its iCloud service offering.  I do very much like my iPhone.

Maybe the smartest thing for me to do would be focus on developing a unified portal to all of these free public clouds so I could use them all for what they are best at.  Hmmm... just what I need, another shiny thing to catch my attention.  Stupid ADD.  Oh well, gotta stay busy somehow!  (just a hint of sarcasm there in case you couldn't tell)