thumperthoughts » technology

Resizing NTFS Volumes in VMWare

arielnh56 — Wed, 24 Feb 2010 06:44:18 +0000

VMWare allows for some tricks for resizing volumes which are particularly useful for Windows NTFS.
We have been doing Physical to Virtual (P2V) imports of a number of our older windows systems recently. We imported a number of these with the original disk partitioning which had a paltry 4GB C: drive configured. Of course we now find we need to grow this, and we have the same limitation we had on the physical machine – the D: partition is in the way.
What we should have done while importing was to check the box tell VMWare to create a separate virtual disk for each volume. Too late now – and resorting to traditional partition manipulation tools would be painful.
No worries though – VMWare will let you import an existing virtual machine. So do the import again, using the VM as a source. Check the box this time and re-size the volumes at the same time. When the import completes, power up the new VM and delete the old one. Now that the volumes are on separate virtual disks future growth will be much easier.
Another trick VMWare makes possible is easy extension of the C: drive NTFS filesystem – windows generally won’t let you do that on a running system.

Power down the VM
Go to another windows VM and add a hard disk
Select “use an existing disk” and browse the data store for the VMDK you want to fix.
Use the normal windows storage management tools to expand the filesystem.
Remove the disk from the second VM.
Power up your modified VM and Bob’s your Uncle.

Uptime Humor

arielnh56 — Mon, 22 Feb 2010 18:49:23 +0000

Devotion To Duty

http://xkcd.com/705/

GoDaddy (Virtuozzo) Virtual Server in cacti

arielnh56 — Sat, 13 Feb 2010 21:46:02 +0000

A business site I provide some spport to runs on a GoDaddy virtual Linux server. We were having some performance issues so I dug into the available data and found we were hitting our memory limits. It seems the Virtuozzo system used by GoDaddy has hard and soft resource limits – you can go over the soft limits for short periods but after a while bad things start to happen. Like Virtuozzo shooting your processes in the head. As the applications and OS think they have the hard limit available, staying below the soft limit is problematic.

Virtuozzo makes status information in /proc/user_beancounters. The different fields are pretty cryptic – their meanings are described in detail here.

The attached zip file contains two cacti template exported from Cacti version Version 0.8.7b, a snippet to add to the snmpd.conf on the server and a one line shell script called readbeans which is called by the snmpd agent. Deploy these latter items on your GoDaddy VM and suck the templates into Cacti. This will give you a “Virtuozzo Beancounters” data query which you can add to your device which should give you access to 20 graphs (don’t check resource, that one is bogus).

I found that building a compound graph like this brought together the most interesting values:

This long term graph shows where I implemented the crude but effective solution of restarting various system processes nightly (snmpd, httpd, mysqld, named and Plesk), some of which were gradual memory hogs.

Download this template

NetApp Consistency Points in Cacti

arielnh56 — Thu, 11 Feb 2010 08:37:14 +0000

Recently went on a couple of training courses on NetApp fundamentals and performance. The tutors both recommended that every good NetApp admin should spend a few minutes every day looking at sysstat -x 1 scrolling up their screen. Of particular interest from a performance standpoint is consistency point activity. Consistency points are a checkpoint in time in the write cycle of the WAFL filesystem, and happen at any time OnTAP thinks it needs one, or every 10 seconds otherwise.

Gazing at sysstat is sound advice, but sounds like a job for a robot or a computer to me. Make the machines do the work is one of my maxims.

So I conjured up this cacti graph in class which will watch consistency point activity all the time. So far this has caught some very interesting middle-of-the-night activity, some of which coincided with application performance problems.

This sample graph shows the normal green of timer CPs from a relatively idle system, with periodic blue spikes triggered by the automatic hourly snapshot activity. The big flesh colored blob in the middle is a large copy operation that occurs as part of a legacy database backup process that we have not yet re-engineered to take advantage of snapshots. The right hand end of the graph shows what happens when doing a large myISAM to InnoDB database conversion – more on this when that project is completed. I’ve deliberately made the back-to-back and deferred-back-to-back CPs red as they are evil. The only time I have seen these is when a RAID group was reconstructing after the NetApp pulled a disk out for testing – performance really sucked that day for our gigantic myISAM tables. Most of our normal high activity periods are characterized by High Water CPs, though I have also seen occasional pink tinges of Log Full.

Here are what the different CP types mean, along with the corresponding letter prefix you will observe in sysstat and SNMP MIB name:

Back to Back (B – cpFromCpOps) If you are getting these, your filer is overloaded. The write traffic is coming in faster than it can be written and the filer is running out of NVRAM capacity in one bank before the data in the other bank can be written. We got some of these during a RAID reconstruct. Bad news.
Deferred Back to Back (b – cpFromCpDeferredOps). these are a worse version of the above. Definitely an overload problem.
Timer (T – cpFromTimerOps) 10 seconds since the last CP – it means the system is pretty idle write-wise. Note that lightly writing to a NetApp may lead to more fragmentation than writing a bit more at a time, as it does not get the opportunity to stack stripes. Exercise is good.
Flush (U – cpFromFlushOps) a flush happened. I’m sure it cleared something away.
High Water (H – cpFromHighWaterOps) the number of RAM buffers holding modified data exceeded a threshold. Heavy write activity.
Log Full (F – cpFromLogFullOps) the current NVRAM bank is full and the system switches to the other one and starts writing this to disk. Very heavy write activity.
Low Water (L – cpFromLowWaterOps) the number of available RAM buffers dropped below a threshold.
Low Datavec (D – cpFromLowDatavecsOps) the number of available datavecs (data vectors?) dropped below a threshold.
Low Vbuf (V – cpFromLowVbufOps) the number of available virtual buffers dropped below a threshold.
Snapshot (S – cpFromSnapshotOps) snapshots happen. It is a NetApp. Usually triggers a bunch of Sync (Z) CPs.
Sync (Z – cpFromSyncOps) an internal sync – usually preparing to complete a snapshot.

This template was exported from Cacti Version 0.8.7e

Dowload it here.

EMC Networker NDMP index oddity

arielnh56 — Sun, 31 Jan 2010 01:08:35 +0000

We have had a real headscratcher with our backups of late. The setup has the tape jukebox connected to the dual-head NetApp 3170 as an NDMP device. EMC (formerly Legato) Networker runs on a CentOS Linux box and directs the NDMP jobs on the filer. This worked just fine for months, backing up multiple volumes. Late last year we installed a new Microsoft Exchange and Enterprise Vault infrastructure with the multiple volumes exported as iSCSI LUNs to the Windows servers. This also worked fine until we configured the backup jobs.
Whatever we did the two volumes hosting the I: drives for the Vault servers just would not back up. The error message essentially said there was no backup device (tape drive) available, which was nonsense as other similar volumes in the same group were backing up just dandy.
EMC scratched their heads too.
Eventually, inspired by the Sherlock Holmes maxim that “if you eliminate the impossible, whatever is left, however improbable, must be the truth” I started mucking around with volume names. You see the I: drives hold the indexes for the Vault mailbox indexes, so we helpfully named them evmb01_indexes and evmb02_indexes. No other volumes had the string “index” embedded in their name, and that was all that differentiated them from the volumes that could find the tape drive. Improbable but true: Networker will not let you backup a volume with “index” in its name to an NDMP device.

Following up with EMC this is a deliberate feature, apparently put in place to make sure you do not back up your NDMP indexes to an NDMP device. This makes sense, as the scanner utility cannot be used on NDMP volumes to retrieve indexes as it does on native Networker volumes, leaving you with a Catch-22 in the event of a site recovery. The fact that they assume that NDMP index volumes shall always have “index” in their names, and no other volumes ever will is just false logic.

In the end we renamed the volumes with the more grammatically correct “indices” and called it a day.

Blackberry wordpress client

arielnh56 — Sun, 31 Jan 2010 00:29:42 +0000

Finally loaded some updated blogging code to ye olde Powweb site. WordPress. Very nice and simple to use. Trying to post something from the Blackberry client … Now!