Let me focus on VCAI, this is the most impressive one for me. View Composer API for Array Integration enables View administrators to create virtual desktops using NetApp FlexClone technology directly from the View Composer. This feature is using VAAI – vStorage API for Array Integration – to offload the clone creation to the storage array. Creating virtual desktops with FlexClone is not a new thing, it exists in the Virtual Storage Console (NetApp vCenter plugin) for a long time, having integration with the mostly used connection brokers. Cormac describes the benefits offloading the clone creation in his article, so you can save CPU on the ESXi hosts and also network bandwidth. Please note, this feature is a Technology Preview in VMware View 5.1.

If you want to see how is it working, you can check out this demo on NetAppTube:

Make sure you switch to 720p mode and please note the following things while you are watching the wizard process of creating the virtual desktops:

• Datastore capacity before cloning (0:15)
• Datastore contents before cloning (0:41)
• Datastore and network performance (3:25 – 5:37)
• Datastore capacity after cloning (6:59)
• Datastore contents after cloning (7:08)

I realized later it’s something new. When I first blogged about the Rapid Cloning Utility, I also ran into an error. If you check that post, RCU started without checking the partition alignment, and if it wasn’t aligned properly, it failed. Actually the error message was not useful in the vSphere client, I had to check the console log. Now it changed, Rapid Cloning Utility checks the source before starting the wizard:

It doesn’t look like such a big deal, but it’s very important. Now the Rapid Cloning is successful even if the source is not aligned, and if you create 1000 non aligned Virtual Machines, that’s not funny. If you see this error message, choose no and go ahead, download the appropriate host utility to allign the VM. To find it just click on the NetApp logo on the vSphere client home screen, click on ‘Tools’, and choose the appropriate host type.

You can download and see the full list of changes on the NOW page.

Documentation:

• Release notes
• Installation and Administration Guide
• Provisioning and Cloning Administration Guide
• Backup and Recovery Administration Guide

I’ll test the old and new features in the near future, and post my experiences here.

The rapid cloning capabilities provided by the Provisioning and Cloning feature require a NetApp FlexClone license on the storage array where clones are being created.

Of course the free dedupe license is also needed, but I can’t imagine any NetApp box without that. Let’s see the steps of the clone creation. After you push the ‘finish’ button at the end of the RCU wizard, the following things will happen:

1. Initial copy of image: this step will be different in the examples below. RCU doesn’t clone the original disk image, it creates an initial copy first
2. FlexClone virtual disk: this is where the magic happens. RCU creates the number of clones of the virtual disk we requested without copying the data, only using NetApp block sharing technology. In the examples below it’s 10 clones and took less than 4 minutes.
3. Clone virtual machine: in this step RCU clones the VM without the disk.
4. Reconfigure virtual machine: The VM which was cloned in the step 3. gets a virtual disk from step 2.

The last two steps are repeating till we get the number of VMs we configured. That’s it, easy and effective, let’s see the differences. For testing I initially have a single Windows 2008 VM on my DataStore with size of ~8GB.

1. NFS volume, dedupe enabled, VM deduped

The screenshot above shows the process, the four steps described. The volume looks like this before we running RCU:

3210b> df -hs /vol/nfsb
Filesystem                used      saved       %saved
/vol/nfsb/              5995MB     2179MB          27%

The VM is already deduplicated, there are some savings on the volume. The tool will physically copy the VM in this case, and after the creation of 10 clones:

3210b> df -hs /vol/nfsb
Filesystem                used      saved       %saved
/vol/nfsb/                14GB       73GB          84%

Basically at the end of the test we have 10 new virtual machines for the ‘cost’ of one.

2. NFS volume, dedupe enabled, VM not deduped

This scenario could happen if you have an NFS DataStore with dedupe, but the VM is just created, scheduled dedupe did not run or disabled. In this case df shows this:

3210b> df -hs /vol/nfsb
Filesystem                used      saved       %saved
/vol/nfsb/              8170MB        0MB           0%

The test is the same, we create 10 clones, and finally we’ll have this:

3210b> df -hs /vol/nfsb
Filesystem                used      saved       %saved
/vol/nfsb/              8496MB       79GB          91%

Wow. In this case the initial copy is using the block sharing technology too!

3. NFS volume, dedupe disabled

For this test, I manually disabled deduplication on the volume:

3210a> sis status /vol/nfsa
Path                           State      Status     Progress
/vol/nfsa                      Disabled   Idle       Idle for 00:15:17

The provisioned VM is about the same as last time:

3210a> df -hs /vol/nfsa
Filesystem                used      saved       %saved
/vol/nfsa/              8078MB        0MB           0%
3210a>

…and it looks about the same after cloning:

3210a> df -hs /vol/nfsa
Filesystem                used      saved       %saved
/vol/nfsa/              8401MB       78GB          91%

We can say, RCU does not care if dedupe is enabled or not.

4. the VMFS case

With VMFS (I’ve tested with FC but I’m sure it’s the same with iSCSI) the initial copy and the cloning copies are done with normal copies. Of course, if the array (DataONTAP) supports VAAI the copy is HW accelerated, but it’s slower. Takes longer, but it still has the advantage to make a lot of clones in one step. For me, it’s one more reason to use NFS in VMware vSphere environment.

So many bloggers filed a lot of posts about VMworld session voting. I’m late with this, but still there’s a little time to vote. I’ve checked the session list several times, started from the top and started from the bottom also. There’s no chance to read everything, I think it’s not possible, at least for me. We are getting closer to the end of the voting time and I have to say, I have a session proposal there.

Why did I submit? Why not? I picked up my favorite topic and created the proposal. Here you are:

1741: One Year After – VAAI: Who Did What and How?

With vSphere 4.1 VMware announced vStorage API for Array Integration. With this API it’s possible to offload some functions from the host/SAN layer to the storage SP/CPU layer. After the announcement storage vendors started to develop their solutions. With this presentation I’ll show the well known storage vendors and what they implemented. This would be a multivendor presentation, which covers the implemented functions, time of the release or the planned announcement. Also I’ll present the performance benefits if it’s available from the vendor or bloggers (as a partner engineer I have access to most of the boxes to test, and there are very good test numbers out on various blog sites – I did some tests also). This API is a very important integration point and should be implemented by all storage vendor. It’s also very important for VMware customers, we should help them how to select vendor/product and very important for VMware partners (service providers, solution providers). The performance section shows real world numbers from tests, and shows the offload operation and its effects. It will show the process how increase the CPU load on the storage side and how decrease the SAN load and the host (ESX/ESXi) side. It also covers design questions because with this feature the vSphere environment relies more on the storage architecture. The presentation will also show unique developments, where vendors where more creative (for example 3PARs VMFS metadata comparsion acceleration or HDS VAAI function inherit for virtualized arrays). It will cover also the future of VAAI and update from vSphere 5 if it’s available at the time of the presentation.

I’m not a naive person, I know very well I have no real chance, but at least I have a blog post about my session proposal at the public voting level. There are a lot of big names out there, of course I have my favorites. Did you vote? No? Go for it!

As I wrote in my last post, this week I was lucky to test a new NetApp FAS3210 system which actually had Flash Cache cards installed. So I have the same piece of HW:

• Cisco UCS B200 blade system with two X5550 sockets and 48GB of RAM
• Cisco Nexus 5010 with 4Gb FC connection to the NetApp box
• NetApp FAS3210 with 24×300 SAS 15krpm disks. For this test I’ve created the same aggregate of 9 disks.
• Flash Cache (PAM II) card, 256GB per controller

For this test, I’ve cloned 25 Windows 2008R2 Standard Server virtual machines to create a boot storm. Unfortunately I wasn’t able to create more, it would be a big fun to repeat this test with a couple of hundreds. I booted up the VMs first with disabled Flash Cache. I’ve recorded the performance numbers at the storage controller with ‘sysstat -f 1′. I repeated the boot after turning on the Flash Cache. Here are the charts I’ve created using the sysstat outputs:

• The horizontal axis shows the seconds
• Red line shows the enabled and the blue line shows the disabled state

1. CPU usage

2. FCP operations

3. Disk read kB

4. FCP output kB

It took around 55 seconds with and 120 seconds without the cache to serv the boot data. These charts shows the difference very well I think. Please note, there are several ways to optimize Flash Cache efficiency, take a look at this discussion at NetApp Community site, covers a few Technical Reports also regarding the use of Flash Cache.

I have the same piece of HW:

• Cisco UCS B200 blade system with two X5550 sockets and 48GB of RAM
• Cisco Nexus 5010 with 4Gb FC connection to the NetApp box

New storage box:

• NetApp FAS3210 with 24×300 SAS 15krpm disks. For this test I’ve created the same aggregate of 9 disks.

I went throught the same steps with VAAI enabled and disabled configurations:

• add a new Hard Disk to the VM, 100GB, thick, cluster supported (zeroed)
• clone the VM (with the added disk) within the same LUN
• clone the VM to another LUN
• Storage VMotion the VM

Just for the easy comparsion I show the FAS2040 numbers also, not just the FAS3210 ones.

FAS2040:

FAS3210:

Let’s see VAAI operations’ time requirement compared to legacy operations

Storage VMotion with the FAS3210 gives the best result.

Let’s take a look at the graphs what happened on the controller.

1. VMDK addition

With VAAI enabled the CPU load on the storage controller were 88% and from the host side there were a 80ms command latency.

With disabled VAAI the CPU load on the storage controller was less, around 70%. The next graph shows the write rate on the ESXi host.

2. Clone within the same LUN

The CPU usage on the storage controller were about 75%, there were around 85% command latency and 25ms physical device read latency on the ESXi side.

With disabled VAAI the storage controller made 5500 operations/sec with 56% CPU load:

The ESXi host shows about 165000 KBps with 5-7ms latency:

3. Clone between LUNs

No real difference. With VAAI the storage CPU usage were about 70%, without VAAI it was around 57% (with the same Read/Write rates)

4. Storage VMotion

With VAAI, storage CPU load were 69%:

Disabled VAAI made 48% storage CPU load with 3.9k operations/sec and 142000 KBps Read/Write rate on the ESXi:

As the previous posts already shown, the storage controller were the bottleneck. If you have stronger CPU on the storage side you’ll get higher performance gain with VAAI operations.

]]> http://tt.yyaazz.hu/2011/04/19/netapp-fas3210-vmware-vaai-performance-test/feed/ 0 yyaazz Screen shot 2011-04-18 at 5.56.34 AM Screen shot 2011-04-18 at 6.59.11 AM Screen shot 2011-04-18 at 6.12.19 AM Screen shot 2011-04-18 at 7.09.47 AM Screen shot 2011-04-18 at 7.21.18 AM Screen shot 2011-04-18 at 6.47.39 AM Screen shot 2011-04-18 at 7.57.47 AM http://tt.yyaazz.hu/2011/03/12/vsphere-clone-vs-netapp-rapid-clone/ http://tt.yyaazz.hu/2011/03/12/vsphere-clone-vs-netapp-rapid-clone/#comments Fri, 11 Mar 2011 23:02:32 +0000 yyaazz http://tt.yyaazz.hu/?p=275 ]]> In a typical NetApp & vSphere environment there are two well-known options to create a clone of a Virtual Machine. The first one is the vSphere default and the second is the NetApp rapid clone. The vSphere default clone is a simple copy operation from filesystem point of view, so if you browse the datastore you’ll see the same files in the source and target VM’s directory, except for the log files. The NetApp rapid cloning utility is doing the things in an other way, we’ll see how. But first of all, let’s talk about the configuration.

• NetApp FAS2040, 500GB NFS datastore (standard 1Gbit onboard port)
• vSphere 4.1 with NetApp Virtual Storage Console 2.0.1
• Windows 2008R2 VM, contains a ~7GB VMDK

Let’s see first how to use Rapid Cloning Utility. The wizard starts from the GUI (right-click on the source VM):

The following 3 parameters have to be selected within the wizard

• NetApp controller
• target, where the clone should be placed
• provisioning format (thick, thin, same)

The next screen is the most important part of the cloning configuration:

We have some options to fine-tune the cloning. Set the name base, numbers, increasing… The interesting part just starts when the finish button pushed on the summary screen. Let’s see what’s happening on the controller:

ntap>

Fri Mar 11 06:01:38 GMT [app.log.info:info]: vcenter: Provisioning and Cloning Capability 3.1.1: (0)  : Provisioning and Cloning Capability (create) v3.1.1: vcUserId: Administrator VMware Version:4.1.0 Build: 258902 API: 4.1

Fri Mar 11 06:01:39 GMT [app.log.info:info]: vcenter: Provisioning and Cloning Capability 3.1.1: (0)  : gold=null,aggrOrVol=null,proto=null,DsNames=null,vmFile=Configuration File

Fri Mar 11 06:01:39 GMT [app.log.info:info]: vcenter: Provisioning and Cloning Capability 3.1.1: (0)  : gold=null,aggrOrVol=null,proto=null,DsNames=null,vmFile=[flexvol] clone1/clone1.vmdk

Fri Mar 11 06:01:39 GMT [app.log.info:info]: vcenter: Provisioning and Cloning Capability 3.1.1: (0)  : vms=1,vmsPerDatastore=0,vcVersion=4.1.0,rcuGroup=ntap_rcu1299823290303

Fri Mar 11 06:01:39 GMT [app.log.info:info]: vcenter: Provisioning and Cloning Capability 3.1.1: (0)  : hostInfo,version=VERSION_4_0

Fri Mar 11 06:01:42 GMT [wafl.snap.delete:info]: Snapshot copy dense_clone.0.ca2ba564-4d35-11e0-be34-00a0982880ba on volume flexvol NetApp was deleted by the Data ONTAP function dense_clone_delete_snapshot. The unique ID for this Snapshot copy is (2, 48).

Fri Mar 11 06:01:42 GMT [dense.clone.aborted:info]: Clone operation on file ‘/vol/flexvol/ntap_rcu1299823290303/0000hd0000-flat.vmdk’ was aborted successfully. The clone operation ID was 20.

OK-OK, blah-blah, but look this:

Ndmpcopy: Starting copy [ 2 ] …

Ndmpcopy: ntap: Notify: Connection established

Ndmpcopy: ntap: Notify: Connection established

Ndmpcopy: ntap: Connect: Authentication successful

Ndmpcopy: ntap: Connect: Authentication successful

Ndmpcopy: ntap: Log: DUMP: creating “/vol/flexvol/../snapshot_for_backup.4″ snapshot.

Ndmpcopy: ntap: Log: DUMP: Using Partial Volume Dump of selected subtrees

Ndmpcopy: ntap: Log: DUMP: Date of this level 0 dump: Fri Mar 11 06:01:44 2011.

Ndmpcopy: ntap: Log: DUMP: Date of last level 0 dump: the epoch.

Ndmpcopy: ntap: Log: DUMP: Dumping /vol/flexvol/clone1/clone1-flat.vmdk to NDMP connection

Ndmpcopy: ntap: Log: DUMP: mapping (Pass I)[regular files]

Ndmpcopy: ntap: Log: DUMP: Reading file names from NDMP.

Ndmpcopy: ntap: Log: DUMP: mapping (Pass II)[directories]

Ndmpcopy: ntap: Log: DUMP: estimated 7010605 KB.

Ndmpcopy: ntap: Log: DUMP: dumping (Pass III) [directories]

Ndmpcopy: ntap: Log: DUMP: dumping (Pass IV) [regular files]

Ndmpcopy: ntap: Log: RESTORE: Fri Mar 11 06:01:46 2011: Begin level 0 restore

Ndmpcopy: ntap: Log: RESTORE: Fri Mar 11 06:01:46 2011: Reading directories from the backup

Ndmpcopy: ntap: Log: RESTORE: Fri Mar 11 06:01:46 2011: Creating files and directories.

Ndmpcopy: ntap: Log: RESTORE: Fri Mar 11 06:01:46 2011: Writing data to files.

Ndmpcopy: ntap: Log: DUMP: dumping (Pass V) [ACLs]

Ndmpcopy: ntap: Log: RESTORE: Fri Mar 11 06:03:06 2011: Restoring NT ACLs.

Ndmpcopy: ntap: Log: RESTORE: RESTORE IS DONE

Ndmpcopy: ntap: Log: RESTORE: The destination path is /vol/flexvol/ntap_rcu1299823290303/.tmpNdmpDir/

Ndmpcopy: ntap: Log: DUMP: 6996887 KB

Ndmpcopy: ntap: Log: DUMP: DUMP IS DONE

Ndmpcopy: ntap: Log: DUMP: Deleting “/vol/flexvol/../snapshot_for_backup.4″ snapshot.

Ndmpcopy: ntap: Notify: restore successful

Ndmpcopy: ntap: Notify: dump successful

Ndmpcopy: Transfer successful [ 1 minutes 28 seconds ]

Ndmpcopy: Done

Fri Mar 11 06:03:40 GMT [app.log.info:info]: vcenter: Provisioning and Cloning Capability 3.1.1: (0)  : Finished work, cleaning up temporary directories.

Fri Mar 11 06:03:46 GMT [app.log.info:info]: vcenter: Provisioning and Cloning Capability 3.1.1: (0)  : Finished executing.

So, the rapid cloning is basically an NDMP dump/restore operation. It looks the following in vSphere:

Let’s see some interesting info.

Time

On a normal 1Gbit connection with NFS datastore (no VAAI here) the vSphere clone took 7 minutes while the RCU only 2 minutes. On the other hand, rapid cloning has the ability to create more then one clone in one step. For example 5 clone creation will look like this:

….

Ndmpcopy: ntap: Notify: restore successful

Ndmpcopy: ntap: Notify: dump successful

Ndmpcopy: Transfer successful [ 1 minutes 30 seconds ]

Ndmpcopy: Done

Fri Mar 11 06:28:42 GMT [dense.clone.finish:info]: Clone operation on file ‘/vol/flexvol/ntap_rcu1299824793374/0001hd0000-flat.vmdk’ completed successfully. The clone operation ID was 23.

Fri Mar 11 06:28:46 GMT [dense.clone.finish:info]: Clone operation on file ‘/vol/flexvol/ntap_rcu1299824793374/0002hd0000-flat.vmdk’ completed successfully. The clone operation ID was 24.

Fri Mar 11 06:28:50 GMT [dense.clone.finish:info]: Clone operation on file ‘/vol/flexvol/ntap_rcu1299824793374/0003hd0000-flat.vmdk’ completed successfully. The clone operation ID was 25.

Fri Mar 11 06:28:54 GMT [dense.clone.finish:info]: Clone operation on file ‘/vol/flexvol/ntap_rcu1299824793374/0004hd0000-flat.vmdk’ completed successfully. The clone operation ID was 26.

Fri Mar 11 06:29:54 GMT [app.log.info:info]: vcenter: Provisioning and Cloning Capability 3.1.1: (0)  : Finished work, cleaning up temporary directories.

Fri Mar 11 06:29:59 GMT [app.log.info:info]: vcenter: Provisioning and Cloning Capability 3.1.1: (0)  : Finished executing.

Skipped the dump/restore part, as it’s the same as the previous example, but after the successful restore another 4 vmdk has been created. How long did it take? Around 2.5 minutes. Not that much, huh? For ten clones, around 3-3.5 minutes, and so on.

Space

At the beginning, with one VM:

ntap> df -hs

Filesystem                used      saved       %saved

/vol/flexvol/           6801MB        0MB           0%

dedupe, just for fun…

ntap> df -hs /vol/flexvol/

Filesystem                used      saved       %saved

/vol/flexvol/           5079MB     1711MB          25%

create a vSphere clone:

ntap> df -hs /vol/flexvol/

Filesystem                used      saved       %saved

/vol/flexvol/             11GB     1711MB          13%

create a rapid clone:

ntap> df -hs /vol/flexvol/

Filesystem                used      saved       %saved

/vol/flexvol/             18GB     1711MB           8%

one more…

ntap> df -hs /vol/flexvol/

Filesystem                used      saved       %saved

/vol/flexvol/             24GB     1711MB           6%

Both clones took around 7GB above. Not surprising, fun starts here. Let’s create 5 clones:

ntap> df -hs /vol/flexvol/

Filesystem                used      saved       %saved

/vol/flexvol/             31GB       28GB          47%

wow. 7GB again. Let’s create 10:

ntap> df -hs /vol/flexvol/

Filesystem                used      saved       %saved

/vol/flexvol/             38GB       28GB          42%

7GB again. hmm, one more time

ntap> df -hs /vol/flexvol/

Filesystem                used      saved       %saved

/vol/flexvol/             44GB       87GB          66%

OK, turn off dedupe operation, clone 10 again:

ntap> sis off /vol/flexvol

SIS for “/vol/flexvol” is disabled.

ntap> sis status /vol/flexvol

Path                           State      Status     Progress

/vol/flexvol                   Disabled   Idle       Idle for 00:54:57

ntap> df -hs /vol/flexvol/

Filesystem                used      saved       %saved

/vol/flexvol/             51GB      146GB          74%

7GB again and the %saved is growing. Summary? RCU creates deduped, space efficient clones very fast. That’s all.

I have the same piece of HW except for the connection:

• Cisco UCS B200 blade system with two X5550 sockets and 48GB of RAM
• NetApp FAS2040 with 12×300 FC 15krpm disks. For this test I’ve created an aggregate of 9 disks.

In the last test I had 4Gb Fibre Channel connection and now I have simple 1Gb iSCSI. I repeated the same steps, the only difference is the size of the VMDK I’ve added to the Virtual Machine:

• add a new Hard Disk to the VM, 50GB (was 100GB in the FC post), thick, cluster supported (zeroed)
• clone the VM (with the added disk) within the same LUN
• clone the VM to another LUN
• Storage VMotion the VM

It’s not a surprise, the trend is the same.

The tipical VAAI operation looks like this on the storage controller:

The CPU load is almost 100% and the Operation (ops/sec) is near zero. With disabled VAAI looks different: 

The CPU usage is 80%, and the chart above shows around 900-1000 Ops/sec iSCSI operations. This is for the simple write (zeroed VMDK creation), for clone and Storage VMotion looks a bit different:

The CPU load is the same, but the iSCSI operation is more like ~2000 Ops/sec. Let’s see what’s happening on the ESXi host.

With VAAI enabled, there’s no write or read rate (as there’s no read or write from the host side), but the charts shows latency around 8-10ms. With disabled VAAI the chart looks a bit different. For the VMDK creation the write rate is around 100000KBps with 160ms latency (write only, no reads). The read/write operation shows 70000KBps IO rate with 10-15ms latency.

Update: Yellow-Bricks author Duncan Epping was kind enough to include this post in his “VAAI sweetness” article with other examples. Must read!

About the configuration:

• Cisco UCS B200 blade system with two X5550 sockets and 48GB of RAM
• NetApp FAS2040 with 12×300 FC 15krpm disks. For this test I’ve created an aggregate of 9 disks.
• Cisco Nexus 5010 with 4Gb FC connection to the NetApp box

I went throught the same steps with VAAI enabled and disabled configurations. At the beginning I had a Windows VM with 40GB hard disk.

• add a new Hard Disk to the VM, 100GB, thick, cluster supported (zeroed)
• clone the VM (with the added disk) within the same LUN
• clone the VM to another LUN
• Storage VMotion the VM

Let’s see the results:

1. VMDK addition

Without VAAI: 10:30

With VAAI: 9:46

From the storage controller side the CPU usage were almost the same. With VAAI the CPU usage were 100% without that were about 95-100%, no real difference. On the other hand, with VAAI there’s no load on the host side nor on the SAN. You can see the following screenshots I made in the vSphere client when the VAAI were turned off.

So the zeroed VMDK creation, which without VAAI means 100GB zero write to the disk made some latency and some traffic for the SAN. With VAAI these graphs were near zero.

2. Clone within the same LUN

Without VAAI: 19:10

With VAAI: 15:41

Looking at the graphs the VAAI operation looks exactly the same. It took 100% CPU usage on the storage side and nothing on the host side:

As the graph above shows, the storage CPU is running at 100% and basically there’s no FC operations.

Without VAAI as you see above the CPU load were about 80-90% and there were about 3000 Ops/sec on the fibre channel adapter. The performance charts of vSphere:

3. Clone between LUNs

Without VAAI: 19:03

With VAAI: 15:37

As predictable with VAAI both the host side and the SAN side has no load and the storage CPU is working at 100%.

As my configuration has only one aggregate and raid group, cloning between LUNs means the same disks, so the graphs looks the same:

The only difference is the read latency.

4. Storage VMotion

Without VAAI: 20:26

With VAAI: 17:23

However the completed time differs a little bit from the last cloning, the graphs looks exacty the same.

I think this few examples show well how vStorage API for Array Integration moves the workload from the host and SAN side to the storage CPU. I’ve made these tests via FC connection, I’ll repeat these with iSCSI.

iSCSI performance test post is here.