Hi Cluster Fans,

Resources can be hard to find, so every few months we will be publishing an updated list of useful documents, guides and information to this blog (http://blogs.msdn.com/clustering/). 

If there is any other useful Microsoft content which you feel is missing, let us know by clicking the 'email' link in the upper right corner of the page and send us the resource and URL.

Thanks,
Symon Perriman
Program Manager
Clustering & HA

Microsoft

Useful Sources

·         Blog: Cluster Team: http://blogs.msdn.com/clustering/

·         Blog: Ask Core: Clustering

·         Training: Course 6423a: Implementing and Managing WS08 Clustering

·         Website: Cluster Technical Resources

·         Website: Cluster Information Portal

·         Website: Clustering Forum (2008)

·         Website: Clustering Forum (2008 R2)

·         Website: Clustering Newsgroup

Windows Server 2008 R2

·         Blog Guide: Deploying Cluster Shared Volumes (CSV)

·         Blog Guide: Cluster Shared Volumes (CSV): Disk Ownership

·         Blog Guide: PowerShell for Network Load Balancing (NLB) in Windows Server 2008 R2

·         Blog Guide:  PowerShell for Failover Clustering in Windows Server 2008 R2

·         Blog Guide: Live Migration Traffic

·         Blog Guide: How to manually defrag or ChkDisk a CSV disk

·         TechNet Guide:Using Live Migration in Windows Server 2008 R2

·         TechNet Guide: What’s new in R2 Clustering

·         Webcast: Innovating High Availability with Cluster Shared Volumes (CSV)

·         Webcast: Failover Clustering Feature Roadmap in WS08 R2

·         Whitepaper:Windows Server 2008 R2 & Microsoft Hyper-V Server 2008 R2 - Hyper-V Live Migration Overview & Architecture

·         Webcast: Windows Server 2008 R2 Live Migration

·         Webcast: Clustering in a Virtual World

·         Website: Clustering Forum (2008 R2)

Architecture

·         Blog Guide: Cluster Virtual Adapter (NetFT)

·         Blog Guide: PlumbAllCrossSubnetRoutes

·         Whitepaper: Failover Cluster Architecture Overview

Core

·         Guide:Server Core

·         TechNet: Installation

·         Utility:Remote Server Administration Tools (simplifies Server Core configurations)

·         Webcast:How Microsoft does IT: Enhancing High Availability with Server Core in Windows Server 2008

Deployment / Migration / Upgrade

·         Blog Guide:Migration Options for Hardware

·         Blog Guide: PrintBRM Error 0x80070043 workaround

·         Blog Guide:DHCP Database migration to Windows Server 2008

·         Blog Guide:PrintBRM.exe, 0×80070043 and Print Clusters - A Workaround

·         KB Guide: Exchange 2003: Move Mailbox 

·         KB Article: SQL Server 2008 Rolling Upgrades

·         KB Article: Cluster Nodes as Domain Controllers (DCs)

·         MSDN Guide: SQL Upgrade Paths

·         MSDN Guide: SQL Cluster Upgrade

·         TechNet Guide: Migrating Cluster Settings

·         TechNet Guide: Failover Clustering Deployment

·         TechNet Guide: Validating Hardware for a Failover Cluster

·         TechNet Guide: Installing a Failover Cluster

·         TechNet Guide: Creating a Failover Cluster

·         TechNet Guide: Cluster Requirements

·         TechNet Guide: Validating a cluster

·         TechNet Guide: Configuring Accounts in Active Directory

·         TechNet Guide:Recommended Clustering Hotfixes (2003)

·         TechNet Guide:Recommended Clustering Hotfixes (2003 SP2)

·         TechNet Guide:Recommended Clustering Hotfixes (2008)

·         TechNet Guide: Exchange 2007 Overview

·         TechNet Guide: Exchange 2007 Cmdlets

·         TechNet Guide: Print Migration Overview

·         TechNet Guide: UI: Print Migration Tool/Wizard

·         Utility: File Server Migration Toolkit (FSMT) (2008)

• Webcast: TechNet Webcast: Failover Cluster Validation and Troubleshooting with Windows Server 2008  
• Website: File Server Migration Toolkit (2008) 
• Whitepaper: File Server Migration Toolkit (2008) 
• Whitepaper: Configuring and Troubleshooting AD CS (2008)

Exchange Server

·         Lab:TechNet Virtual Lab: Exchange Server 2007 Standby Continuous Replication

·         Lab:TechNet Virtual Lab: Using Cluster Continuous Replication (CCR) in Exchange 2007

·         TechNet:Installing Cluster Continuous Replication (CCR) on 2008

·         TechNet:Deploying Exchange 2003 in a Cluster

·         TechNet: Planning for Cluster Continuous Replication (CCR)

·         TechNet: Installing CCR on Windows Server 2008

·         TechNet: How to create an Exchange SCC Failover Cluster with CMD

·         Webcast:How Microsoft IT Implemented New Storage Designs for Exchange Server 2007

·         Webcast: Exchange 2007 High Availability Deep Dive

File Server

·         Blog Guide: File Share ‘Scoping’ in Windows Server 2008 Failover Clusters

·         Blog Guide: Share Subdirectories in Windows Server 2008

·         TechNet Guide: Configuring a Two-Node File Server Failover Cluster

·         TechNet Guide: Creating a Clustered File Server checklist

·         TechNet Guide: Create a Shared Folder in a Clustered File Server

·         WebCast:TechNet Webcast: Prepare Yourself for Windows Server 2008 (Part 5 of 8): New File Server Features

·         WebCast: How Microsoft IT Deploys Windows 2008 Clusters for File Services

·         Webcast: New File Server Features of Windows Server 2008 (Level 200)

Hyper-V

·         Blog Guide: Deploying a HA Virtual Machine (2008)

·         Blog Guide: HA Virtual Machine Deployment Considerations (2008)

·         Blog Guide: Network Load Balancing (NLB) and Virtual Machines

·         Blog Guide: Adding a Pass-Through Disk to a HA VM

·         Blog Guide: SCVMM: Intelligent Placement

·         Blog Guide: Monitor Network Traffic for a VM on a Cluster

·         TechNet Case Study:How Microsoft IT Designs the Virtualization Host & Network Infrastructure

·         TechNet Case Study: Best Practices for Deploying VMs using Hyper-V

·         TechNet Guide: Getting Started with Hyper-V

·         TechNet Guide: High-Availability for a Server Running Hyper-V

·         TechNet Guide: Design for a Failover Cluster in Which All Nodes Run Hyper-V

·         TechNet Guide: Requirements and Recommendations for Failover Clusters in Which All Nodes Run Hyper-V

·         TechNet Guide: Failover Cluster in which the Servers run Hyper-V

·         TechNet Webcast: 24 Hours of Windows Server 2008 (Part 24 of 24): High Availability with Hyper-V

·         TechNet Webcast: Creating Business Continuity Solutions Using Windows Virtualization

·         TechNet Webcast:High Availability with Hyper-V

·         Webcast: Top 10 VMWare Myths, including CSV and live migration

·         Webcast: Hyper-V Quick Migration on a Failover Cluster

·         Whitepaper: Quick Migration with Hyper-V

·         Whitepaper: Testing Hyper-V and Failover Clustering

Miscellaneous Resources

·         Blog Guide: Add a New Disk to a Cluster (2008)

·         Blog Guide: Configuring Auditing for a Cluster (2008)

·         Blog Guide: Cluster Recovery (2003)

·         KB Article:The Microsoft Support Policy for Windows Server 2008 Failover Clusters

·         TechNet Guide: Configuring the Quorum in a Failover Cluster

·         TechNet Guide: Managing a Failover Cluster

·         TechNet Guide: Modifying Settings for a Failover Cluster

·         TechNet Guide: The Failover Cluster Management Snap-In

·         TechNet Guide:Understanding Backup and Recovery Basics for a Failover Cluster

·         TechNet Guide: Support Policy

·         TechNet Guide: Windows Server 2008 Itanium / IA64 support

·         Webcast: Top 10 Windows Server 2008 Failover Clustering Enhancements over Windows Server 2003 Clustering, Based on Best Practices (Level 300)

·         Webcast: Failover Clustering 101

·         Webcast: Achieving High Availability with Windows Server “Longhorn” Clustering (Level 200)

• Webcast:Microsoft Webcast: Reducing IT Overhead with Windows Server 2008 Storage Features
• Webcast:TechNet Webcast: Build High-Availability Infrastructures with Windows Server 2008 Failover Clustering
• Webcast:IT Manager Webcast: Delivering High Availability to Your Infrastructure
• Webcast:TechNet Webcast: Failover Clustering and Quorum in Windows Server 2008 Enterprise Storage
• Webcast: TechNet Virtual Lab: Windows Server 2008 Enterprise Failover Clustering Lab

·         Whitepaper: Microsoft’s HA Strategy

·         Whitepaper: Overview of Failover Clustering

·         Whitepaper: HA with Microsoft MPIO (2003, 2008)

·         Website: Windows Logo site

·         Webcast: Introduction to Failover Clustering

Multi-Site Clustering

·         Cluster Team Site:http://www.microsoft.com/windowsserver2008/en/us/failover-clustering-multisite.aspx

·         KB Article: Deployment Considerations for Windows Server 2008 failover cluster nodes on different, routed subnets

·         Webcast: TechNet Webcast: Geographically Dispersed Failover Clustering in Windows Server 2008 Enterprise

·         Webcast: How You Can Achieve Greater Availability with Failover Clustering Across Multiple Sites (Level 300) 

·         Whitepaper: Multi-site Clustering

·         Webcast: Multi-Site Clustering in Windows Server 2008

Network Load Balancing

·         Blog Guide: Network Load Balancing (NLB) and Virtual Machines

·         KB Article: NLB Troubleshooting Overview

·         KB Article: Create/manage/destroy NLB clusters via NLB Manager remotely from another server, or from RSAT client (admin pack) on Vista

·         Presentation: Server Core: Install the NLB feature

·         TechNet Guide: Configuring NLB with Terminal Services

·         TechNet Guide: NLB Deployment Guide

·         TechNet Guide: Implementing a new NLB Cluster

·         TechNet Guide:Verifying the NLB Cluster and Enabling Client Access

·         TechNet Guide: Overview of NLB

·         TechNet Guide: Creating NLB Clusters

·         TechNet Guide: Managing NLB Clusters

·         TechNet Guide: Setting NLB Parameters

·         TechNet Guide: Controlling Hosts on NLB clusters

·         TechNet Guide:Troubleshooting for System Event Messages Related to NLB Cluster

·         TechNet Guide: User Interface: NLB Manager

·         TechNet Guide:Upgrading a NLB Cluster

·         TechNet Guide: Upgrading a Network Load Balancing (NLB) Cluster

·         Webcast:24 Hours of Windows Server 2008 (Part 23 of 24): Failover Clustering and Network Load Balancing z

Other Resources / Workloads

·         Blog Guide: Configure Multiple Instances of MSDTC (2008)

·         Blog Guide: Installing MSDTC (2003)

·         Blog Guide: Optimize Print Cluster (2003)

·         Blog Guide: Creating and Configuring a Generic Application Resource

·         TechNet Guide:Configuring Generic Resources

·         TechNet Guide: Configure a Service or Application for High Availability

Scripting

·         Blog Guide: Creating a Cluster using WMI

·         Blog Guide: CLI: Cluster Resource Groups

·         Blog Guide: CLI: Quorum

·         Blog Guide: CLI: Disk Resources

·         Blog Guide: CLI: Cluster Creation

·         Blog Guide: CLI: Adding Disks

·         TechNet: How to create an Exchange SCC Failover Cluster with CMD

SQL Server

• MSDN: Getting Started with SQL Server 2008 Failover Clustering
• MSDN: Configuring SQL Server 2005 Failover Clustering
• MSDN: Configuring SQL Server 2000 Failover Clustering
• MSDN: Add or Remove nodes in a SQL Server 2005 with MSCS
• MSDN: How to create a new SQL Failover Cluster 
• MSDN: Books Online: Maintaining a SQL Failover Cluster 
• Podcast: FAQs: HA with SQL Server 2008
• TechNet Guide: SQL Server 2000 with Failover Clustering 
• TechNet Guide: Database Mirroring and Failover Clustering 
• TechNet Guide:  SQL Server 2005 Database Mirroring FAQs 
• TechNet Guide: Database Mirroring in SQL Server 2005 
• Webcast: Microsoft SQL Server 2005 Failover Clustering on Windows Server 2008
• Webcast: TechNet Webcast: 24 Hours of SQL Server 2008: Ensuring Business Continuity
• Webcast:TechNet Webcast: Overview of SQL Server Availability Features and Upcoming Improvements
• Webcast:TechNet Webcast: Achieving Higher Scalability with SQL Server on Windows Server 2003 R2 Enterprise Edition (Level 300)
• Webcast: Ensuring Business Continuance with SQL Server 2005 Data Availability Solutions (Level 300)
• Webcast: Clustering Windows Server 2003 and SQL Server 2000/2005 (Level 200)
• Webcast: SQL Server 2005 for the IT Professional (Part 7 of 11): Technologies and Features to Improve Availability (Level 200)
• Webcast: SQL Server 2005 Failover Clustering for SQL Server 2000 Users (Level 200)
• Webcast: An Introduction to How SQL Server 2005 Can Help You Deliver Highly Available Database Systems (Level 200)
• Webcast: Clustering Windows Server 2003 and SQL Server 2000 and 2005 (Level 200) 
• Webcast: How to cluster Microsoft SQL Server 2005 by using Microsoft Virtual Server 
• Whitepaper: SQL Server 2008 Replication: Providing High Availability using Database Mirroring

Utilities

·         Utility: Failover Cluster Management Pack for Operations Manager 2007

·         Utility: ClusPrep: Cluster Configuration Validation Wizard (2003)

·         Utility:RSAT Remote Server Administration Tools (simplifies Server Core configurations)

·         Utility: File Server Migration Toolkit (FSMT) (2008)

Hello!

I am Rohan Mutagi.  My job at Microsoft is to do something that everyone likes: criticize others J, specifically, other people’s code.  Yes, I am a tester and it’s my role to find bugs in Network Load Balancing (NLB).  Over the next few months I will be blogging more about changes that NLB went through in Windows Server 2008 R2.  In this blog, I will focus NLB Extended Affinity (TCP). 

What is Extended Affinity?

To understand how NLB does load balancing, please refer to this TechNet article about various forms of affinity and their impact on load balancing decisions.

Extended Affinity is an extension to the Single and Network affinity.  NLB does not rely on any network protocol’s state to make its load balancing decisions.  As a result, NLB will work with a wide variety of protocols, documented and undocumented, stateless (HTTP/UDP etc.) and stateful (RDP/SSL etc).  This makes NLB more flexible in deployment and easier to manage since we don’t have to configure the load balancer to work with every protocol that it needs to handle.  However, some applications would benefit from being able to explicitly associate a connection to a server.

An example would be using IIS by online retailer using shopping carts.  When a customer shops at their store, they save the intended purchases in a shopping cart which is stored on one of the nodes in the cluster.  To keep the products in the shopping cart, the customer must stay connected to that same node.  However configuration changes to the cluster (such as adding a new VIP or node) which cause cluster convergence may then directed customers to another cluster node, and they have lost the purchases saved in that shopping cart.  Now the customer may become frustrated and the retailer may lose money.

Another instance could be with SSL where the SSL session can consist of multiple TCP Connections.  In normal operations, if single affinity is used, NLB will guarantee that all connections coming from the same source IP will hit the same server.  This would include multiple TCP connections of the same SSL session.  However, configuration changes might cause NLB to accept different connections of the same SSL session by different servers during the convergence.  As a result, the SSL session is broken.

With Extended Affinity, NLB provides the ability to associate a client connection to a NLB server over re-convergence.  This association holds true until the timeout specified by admin for the given port rule expires without any new traffic on the same connection.

Scenario

1.       We have a 2 node NLB cluster. (VIP: 2.2.2.2)

2.       Web browser Client (1.1.1.1) connects via SSL to NLB VIP (2.2.2.2).

3.       That particular connection is handled by IIS Server on NLB NODE1.

4.       Client Requests a web page that involves filling a web form.

5.       Client spends 20 minutes filling this form that would, once submitted, need to be stored on NODE1.

6.       In the meantime, on the server, admin adds a new node (NODE3) to the NLB Cluster.

7.       Now the connection (1.1.1.1 -> 2.2.2.2) is owned by NODE3

8.       The client submits his web form.

Without Extended Affinity

9.       Since the ownership of the connection (1.1.1.1 -> 2.2.2.2) has moved to NODE3, The server rejects the packet from the client.

10.   The browser tries to re-establish the SSL connection and this time hits a new server

11.   The new server will reject the “form data” that the browser provides since there is no authentication for this client on this node (NODE3). Thus the data that client filled-in is lost.

With Extended Affinity

9.       The server notices that stickiness in enabled for that particular connection (1.1.1.1 -> 2.2.2.2) and will route the connection to the correct owner of the connection (NODE1) despite the configuration change that caused the connection ownership to move to the new node (NODE3).

10.   The browser successfully communicates with the server and the transaction completes.

Applying Extended Affinity

The following sections detail how to use Extended Affinity in your Windows Server 2008 R2 NLB Cluster.

Using NLBManager

Extended Affinity can be modified by following the below steps:

1.       Right click on the cluster and Select "Cluster Properties"

2.       In the Cluster properties dialog box, Click on the "Port Rules" tab.

3.       Choose the appropriate port rule and click Edit:

4.       Select the appropriate affinity and set the "Timeout" value to required value. Click OK.

5.       Now you should see the new “Timeout” to be the set amount (10 mins).

Using PowerShell

Using Powershell, you can set the timeout for the default port rule using the CMDlet Set-NLBClusterPortRule.  For more information about using PowerShell with NLB, visit: http://blogs.msdn.com/clustering/archive/2008/12/26/9253786.aspx.

The below CMDlet will display all the port rules that are configured on a cluster on the current machine.  The “Timeout” shows the currently configured “Extended Affinity” timeout.  If this value is set to 0, that would mean that Extended Affinity is currently not enabled for the given port rule.  The below example shows that the timeout for all the port rules is set to 0.  This means that Extended Affinity is not enabled on any of the 3 port rules.

Now let’s enable Extended Affinity for the 2^nd port rule using PowerShell.

1.       Get the required port role using the Get-NlbClusterPortRule command we used above, but this time lets add a filter to find a port rule that is configured on a port 443 and bound to cluster on network interface Test-4

2.       Apply Extended Affinity to this port rule by using Set-NlbclusterPortRule to modify its timeout value.

Review:

Get-NlbClusterPortRule -Port <YourPortNumberHere> -InterfaceName <NetworkInterfaceName> | Set-NlbClusterPortRule –NewTimeout <NewTimeoutValueInMinutes>

That concludes the overview of the new Extended Affinity feature for NLB in Windows Server 2008 R2.  Thanks for reading this blog post.  If you have any questions feel free to contact is by clicking the ‘Email’ link on the upper right-corner of the page.

Thanks,

Rohan Mutagi
Software Development Engineer in Test
Clustering & High-Availability
Microsoft

Hi NLB Fans,

NLB provides users with various methods to manage clusters.  In Windows Server 2008, there are 3 ways to manage an NLB cluster:

1.       Network Load balancing Manager GUI (nlbmgr.exe)

2.       NLB command line tool (Nlb.exe)

3.       NLB WMI Provider (root\MicrosoftNLB namespace)

In Windows Server 2008 R2, the NLB team has introduced a PowerShell interface for configuring, managing and debugging NLB.  This awesome new feature makes it very easy to administer systems in an automated way.

In this blog post we will explore NLB's support for PowerShell.  We will elaborate on the original post PowerShell for NLB, providing more details on naming mechanism, samples and CMDlet discovery.

This blog post contains the following sections:

·         PowerShell Naming convention

·         Exploring NLB CMDlets

o   Using Get-Command

o   Using command Auto-completion

o   Using Argument auto completion

o   Getting examples to use

Future blog posts in this series will discuss:

·         NLB common scenarios

·         Basics of Debugging NLB with PowerShell

NLB PowerShell follows the PowerShell CMDlet guidelines in naming and execution of the NLB CMDlets. Here we will explore the general naming conventions that will make it easy to further understand and explore NLB CMDlets.

PowerShell Naming Convention

A CMDlet is made up of two parts a Noun and a Verb. These two parts of speech are combined together with a hyphen in between. A NLB example would be:

PS > Get-NlbCluster

The ‘Get’ example above is split into 2 parts, the verb (Get) and the noun (NlbCluster), and these 2 words are separated by a hyphen.  As rule of thumb, the verb defines the action to be performed on the noun.  In the above example, we want to "Get" all instances of "NlbCluster".

To view all the NLB CMDlets, run PS > Get-Command –module NetworkLoadBalancingClusters

A list of all the NLB supported verbs can be seen below:

A list of all the NLB supported nouns can be seen below:

Exploring NLB CMDlets

PowerShell makes it quite easy to use CMDlets, even if you have no prior knowledge of the NLB CMDlets.  PowerShell provides two main features that help with exploring/learning CMDlets.

Get-Command

You can use Get-Command to explore existing CMDlets that are available. This CMDlet, in conjunction with the knowledge of Verb-Noun pairing is a powerful way to getting to the CMDlet of interest.

Quick Syntax for this command

> Get-command -module NetworkLoadBalancingClusters [-Noun | -Verb <String>]

> Get-command <CommandFilter> -commandtype <commandtype>

Example usage

Let say we want to delete a node from the current cluster.  We know our end goal, but don’t know how to achieve it via PowerShell.  Using the above syntax we can try to reach our goal.  So the action we want to perform is "delete", and the noun that we want to act on is "NLB Cluster Node".

First we try to find all commands that start with "delete" verb and are of type CMDlet, by running > Get-Command delete-* -commandType cmdlet, but do not find any results.

Instead of "delete" let’s try “Remove".  Below we see that we found the CMDlet we are looking for.

We could have approached this in a different way.  We could have searched for the noun "Node" and filtered further on the exact verb.

As we can see with the above examples, we can intuitively guess the Verb-Noun pair for the NLB operation we want to perform, and use the Get-Command CMDlet to get the exact CMDlet.

The list below shows the usage of Get-Command to list out all the supported NLB CMDlets:

Command Auto-Completion

Another way to find out what CMDlets exist is to use the command auto-completion key <TAB>.

PowerShell provides a feature where the arguments of a CMDlet autocomplete.  

Examples

1.       Open PowerShell window with the NLB modules loaded.

2.       Type Add-NLBCluster<Tab>

This will automatically complete the above CMDlet, and display "Add-NLBClusterNode" on the screen.

This is another handy way to see what all CMDlets are supported.  Another example would be:

Start-NLB<TAB> would display Start-NLBCluster

Hitting <TAB> again, would display Start-NLBClusterNode

Argument Auto-Completion

Now that we know how to find the CMDlet of interest, let’s see how we further use this information to formulate the exact command that we need to execute.  PowerShell supports automatic expansion of the command arguments.  Once you have typed in a CMDlet you can type a hyphen (-) and hit <TAB> key to automatically expand the available arguments for the given CMDlet.

Examples

1.       Open a PowerShell Window with the NLB Module loaded

2.       Enter > Get-NLBCluster-<TAB>

3.       You will see that the “HostName” parameter will be auto-completed

4.       Hit <TAB> again and you will see the text “InterfaceName” appear by the text prompt.

Using the <TAB> you can cycle through all the available arguments that the give CMDlet supports.  If you went past an argument while hitting <TAB>, you can go back to it using the <SHIFT+TAB> key sequence.

This auto-completion can be further “filtered” by typing the first few characters of the argument you are interested in.  For example, if I want to look for a parameter “InterfaceName”, you can try the following:

1.       Open a PowerShell Window with the NLB Module loaded

2.       Type “Get-NLBCluster -i“ <TAB>

3.       This will directly show you all the CMDlets that begin with the letter “I”, in this case “InterfaceName”

Get-Help

 As you may know from the ‘Help Documentation’ section of the earlier NLB blog post that the Get-Help CMDlet is incredibly powerful.

The final thing that I would like to bring up in this section is the use of the –example argument for the help.  As the name suggests, you can quickly see the examples of a given CMDlet via the “-example” argument.

Example

Another awesome support option is the “-Online” option. This will launch the web browser with online content that is up-to-date with the latest information regarding the CMDlet (of course, this may not work if you are using a Server Core installation which cannot access Internet Explorer).

Example:

> Get-Help –Online New-NlbCluster

Rohan Mutagi & Gary Jackman
Clustering & High-Availability Test Team
Microsoft

Hi Cluster Fans,

We have added Windows Server 2008 R2 PowerShell help on TechNet and have updated our Management Packs to support 2008 R2 for System Center Operation Manager (SCOM), for both Failover Clustering and Network Load Balancing.

PowerShell Help Online 

With PowerShell help online you can see the same information as the inbox Get-Help CMDlet in an easier to browse format, with more examples and information added over time.  The website can also be launched in your default web browser directly from PowerShell, assuming your machine has a web browser.  This is done by adding -Online to the Get-Help CMDlet.

Failover Clustering

The main site for PowerShell for Failover Clustering is at: http://technet.microsoft.com/en-us/library/ee461009.aspx

Here’s an example for Failover Clustering:  PS > Get-Help Test-Cluster -Online

Network Load Balancing

The main site for PowerShell for Network Load Balancing is at: http://technet.microsoft.com/en-us/library/ee817138.aspx

Here’s an example for Failover Clustering:  PS > Get-Help New-NLBCluster -Online

SCOM Management Pack Updates

We’ve updated our Management Packs for System Center Operation Manager (SCOM) for both Failover Clustering & NLB to add news features and support Windows Server 2008 R2. 

Failover Clustering

The SCOM Failover Clustering Management Pack provides both proactive and reactive monitoring of your Windows Server 2003, Windows Server 2008 and Windows Server 2008 R2 cluster deployments. It monitors Cluster services components—such as nodes, networks, and resource groups—to report issues that can cause downtime or poor performance.

The main site for the Management Pack for Failover Clustering is at: http://www.microsoft.com/downloads/details.aspx?FamilyId=AC7F42F5-33E9-453D-A923-171C8E1E8E55

Some of the improvements include:

·         Support for discovery and monitoring of Windows Server 2008 R2 clusters and functionality such as Cluster Shared Volumes

·         MP scalability improvements (the MP supports monitoring of 300 resource groups per cluster)

·         Noise reduction, for example clustered resources are no longer discovered and monitored by default (resource groups are monitored by default)

·         Configuration or hardware issues that interfere with starting the Cluster service

·         Alerts about connectivity problems that affect communication between cluster nodes or between a node and a domain controller

·         Active Directory Domain Services (AD DS) settings that affect the cluster; for example, permissions needed by the computer account that is used by the cluster

·         Configuration issues with the network infrastructure needed by the cluster; for example, issues with Domain Name System (DNS)

·         Issues with the availability of a cluster resource, such as a clustered file share

·         Issues with the cluster storage

Network Load Balancing

The SCOM Network Load Balancing (NLB) Management Pack provides discoveries, monitors, alerts, and warnings to help the operator understand the state of NLB clusters and NLB servers running Windows Server 2008 and Windows Server 2008 R2. The Management Pack can provide early warnings that an operator can use to proactively monitor the state of the NLB servers in the computing environment.

The main site for the Management Pack for Network Load Balancing is at: http://www.microsoft.com/downloads/details.aspx?FamilyID=dc17e093-bdd7-4cb3-9981-853776ed90be

Some of the improvements include:

·         Support for discovery and monitoring of Windows Server 2008 R2 NLB clusters

·         Monitor the NLB Node status

·         Based on the status of individual cluster nodes, determine the overall state of the cluster.

·         Where an integration management pack exists, determine the health state of a cluster node by looking at the health state of the load balanced application, such as IIS

·         Alert on errors and warnings that are reported by the NLB driver, such as an incorrectly configured NLB cluster

·         Ability to the node out of the NLB cluster if the underlying load-balanced application becomes unhealthy, and add the node back to the cluster when the application becomes healthy again

·         Noise reduction on some alerts

Enjoy these improvements to your clustering experience!

Thanks,

Symon Perriman

Program Manager II

Clustering & High-Availability

Microsoft

Hi,

This is the second blog is our series of posts on PowerShell for Network Load Balancing (NLB).  The first post introduces you to the CMDlets: http://blogs.msdn.com/clustering/archive/2009/10/28/9913877.aspx

Most of NLB CMDlets have the following common parameters.

Most CMDlets require reference to a Cluster object.   To get a Cluster object you can run Get-NLBCluster and pass the output object to the desired CMDlet or use the -interfaceName parameter. 

We will discuss running CMDlets and using the output as input of another CMDlet in future posts.

Creating a New Cluster

New-NLBCluster

A new cluster can be created via NLB using New-NLBCluster CMDlet. This is a synchronous command, meaning that it will only return after completing the operation.  You can also use this CMDlet to create a new cluster on remote nodes.  To achieve this, the managing system must have Windows Server 2008 R2 installed and the cluster node must be Windows Server 2008 or higher.

New-NLBCluster has the following parameters of interest.

Example

Adding Nodes to a Cluster

Add-NLBClusterNode

Once a cluster has been created, we may want to add more nodes to the cluster. This can be achieved via the Add-NLBClusterNode CMDlet.

Parameters of interest:

Example

Managing Port Rules

Set-NLBClusterPortRule

After creating a new NLB cluster you may want to modify the port rules before adding any nodes.  To do so you will want to use the Set-NLBClusterPortRule CMDlet.

Set-NLBClusterPortRule will modify existing port rules.  For example, when creating a new cluster, the default port rule is added.  If you want to customize the port rule you can either delete the existing port rule or modify the existing port rule.  Modifying the existing port rule is the best approach because you run only one command rather than two commands.

Set-NLBClusterPortRule has the following parameters that I believe are the most useful.   As always, for detailed help on this please run Get-HelpSet-NLBClusterPortRule.

Example

This shows how to change the port rule:

The previous example assumes that only one port rule exists prior to modifying the port rule.  If multiple port rules exist prior to running the command and you wanted to modify the StartPort or EndPort,  you will get an error because the port ranges (as specified by the start port and end ports) overlap.

Example

If you want to modify the port range, you should use the -port parameter:

You may have noticed that the example shows changing affinity instead of the port range.   I did this to set up for the next example where I change the affinity to single affinity on both port rules. 

Managing Cluster Nodes

Set-NLBClusterNode

To manage NLB node properties such as host priority, initial host state or persisted suspend state, you need to use Set-NLBClusterNode.

 By default Set-NLBClusterNode manages only one node at a time.  For example, when running a command from one of the nodes the local host is the node that is managed.

If you want to run a command that executes on all nodes you can first run the Get-NLBClusterNode and redirect the output to Set-NLBClusterNode.

To view all node properties you can run the following Get-NLBClusterNode and pipe the output through Format-List CMDlet.

Controlling Cluster Nodes

 Start-NLBClusterNode & Stop-NLBClusterNode

To control the state (such as stop or start) of the cluster or a node there is a CMDlet for the respective action or "verb" and the respective object.  For example to stop a cluster you could run Stop-NLBClusterNode while Start-NLBClusterNode CMDlet will start the specific cluster node.

The CMDlet I want to discuss here is the Stop-NLBClusterNode command, specifically the parameter, -Timeout.  This new parameter lets you control the time you want to wait before forcing the Stop operation on the node. Now you don’t have to wait for Drain to complete, before doing a stop. You can simply run this command with a timeout value, like in the example below.

In creating the CMDlets we combined stop and drainstop in to one CMDlet, Stop-NLBCluster and Stop-NLBClusterNode.

Example

This example will do the following:

1.       Drain all the connections on the Cluster

2.       If there are no outstanding connections, stop the cluster immediately

3.       If all connections are not drained in less than 10 minutes, force stop the node, breaking all existing connections to that particular node.

Debugging NLB with PowerShell

Get-NLBClusterDriverInfo

The NLB team has added an awesome CMDlet, Get-NLBClusterDriverInfo, this CMDlet is a replacement for the nlb.exe binary that you may have used. This is a loaded CMDlet with lots of options. Note, this CMDlet does not provide any remoting capabilities, so it does not take hostname as input parameter.

1.       Getting the Cluster configuration: When this CMDlet is run without any arguments, it returns the basic cluster configuration on the current machine.

2.       We can determine if a given connection will be handled by the current node using the -filter argument.  This argument requires the following additional arguments to be set:

In the following example, we are checking to see if a TCP connection coming from client: 1.1.1.1 will be accepted by the NLB server on Port 80, whose VIP is 1.1.1.2

Stay tuned for more NLB PowerShell information!

Thanks,

Rohan Mutagi & Gary Jackman
Clustering & High-Availability Test Team
Microsoft

Windows Server 2016, debuts the birth of site-aware clusters. Nodes in stretched clusters can now be grouped based on their physical location (site). Cluster site-awareness enhances key operations during the cluster lifecycle such as failover behavior, placement policies, heartbeating between the nodes and quorum behavior. In the remainder of this blog I will explain how you can configure sites for your cluster, the notion of a “preferred site” and how site awareness manifests itself in your cluster operations.

Configuring Sites

A node’s site membership can be configured by setting the Site node property to a unique numerical value.

For example, in a four node cluster with nodes – Node1, Node2, Node3 and Node4, to assign the nodes to Sites 1 and Site 2, do the following:

• Launch Microsoft PowerShell^© as an Administrator and type:

Technical Preview 4

(Get-ClusterNode Node1).Site=1
(Get-ClusterNode Node2).Site=1
(Get-ClusterNode Node3).Site=2
(Get-ClusterNode Node4).Site=2

Technical Preview 5

In Technical Preview 5 the syntax to configure sites has changed to the following:

#Create Site Fault Domains
New-ClusterFaultDomain –Name Seattle –Type Site –Description “Primary” –Location “Seattle DC”
New-ClusterFaultDomain –Name Denver –Type Site –Description “Secondary” –Location “Denver DC”

#Set Fault Domain membership
Set-ClusterFaultDomain –Name Node1 –Parent Seattle
Set-ClusterFaultDomain –Name Node2 –Parent Seattle

Set-ClusterFaultDomain –Name Node3 –Parent Denver
Set-ClusterFaultDomain –Name Node4 –Parent Denver

Configuring sites enhances the operation of your cluster in the following ways:

Failover Affinity

• Groups failover to a node within the same site, before failing to a node in a different site
• During Node Drain VMs are moved first to a node within the same site before being moved cross site
• The CSV load balancer will distribute within the same site

Storage Affinity

Virtual Machines (VMs) follow storage and are placed in same site where their associated storage resides. VMs will begin live migrating to the same site as their associated CSV after 1 minute of the storage being moved.

Cross-Site Heartbeating

You now have the ability to configure the thresholds for heartbeating between sites. These thresholds are controlled by the following new cluster properties:

 To configure the above properties launch PowerShell^© as an Administrator and type:

(Get-Cluster).CrossSiteDelay = <value>
(Get-Cluster).CrossSiteThreshold = <value>

You can find more information on other properties controlling failover clustering heartbeating here.

The following rules define the applicability of the thresholds controlling heartbeating between two cluster nodes:

• If the two cluster nodes are in two different sites and two different subnets, then the Cross-Site thresholds will override the Cross-Subnet thresholds.
• If the two cluster nodes are in two different sites and the same subnets, then the Cross-Site thresholds will override the Same-Subnet thresholds.
• If the two cluster nodes are in the same site and two different subnets, then the Cross-Subnet thresholds will be effective.
• If the two cluster nodes are in the same site and the same subnets, then the Same-Subnet thresholds will be effective.

Configuring Preferred Site

In addition to configuring the site a cluster node belongs to, a “Preferred Site” can be configured for the cluster. The Preferred Site is a preference for placement. The Preferred Site will be your Primary datacenter site.

Before the Preferred Site can be configured, the site being chosen as the preferred site needs to be assigned to a set of cluster nodes. To configure the Preferred Site for a cluster, launch PowerShell^© as an Administrator and type:

(Get-Cluster).PreferredSite = <Site assigned to a set of cluster nodes>

Configuring a Preferred Site for your cluster enhances operation in the following ways:

Cold Start

During a cold start VMs are placed in in the preferred site

Quorum

• Dynamic Quorum drops weights from the Disaster Recovery site (DR site i.e. the site which is not designated as the Preferred Site) first to ensure that the Preferred Site survives if all things are equal. In addition, nodes are pruned from the DR site first, during regroup after events such as asymmetric network connectivity failures.
• During a Quorum Split i.e. the even split of two datacenters with no witness, the Preferred Site is automatically elected to win
  • The nodes in the DR site drop out of cluster membership
  • This allows the cluster to survive a simultaneous 50% loss of votes
  • Note that the LowerQuorumPriorityNodeID property previously controlling this behavior is deprecated in Windows Server 2016

Preferred Site and Multi-master Datacenters

The Preferred Site can also be configured at the granularity of a cluster group i.e. a different preferred site can be configured for each group. This enables a datacenter to be active and preferred for specific groups/VMs.

To configure the Preferred Site for a cluster group, launch PowerShell^© as an Administrator and type:

(Get-ClusterGroup -Name <GroupName>).PreferredSite = <Site assigned to a set of cluster nodes>

Placement Priority

Groups in a cluster are placed based on the following site priority:

1. Storage affinity site
2. Group preferred site
3. Cluster preferred site

One of the big hot features in Windows Server 2016 which has me really excited is Storage Spaces Direct (S2D).  With S2D you will be able to create a hyper-converged private cloud.  A hyper-converged infrastructure (HCI) consolidates compute and storage into a common set of servers.  Leveraging internal storage which is replicated, you can create a true Software-defined Storage (SDS) solution.

This is available in the Windows Server 2016 Technical Preview today!  I encourage you to go try it out and give us some feedback.  Here’s where you can learn more:

Presentation from Ignite 2015:

Storage Spaces Direct in Windows Server 2016 Technical Preview
https://channel9.msdn.com/events/Ignite/2015/BRK3474

Deployment guide:

Enabling Private Cloud Storage Using Servers with Local Disks

https://technet.microsoft.com/en-us/library/mt126109.aspx

Claus Joergensen’s blog:

Storage Spaces Direct
http://blogs.technet.com/b/clausjor/archive/2015/05/14/storage-spaces-direct.aspx

Thanks!
Elden Christensen
Principal PM Manager
High-Availability & Storage
Microsoft

In a previous blog,I discussed the introduction of site-aware Failover Clusters in Windows Server 2016. In this blog, I am going to walk through how you can configure site-awareness for your multi-active disaggregated datacenters. You can learn more about Software Defined Storage and the advantages of a disaggregated datacenter here.

Consider the following multi-active datacenters, with a compute and a storage cluster, stretched across two datacenters. Each cluster has two nodes on each datacenter.

To configure site-awareness for the stretched compute and storage clusters proceed as follows:

Compute Stretch Cluster

1)     Assign the nodes in the cluster to one of the two datacenters (sites).

• Open PowerShell^© as an Administrator and type:

(Get-ClusterNode Node1).Site = 1
(Get-ClusterNode Node2).Site = 1
(Get-ClusterNode Node3).Site = 2
(Get-ClusterNode Node4).Site = 2

2)     Configure the site for your primary datacenter.

(Get-Cluster).PreferredSite = 1

Storage Stretch Cluster

In multi-active disaggregated datacenters, the storage stretch cluster hosts a Scale-Out File Server (SoFS). For optimal performance, it should be ensured that the site hosting the Cluster Shared Volumes comprising the SoFS, follows the site hosting the compute workload. This avoids the cost of cross-datacenter network traffic.

1)     As in the case of the compute cluster assign the nodes in the storage cluster to one of the two datacenters (sites).

(Get-ClusterNode Node5).Site = 1
(Get-ClusterNode Node6).Site = 1
(Get-ClusterNode Node7).Site = 2
(Get-ClusterNode Node8).Site = 2

2)     For each Cluster Shared Volume (CSV) in the cluster, configure the preferred site for the CSV group to be the same as the preferred site for the Compute Cluster.

$csv1 = Get-ClusterSharedVolume "Cluster Disk 1" | Get-ClusterGroup
($csv1).PreferredSite = 1 

3)  Set each CSV group in the cluster to automatically failback to the preferred site when it is available after a datacenter outage.

($csv1).AutoFailbackType = 1

Note: Step 2 and 3 can also be used to configure the Preferred Site for a CSV group in a hyper-converged data-center deployment. You can learn more about hyper-converged deployments in Windows Server 2016 here.

Do you want your server OS deployment and servicing to move faster? We’re a team of Microsoft engineers who want your experiences and ideas around solving real problems of deploying and servicing your server OS infrastructure. We prefer that you don’t love server OS deployment already, and we’re interested even if you don’t use Windows Server. We need to learn it and earn it.

Click the link below if you wish to fill out a brief survey and perhaps participate in a short phone call.

https://aka.ms/deployland

Many Thanks!!!

-Rob.

Hi Cluster Fans,

It is nice to be back on the Cluster Team Blog!  After founding this blog and working closely with the cluster team for almost eight years, I left Microsoft last year to join a Hyper-V software partner, 5nine Software.  I’ve spoken with thousands of customers and I realized that Failover Clustering is so essential to Hyper-V, that a majority of all VMs are using it, and it is businesses of all sizes that are doing this, not just enterprises.  Most organizations need continual availability for their services to run 24/7, and their customers expect it.  Failover Clustering is now commonplace even amongst small and medium-sized businesses.  I was able to bring my passion for cluster management to 5nine’s engineering team, and into 5nine’s most popular SMB product, 5nine Manager.  This blog provides an overview of how 5nine Manager can help you centralize management of your clustered resources.

 Create a Cluster

5nine Manager lets you discover hosts and create a Failover Cluster.  It will allow you to specify nodes, run Cluster Validation, provide a client access point, and then create the cluster.

Validate a Cluster

Failover Cluster validation is an essential task in all deployments as it is required for a support cluster.  With 5nine Manager you can test the health of your cluster during configuration, or afterwards as a troubleshooting tool.  You can granularly select the different tests to run, and see the same graphical report as you are familiar with.

Host Best Practice Analyzer

In addition to testing the clustering configuration, you can also run a series of advanced Hyper-V tests on each of the hosts and Scale-Out File Servers through 5nine Manager.  The results will provide recommendations to enhance your node’s stability and performance.

Configure Live Migration Settings

It is important to have a dedicated network to Live Migration to ensure that its traffic does not interfere with cluster heartbeats or other important traffic.  With 5nine Manager you can specify the number of simultaneous Live Migrations and Storage Live Migrations, and even copy those settings to the other cluster nodes.

View Cluster Summary

5nine Manager has a Summary Dashboard which centrally reports the health of the cluster and its VMs.  It quickly identifies nodes or VMs with problems, and lists any alerts from its resources.  This Summary Dashboard can also be refocused at the Datacenter, Cluster, Host, and VM level for more refined results.

Manage Cluster Nodes

Using 5nine Manager you can configure your virtual disk and network settings.  You can also perform standard maintenance tasks, such as to Pause and Resume a cluster node, which can live migrate VMs to other nodes.  A list of active and failed cluster tasks is also displayed through the interface.

Manage Clustered VMs

You can manage any type of virtual machine that is supported by Hyper-V, including Windows Server, Windows, Linux, UNIX, and Windows Server 2016 Nano Server.  5nine Manager lets you centrally manage all your virtual machines, including the latest performance and security feature for virtualization.  The full GUI console even runs on all versions of Windows Server, including the otherwise GUI-less Windows Server Core and Hyper-V Server.

Cluster Status Report

It is now easy to create a report about the configuration and health of your cluster, showing you information about the configuration and settings for every resource.  This document can be exported and retained for compliance.

Host Load Balancing

5nine Manager allows you to pair cluster nodes and hosts to form a group that will load balance VMs.  It live migrates the VMs between hosts when customizable threshold are exceeded.  This type of dynamic optimization ensures that a single host does not get overloaded, providing higher-availability and greater performance for the VMs.

Cluster Logs

Sometime it can be difficult to see all the events from across your cluster.  5nine Manager pulls together all the logs for your clusters, hosts and VMs to simplify troubleshooting.

Cluster Monitoring

5nine Manager provides a Monitor Dashboard to provide current and historical data about the usage of your clusters, hosts and VMs.  It will show you which VMs are consuming the most resources, the latest alarms, and a graphical view of CPU, memory, disk and network usage.  You can also browse through previous performance data to help isolate a past issue.

Hyper-V Replica with Clustering

Hyper-V Replica allows a virtual machine’s virtual hard disk to be copied to a secondary location for disaster recovery.  Using 5nine Manager you can configure the Replication Settings on a host, then apply them to other cluster nodes and hosts. 

You can also configure replication on a virtual machine that is running a cluster node with the Hyper-V Replica Broker configured.  The health state of replica is also displayed in the centralized console.

Failover Clustering should be an integral part of your virtualized infrastructure, and 5nine Manager provides a way to centrally manage all your clustered VMs.  Failover cluster support will continue to be enhanced in future releases of 5nine Manager.

Thanks!
Symon Perriman
VP, 5nine Software
Hyper-V MVP
@SymonPerriman

In this blog post https://blogs.msdn.microsoft.com/clustering/2014/12/08/troubleshooting-cluster-shared-volume-auto-pauses-event-5120/ we discussed what a Cluster Shared Volumes (CSV) event ID 5120 means, and how to troubleshoot it. In particular, we discussed the reason for auto-pause due to STATUS_IO_TIMEOUT (c00000b5), and some options on how to troubleshoot it. In this post we will discuss how to troubleshoot it using LiveDumps, which enables debugging the system with no downtime for your system.

First let’s discuss what is the LiveDump. Some of you are probably familiar with kernel crash dumps https://support.microsoft.com/en-us/kb/927069. You might have at least two challenges with kernel dump.

1. Bugcheck halts the system resulting in downtime
2. Entire contents of memory are dumped to a file.  On a system with a lot of memory, you might not have enough space on your system drive for OS to save the dump

The good news is that LiveDump solves both of these issues. Live Dump was a new feature added in Windows Server 2012 R2. For the purpose of this discussion you can think of LiveDump as an OS feature that allows you to create a consistent snapshot of kernel memory and save it to a dump file for the future analysis. Taking this snapshot will NOT cause bugcheck so no downtime. LiveDump does not include all kernel memory, it excludes information which is not valuable in debugging. It will not include pages from stand by list and file caches. The kind of livedump that cluster collects for you also would not have pages consumed by Hypervisor. In Windows Server 2016 Cluster also makes sure to exclude from the livedump CSV Cache. As a result LiveDump has much smaller dump file size compared to what you would get when you bugcheck the server, and would not require as much space on your system drive.  In Windows Server 2016 there is a new bugcheck option called an “Active Dump”, which similarly excludes unnecessary information to create a smaller dump file during bugchecks.

You can create LiveDump manually using LiveKD from Windows Sysinternals (https://technet.microsoft.com/en-us/sysinternals/bb897415.aspx ). To generate LiveDump run command “livekd –ml –o <path to a dump file>” from an elevated command prompt. Path to the dump file does not have to be on the system drive, you can save it to any location. Here is an example of creating live dump on a Windows 10 Desktop with 12 GB RAM, which resulted in a dump file of only 3.7 GB.

D:\>livekd -ml -o d1.dmp
LiveKd v5.40 - Execute kd/windbg on a live system
Sysinternals - www.sysinternals.com

Copyright (C) 2000-2015 Mark Russinovich and Ken Johnson

Saving live dump to D:\d1.dmp... done.

D:\>dir *.dmp

Directory of D:\

02/25/2016 12:05 PM     2,773,164,032 d1.dmp
1 File(s) 2,773,164,032 bytes
0 Dir(s) 3,706,838,417,408 bytes free

If you are wondering how much disk space you would need to livedump you can generate one using LiveKD, and check its size.

You might wonder what so great about LiveDump for troubleshooting. Logs and traces work well when something fails because hopefully in a log there will be a record where someone admits that he is failing operations and blames someone who causes that. LiveDump is great when we need to troubleshoot a problem where something is taking long time, and nothing is technically failing. If we start a watchdog when operation started, and if watchdog expires before operation completes then we can try to take a dump of the system hoping that we can walk a wait chain for that operation and see who owns it and why it is not completing. Looking at the livedump is just like looking at kernel dumps. It requires some skills, and understanding of Windows Internals. It has a steep learning curve for customers, but it is a great tool for Microsoft support and product teams who already have that expertise. If you reach out to Microsoft support with an issue where something is stuck in kernel, and a live dump taken while it was stuck then chances of prompt root causing of the issue are much higher.

Windows Server Failover Clustering has many watchdogs which control how long it should wait for cluster resources to execute calls like resource online or offline. Or how long we should wait for CSVFS to complete a state transition. From our experience we know that in most cases some of these scenarios will be stuck in the kernel so we automatically ask Windows Error Reporting to generate LiveDump. It is important to notice that LiveKd uses different API that produces LiveDump without checking any other conditions. Cluster uses Windows Error Reporting. Windows Error Reporting will throttle LiveDump creation. We are using WER because it manages disk space consumption for us and it also will send telemetry information about the incident to Microsoft where we can see what issues are affecting customers. This helps us to priorities and strategize fixes. Starting from Windows Server 2016 you can control WER telemetry through common telemetry settings, and before that there was a separate control panel applet to control what WER is allowed to share with Microsoft.

By default, Windows Error Reporting will allow only one LiveDump per report type per 7 days and only 1 LiveDump per machine per 5 days. You can change that by setting following registry keys

reg add "HKLM\Software\Microsoft\Windows\Windows Error Reporting\FullLiveKernelReports" /v SystemThrottleThreshold /t REG_DWORD /d 0 /f
reg add "HKLM\Software\Microsoft\Windows\Windows Error Reporting\FullLiveKernelReports" /v ComponentThrottleThreshold /t REG_DWORD /d 0 /f

Once LiveDump is created WER would launch a user mode process that creates a minidump from LiveDump, and immediately after that would delete the LiveDump. Minidump is only couple hundred kilobytes, but unfortunately it is not helpful because it would have call stack only of the thread that invoked LiveDUmp creation, and we need all other threads in the kernel to track down where we are stuck. You can tell WER to keep original Live dumps using these two registry keys.

reg add "HKLM\Software\Microsoft\Windows\Windows Error Reporting\FullLiveKernelReports" /v FullLiveReportsMax /t REG_DWORD /d 10 /f
reg add "HKLM\SYSTEM\CurrentControlSet\Control\CrashControl" /v AlwaysKeepMemoryDump /t REG_DWORD /d 1 /f

Set FullLiveReportsMax to the number of dumps you want to keep, the decision on how many to keep depends on how much free space you have and the size of LiveDump.
You need to reboot the machine for Windows Error Reporting registry keys to take an effect.
LiveDumps created by Windows Error Reporting are located in the %SystemDrive%\Windows\LiveKernelReports.

Windows Server 2016

In Windows Server 2016 Failover Cluster Live Dump Creation is on by default. You can turn it on/off by manipulating lowest bit of the cluster DumpPolicy public property. By default, this bit is set, which means cluster is allowed to generate LiveDump.

PS C:\Windows\system32> (get-cluster).DumpPolicy
1118489

If you set this bit to 0 then cluster will stop generating LiveDumps.

PS C:\Windows\system32> (get-cluster).DumpPolicy=1118488

You can set it back to 1 to enable it again

PS C:\Windows\system32> (get-cluster).DumpPolicy=1118489

Change take effect immediately on all cluster nodes. You do NOT need to reboot cluster nodes.

Here is the list of LiveDump report types generated by cluster. Dump files will have report type string as a prefix.

You can learn more about CSV state transition in this blog post:

• http://blogs.msdn.com/b/clustering/archive/2014/10/27/10567706.aspx

Following is the list of LiveDump report types that cluster generates when cluster resource call is taking too long

You can learn more about cluster resource state machine in these two blog posts:

• https://blogs.msdn.microsoft.com/clustering/2010/03/10/creating-a-cluster-resource-dll-part-1/
• https://blogs.msdn.microsoft.com/clustering/2010/03/29/creating-a-cluster-resource-dll-part-2/

You can control what resource types will generate LiveDumps by changing value of the first bit of the resource type DumpPolicy public property. Here are the default values:

C:\> Get-ClusterResourceType | ft Name,DumpPolicy

Name                                DumpPolicy
----                                ----------
Cloud Witness                       5225058576
DFS Replicated Folder               5225058576
DHCP Service                        5225058576
Disjoint IPv4 Address               5225058576
Disjoint IPv6 Address               5225058576
Distributed File System             5225058576
Distributed Network Name            5225058576
Distributed Transaction Coordinator 5225058576
File Server                         5225058576
File Share Witness                  5225058576
Generic Application                 5225058576
Generic Script                      5225058576
Generic Service                     5225058576
Health Service                      5225058576
IP Address                          5225058576
IPv6 Address                        5225058576
IPv6 Tunnel Address                 5225058576
iSCSI Target Server                 5225058576
Microsoft iSNS                      5225058576
MSMQ                                5225058576
MSMQTriggers                        5225058576
Nat                                 5225058576
Network File System                 5225058577
Network Name                        5225058576
Physical Disk                       5225058577
Provider Address                    5225058576
Scale Out File Server               5225058577
Storage Pool                        5225058577
Storage QoS Policy Manager          5225058577
Storage Replica                     5225058577
Task Scheduler                      5225058576
Virtual Machine                     5225058576
Virtual Machine Cluster WMI         5225058576
Virtual Machine Configuration       5225058576
Virtual Machine Replication Broker  5225058576
Virtual Machine Replication Coor... 5225058576
WINS Service                        5225058576

By default, Physical Disk resources would produce LiveDump. You can disable that by setting lowest bit to 0. Here is an example how to do that for the physical disk resource

(Get-ClusterResourceType -Name "Physical Disk").DumpPolicy=5225058576

Later on you can enable it back

(Get-ClusterResourceType -Name "Physical Disk").DumpPolicy=5225058577

Changes take effect immediately on all new calls, no need to offline/online resource or restart the cluster.

The last group is the report types that cluster service would generate when it observes that some operations are taking too long.

Windows Server 2012 R2

We had such a positive experience troubleshooting issues using LiveDump on Windows Server 2016 that we’ve backported a subset of that back to Windows Server R2. You need to make sure that you have all the recommended patches outlined here. On Windows Server 2012 R2 LiveDump will not be generated by default, it can be enabled using following PowerShell command:

Get-Cluster | Set-ClusterParameter -create LiveDumpEnabled -value 1

LiveDump can be disabled using the following command:

Get-Cluster | Set-ClusterParameter -create LiveDumpEnabled -value 0

Only CSV report types were backported, as a result you will not see LiveDumps from cluster resource calls or cluster service watchdog.  Windows Error Reporting throttling will also need to be adjusted as discussed above.

CSV AutoPause due to STATUS_IO_TIMEOUT (c00000b5)

Let’s see how LiveDump help troubleshooting this issue. In the blog post https://blogs.msdn.microsoft.com/clustering/2014/12/08/troubleshooting-cluster-shared-volume-auto-pauses-event-5120/ we’ve discussed that it is usually caused by an IO on the coordinating node taking long time. As a result of that CSVFS on a non-coordinating node would get an error STATUS_IO_TIMEOUT. CSVFS will notify cluster service about that event. Cluster service will create LiveDump with report type CsvIoT on the coordinating node where IO is taking time. If we are lucky, and the IO has not completed before the LiveDump has been generated then we can load the dump using WinDbg to try to find the IO that is taking a long time and see who owns that IO.

Thanks!
Vladimir Petter
Principal Software Engineer
High-Availability & Storage
Microsoft

Additional Resources:

To learn more, here are others in the Cluster Shared Volume (CSV) blog series:

Cluster Shared Volume (CSV) Inside Out
http://blogs.msdn.com/b/clustering/archive/2013/12/02/10473247.aspx

Cluster Shared Volume Diagnostics
http://blogs.msdn.com/b/clustering/archive/2014/03/13/10507826.aspx

Cluster Shared Volume Performance Counters
http://blogs.msdn.com/b/clustering/archive/2014/06/05/10531462.aspx

Cluster Shared Volume Failure Handling
http://blogs.msdn.com/b/clustering/archive/2014/10/27/10567706.aspx

Troubleshooting Cluster Shared Volume Auto-Pauses – Event 5120
http://blogs.msdn.com/b/clustering/archive/2014/12/08/10579131.aspx

Troubleshooting Cluster Shared Volume Recovery Failure – System Event 5142
http://blogs.msdn.com/b/clustering/archive/2015/03/26/10603160.aspx

Cluster Shared Volume – A Systematic Approach to Finding Bottlenecks
https://blogs.msdn.microsoft.com/clustering/2015/07/29/cluster-shared-volume-a-systematic-approach-to-finding-bottlenecks/

Windows Server 2016 introduces the Node Fairness feature to optimize the utilization of nodes in a Failover Cluster. During the lifecycle of your private cloud, certain operations (such as rebooting a node for patching), results in the Virtual Machines (VMs) in your cluster being moved. This could result in an unbalanced cluster where some nodes are hosting more VMs and others are underutilized (such as a freshly rebooted server). The Node Fairness feature seeks to identify over committed nodes and re-distribute VMs from those nodes. VMs are live migrated to idle nodes with no down time. Failure policies such as anti-affinity, fault domains and possible owners are honored. Thus, the Node Fairness feature seamlessly balances your private cloud.

Heuristics for Balancing

Node Fairness evaluates a node’s load based on the following heuristics:

1. Current Memory pressure: Memory is the most common resource constraint on a Hyper-V host
2. CPU utilization of the Node averaged over a 5 minute window: Mitigates a node in the cluster becoming overcommitted

Controlling Aggressiveness of Balancing

The aggressiveness of balancing based on the Memory and CPU heuristics can be configured using the by the cluster common property ‘AutoBalancerLevel’. To control the aggressiveness run the following in PowerShell:

(Get-Cluster).AutoBalancerLevel = <value>

Controlling Node Fairness

Node Fairness is enabled by default and when load balancing occurs can be configured by the cluster common property ‘AutoBalancerMode’. To control when Node Fairness balances the cluster:

Using Failover Cluster Manager:

1. Right-click on your cluster name and select the “Properties” option

2.  Select the “Balancer” pane

Using PowerShell:

Run the following: :

(Get-Cluster).AutoBalancerMode = <value>

Node Fairness vs. SCVMM Dynamic Optimization

The node fairness feature, provides in-box functionality, which is targeted towards deployments without System Center Virtual Machine Manager (SCVMM). SCVMM Dynamic Optimization is the recommended mechanism for balancing virtual machine load in your cluster for SCVMM deployments. SCVMM automatically disables the Node Fairness feature when Dynamic Optimization is enabled.

From time-to-time people ask me for suggestions on what tweaks they can do to make Windows server Failover Cluster failover faster. In this blog I’ll discuss a few tips-n-tricks.

1. Disable NetBIOS over TCP/IP – Unless you need legacy OS compatibility, NetBIOS is doing you nothing but slow you down.  You want to disable NetBIOS in a couple different places:
   1. Every Cluster IP Address resources – Here is the syntax (again, this needs to be set on all IP Address resources).  Note: NetBIOS is disabled on all Cluster IP Addresses in Windows Server 2016 by default.

      Get-ClusterResource “Cluster IP address” | Set-ClusterParameter EnableNetBIOS 0

   2. Base Network Interfaces – In the Advanced TCP/IP Settings, go to the WINS tab, and select “Disable NetBIOS over TCP/IP.  This needs to be done on every network interface.
      
2. Go Pure IPv6 – Going pure IPv6 will give faster failover as a result of optimizations in how Duplication Address Detection (DAD) works in the TCP/IP stack.
3. Avoid IPSec on Servers – Internet Protocol Security (IPsec) is a great security feature, especially for client scenarios. But it comes at a cost, and really shouldn’t be used on servers. Specifically enabling a single IPSec policy will reduce overall network performance by ~30% and significantly delay failover times.

A few things I’ve found you can do to speed up failover and reduce downtime.

Thanks!
Elden Christensen
Principal PM Manager
High-Availability & Storage
Microsoft

In this blog I will discuss what the NetFT Virtual Adapter Performance Filter is and the scenarios when you should or should not enable it.

The Microsoft Failover Cluster Virtual Adapter (NetFT) is a virtual adapter used by the Failover Clustering feature to build fault tolerant communication routes between nodes in a cluster for intra-cluster communication.

When the Cluster Service communicates to another node in the cluster, it sends data down over TCP to the NetFT virtual adapter.  NetFT then sends the data over UDP down to the physical network card, which then sends it over the network to another node.  See the below diagram:

When the data is received by the other node, it follows the same flow.  Up the physical adapter, then to NetFT, and finally up to the Cluster Service.  The NetFT Virtual Adapter Performance Filter is a filter in Windows Server 2012 and Windows Server 2012 R2 which inspects traffic inbound on the physical NIC and then reroutes cluster traffic addressed for NetFT directly to the NetFT driver.  This bypasses the physical NIC UDP / IP stack and delivers increased cluster network performance.

When to Enable the NetFT Virtual Adapter Performance Filter

The NetFT Virtual Adapter Performance Filter is disabled by default.  The filter is disabled because it can cause issues with Hyper-V clusters which have a Guest Cluster running in VMs running on top of them.  Issues have been seen where the NetFT Virtual Adapter Performance Filter in the host incorrectly routes NetFT traffic bound for a guest VM to the host.  This can result in communication issues with the guest cluster in the VM.  More details can be found in this article:

https://support.microsoft.com/en-us/kb/2872325

If you are deploying any workload other than Hyper-V with guest clusters, enabling the NetFT Virtual Adapter Performance Filter will optimize and improve cluster performance.

Windows Server 2016

Due to changes in the networking stack in Windows Server 2016, the NetFT Virtual Adapter Performance Filter has been removed.

Thanks!
Elden Christensen
Principal PM Manager
High-Availability & Storage
Microsoft

Author:
Amitabh Tamhane
Senior Program Manager
Windows Server Microsoft

OS releases: Applicable to Windows Server 2008 R2 or later

Now you can use PowerShell scripts to make any application highly available with Failover Clusters!!!

The Generic Script is a built-in resource type included in Windows Server Failover Clusters. Its advantage is flexibility: you can make applications highly available by writing a simple script. For instance, you can make any PowerShell script highly available! Interested?

We created GenScript in ancient times and it supports only Visual Basic scripts – including Windows Server 2016. This means you can’t directly configure PowerShell as GenScript resource. However, in this blog post, I’ll walk you through a sample Visual Basic script – and associated PS scripts – to build a custom GenScript resource that works well with PowerShell.

Pre-requisites: This blog assumes you have the basic understanding of Windows Server Failover Cluster & built-in resource types.

Disclaimer: Microsoft does not intend to officially support any source code/sample scripts provided as part of this blog. This blog is written only for a quick walk-through on how to run PowerShell scripts using GenScript resource. To make your application highly available, you are expected to modify all the scripts (Visual Basic/PowerShell) as per the needs of your application.

Visual Basic Shell

It so happens that Visual Basic Shell supports calling PowerShell script, then passing parameters and reading output. Here’s a Visual Basic Shell sample that uses some custom Private Properties:

'<your application name> Resource Type

Function Open( )
    Resource.LogInformation "Enter Open()"

    If Resource.PropertyExists("PSScriptsPath") = False Then
        Resource.AddProperty("PSScriptsPath")
    End If

    If Resource.PropertyExists("Name") = False Then
        Resource.AddProperty("Name")
    End If

    If Resource.PropertyExists("Data1") = False Then
        Resource.AddProperty("Data1")
    End If

    If Resource.PropertyExists("Data2") = False Then
        Resource.AddProperty("Data2")
    End If

    If Resource.PropertyExists("DataStorePath") = False Then
        Resource.AddProperty("DataStorePath")
    End If

    '...Result...
    Open = 0

    Resource.LogInformation "Exit Open()"
End Function


Function Online( )
    Resource.LogInformation "Enter Online()"

    '...Check for required private properties...

    If Resource.PropertyExists("PSScriptsPath") = False Then
        Resource.LogInformation "PSScriptsPath is a required private property."
        Online = 1
        Exit Function
    End If
    '...Resource.LogInformation "PSScriptsPath is " & Resource.PSScriptsPath

    If Resource.PropertyExists("Name") = False Then
        Resource.LogInformation "Name is a required private property."
        Online = 1
        Exit Function
    End If
    Resource.LogInformation "Name is " & Resource.Name

    If Resource.PropertyExists("Data1") = False Then
        Resource.LogInformation "Data1 is a required private property."
        Online = 1
        Exit Function
    End If
    '...Resource.LogInformation "Data1 is " & Resource.Data1

    If Resource.PropertyExists("Data2") = False Then
        Resource.LogInformation "Data2 is a required private property."
        Online = 1
        Exit Function
    End If
    '...Resource.LogInformation "Data2 is " & Resource.Data2

    If Resource.PropertyExists("DataStorePath") = False Then
        Resource.LogInformation "DataStorePath is a required private property."
        Online = 1
        Exit Function
    End If
    '...Resource.LogInformation "DataStorePath is " & Resource.DataStorePath

    PScmd = "powershell.exe -file " & Resource.PSScriptsPath & "\PS_Online.ps1 " & Resource.PSScriptsPath & " " & Resource.Name & " " & Resource.Data1 & " " & Resource.Data2 & " " & Resource.DataStorePath

    Dim WshShell
    Set WshShell = CreateObject("WScript.Shell")

    Resource.LogInformation "Calling Online PS script= " & PSCmd
    rv = WshShell.Run(PScmd, , True)
    Resource.LogInformation "PS return value is: " & rv

    '...Translate result from PowerShell ...
    '...1 (True in PS) == 0 (True in VB)
    '...0 (False in PS) == 1 (False in VB)
    If rv = 1 Then
        Resource.LogInformation "Online Success"
        Online = 0
    Else
        Resource.LogInformation "Online Error"
        Online = 1
    End If

    Resource.LogInformation "Exit Online()"
End Function

Function Offline( )
    Resource.LogInformation "Enter Offline()"

    '...Check for required private properties...

    If Resource.PropertyExists("PSScriptsPath") = False Then
        Resource.LogInformation "PSScriptsPath is a required private property."
        Offline = 1
        Exit Function
    End If
    '...Resource.LogInformation "PSScriptsPath is " & Resource.PSScriptsPath

    If Resource.PropertyExists("Name") = False Then
        Resource.LogInformation "Name is a required private property."
        Offline = 1
        Exit Function
    End If
    Resource.LogInformation "Name is " & Resource.Name

    If Resource.PropertyExists("Data1") = False Then
        Resource.LogInformation "Data1 is a required private property."
        Offline = 1
        Exit Function
    End If
    '...Resource.LogInformation "Data1 is " & Resource.Data1

    If Resource.PropertyExists("Data2") = False Then
        Resource.LogInformation "Data2 is a required private property."
        Offline = 1
        Exit Function
    End If
    '...Resource.LogInformation "Data2 is " & Resource.Data2

    If Resource.PropertyExists("DataStorePath") = False Then
        Resource.LogInformation "DataStorePath is a required private property."
        Offline = 1
        Exit Function
    End If
    '...Resource.LogInformation "DataStorePath is " & Resource.DataStorePath

    PScmd = "powershell.exe -file " & Resource.PSScriptsPath & "\PS_Offline.ps1 " & Resource.PSScriptsPath & " " & Resource.Name & " " & Resource.Data1 & " " & Resource.Data2 & " " & Resource.DataStorePath

    Dim WshShell
    Set WshShell = CreateObject("WScript.Shell")

    Resource.LogInformation "Calling Offline PS script= " & PSCmd
    rv = WshShell.Run(PScmd, , True)
    Resource.LogInformation "PS return value is: " & rv

    '...Translate result from PowerShell ...
    '...1 (True in PS) == 0 (True in VB)
    '...0 (False in PS) == 1 (False in VB)
    If rv = 1 Then
        Resource.LogInformation "Offline Success"
        Offline = 0
    Else
        Resource.LogInformation "Offline Error"
        Offline = 1
    End If

    Resource.LogInformation "Exit Offline()"
End Function

Function LooksAlive( )
    '...Result...
    LooksAlive = 0
End Function

Function IsAlive( )
    Resource.LogInformation "Entering IsAlive"

    '...Check for required private properties...

    If Resource.PropertyExists("PSScriptsPath") = False Then
        Resource.LogInformation "PSScriptsPath is a required private property."
        IsAlive = 1
        Exit Function
    End If
    '...Resource.LogInformation "PSScriptsPath is " & Resource.PSScriptsPath

    If Resource.PropertyExists("Name") = False Then
        Resource.LogInformation "Name is a required private property."
        IsAlive = 1
        Exit Function
    End If
    Resource.LogInformation "Name is " & Resource.Name

    If Resource.PropertyExists("Data1") = False Then
        Resource.LogInformation "Data1 is a required private property."
        IsAlive = 1
        Exit Function
    End If
    '...Resource.LogInformation "Data1 is " & Resource.Data1

    If Resource.PropertyExists("Data2") = False Then
        Resource.LogInformation "Data2 is a required private property."
        IsAlive = 1
        Exit Function
    End If
    '...Resource.LogInformation "Data2 is " & Resource.Data2

    If Resource.PropertyExists("DataStorePath") = False Then
        Resource.LogInformation "DataStorePath is a required private property."
        IsAlive = 1
        Exit Function
    End If
    '...Resource.LogInformation "DataStorePath is " & Resource.DataStorePath

    PScmd = "powershell.exe -file " & Resource.PSScriptsPath & "\PS_IsAlive.ps1 " & Resource.PSScriptsPath & " " & Resource.Name & " " & Resource.Data1 & " " & Resource.Data2 & " " & Resource.DataStorePath

    Dim WshShell
    Set WshShell = CreateObject("WScript.Shell")

    Resource.LogInformation "Calling IsAlive PS script= " & PSCmd
    rv = WshShell.Run(PScmd, , True)
    Resource.LogInformation "PS return value is: " & rv

    '...Translate result from PowerShell ...
    '...1 (True in PS) == 0 (True in VB)
    '...0 (False in PS) == 1 (False in VB)
    If rv = 1 Then
        Resource.LogInformation "IsAlive Success"
        IsAlive = 0
    Else
        Resource.LogInformation "IsAlive Error"
        IsAlive = 1
    End If

    Resource.LogInformation "Exit IsAlive()"
End Function

Function Terminate( )
    Resource.LogInformation "Enter Terminate()"

    '...Check for required private properties...

    If Resource.PropertyExists("PSScriptsPath") = False Then
        Resource.LogInformation "PSScriptsPath is a required private property."
        Terminate = 1
        Exit Function
    End If
    '...Resource.LogInformation "PSScriptsPath is " & Resource.PSScriptsPath

    If Resource.PropertyExists("Name") = False Then
        Resource.LogInformation "Name is a required private property."
        Terminate = 1
        Exit Function
    End If
    Resource.LogInformation "Name is " & Resource.Name

    If Resource.PropertyExists("Data1") = False Then
        Resource.LogInformation "Data1 is a required private property."
        Terminate = 1
        Exit Function
    End If
    '...Resource.LogInformation "Data1 is " & Resource.Data1

    If Resource.PropertyExists("Data2") = False Then
        Resource.LogInformation "Data2 is a required private property."
        Terminate = 1
        Exit Function
    End If
    '...Resource.LogInformation "Data2 is " & Resource.Data2

    If Resource.PropertyExists("DataStorePath") = False Then
        Resource.LogInformation "DataStorePath is a required private property."
        Terminate = 1
        Exit Function
    End If
    '...Resource.LogInformation "DataStorePath is " & Resource.DataStorePath

    PScmd = "powershell.exe -file " & Resource.PSScriptsPath & "\PS_Terminate.ps1 " & Resource.PSScriptsPath & " " & Resource.Name & " " & Resource.Data1 & " " & Resource.Data2 & " " & Resource.DataStorePath

    Dim WshShell
    Set WshShell = CreateObject("WScript.Shell")

    Resource.LogInformation "Calling Terminate PS script= " & PSCmd
    rv = WshShell.Run(PScmd, , True)
    Resource.LogInformation "PS return value is: " & rv

    '...Translate result from PowerShell ...
    '...1 (True in PS) == 0 (True in VB)
    '...0 (False in PS) == 1 (False in VB)
    If rv = 1 Then
        Terminate = 0
    Else
        Terminate = 1
    End If

    Resource.LogInformation "Exit Terminate()"
End Function

Function Close( )
    '...Result...
    Close = 0
End Function

Entry Points

In the above sample VB script, the following entry points are defined:

• Open – Ensures all necessary steps complete before starting your application
• Online – Function to start your application
• Offline – Function to stop your application
• IsAlive – Function to validate your application startup and monitor health
• Terminate – Function to forcefully cleanup application state (ex: Error during Online/Offline)
• Close – Ensures all necessary cleanup completes after stopping your application

Each of the above entry points is defined as a function (ex: “Function Online( )”). Failover Cluster then calls these entry point functions as part of the GenScript resource type definition.

Private Properties

For resources of any type, Failover Cluster supports two types of properties:

• Common Properties – Generic properties that can have unique value for each resource
• Private Properties – Custom properties that are unique to that resource type. Each resource of that resource type has these private properties.

When writing a GenScript resource, you need to evaluate if you need private properties. In the above VB sample script, I have defined five sample private properties (only as an example):

• PSScriptsPath – Path to the folder containing PS scripts
• Name
• Data1 – some custom data field
• Data2 – another custom data field
• DataStorePath – path to a common backend store (if any)

The above private properties are shown as example only & you are expected to modify the above VB script to customize it for your application.

PowerShell Scripts

The Visual Basic script simply connects the Failover Clusters’ RHS (Resource Hosting Service) to call PowerShell scripts. You may notice the “PScmd” parameter containing the actual PS command that will be called to perform the action (Online, Offline etc.) by calling into corresponding PS scripts.

For this sample, here are four PowerShell scripts:

• Online.ps1 – To start your application
• Offline.ps1 – To stop your application
• Terminate.ps1 – To forcefully cleanup your application
• IsAlive.ps1 – To monitor health of your application

Example of PS scripts:

Entry Point: Online

Param(
    # Sample properties…
    [Parameter(Mandatory=$true, Position=0)]
    [ValidateNotNullOrEmpty()]
    [string]
    $PSScriptsPath,

    #
    [Parameter(Mandatory=$true, Position=1)]
    [ValidateNotNullOrEmpty()]
    [string]
    $Name,

    #
    [Parameter(Mandatory=$true, Position=2)]
    [ValidateNotNullOrEmpty()]
    [string]
    $Data1,

    #
    [Parameter(Mandatory=$true, Position=3)]
    [ValidateNotNullOrEmpty()]
    [string]
    $Data2,

    #
    [Parameter(Mandatory=$true, Position=4)]
    [ValidateNotNullOrEmpty()]
    [string]
    $DataStorePath
)

$filePath = Join-Path $PSScriptsPath "Output_Online.log"

@"
    Starting Online...
    Name= $Name
    Data1= $Data1
    Data2= $Data2
    DataStorePath= $DataStorePath
"@ | Out-File -FilePath $filePath

$error.clear()

### Do your online script logic here

if ($errorOut -eq $true)
{
    "Error $error" | Out-File -FilePath $filePath -Append
    exit $false
}

"Success" | Out-File -FilePath $filePath -Append
exit $true

Entry Point: Offline

Param(
    # Sample properties…
    [Parameter(Mandatory=$true, Position=0)]
    [ValidateNotNullOrEmpty()]
    [string]
    $PSScriptsPath,

    #
    [Parameter(Mandatory=$true, Position=1)]
    [ValidateNotNullOrEmpty()]
    [string]
    $Name,

    #
    [Parameter(Mandatory=$true, Position=2)]
    [ValidateNotNullOrEmpty()]
    [string]
    $Data1,

    #
    [Parameter(Mandatory=$true, Position=3)]
    [ValidateNotNullOrEmpty()]
    [string]
    $Data2,

    #
    [Parameter(Mandatory=$true, Position=4)]
    [ValidateNotNullOrEmpty()]
    [string]
    $DataStorePath
)

$filePath = Join-Path $PSScriptsPath "Output_Offline.log"

@"
    Starting Offline...
    Name= $Name
    Data1= $Data1
    Data2= $Data2
    DataStorePath= $DataStorePath
"@ | Out-File -FilePath $filePath

$error.clear()

### Do your offline script logic here

if ($errorOut -eq $true)
{
    "Error $error" | Out-File -FilePath $filePath -Append
    exit $false
}

"Success" | Out-File -FilePath $filePath -Append
exit $true

Entry Point: Terminate

Param(
    # Sample properties…
    [Parameter(Mandatory=$true, Position=0)]
    [ValidateNotNullOrEmpty()]
    [string]
    $PSScriptsPath,

    #
    [Parameter(Mandatory=$true, Position=1)]
    [ValidateNotNullOrEmpty()]
    [string]
    $Name,

    #
    [Parameter(Mandatory=$true, Position=2)]
    [ValidateNotNullOrEmpty()]
    [string]
    $Data1,

    #
    [Parameter(Mandatory=$true, Position=3)]
    [ValidateNotNullOrEmpty()]
    [string]
    $Data2,

    #
    [Parameter(Mandatory=$true, Position=4)]
    [ValidateNotNullOrEmpty()]
    [string]
    $DataStorePath
)

$filePath = Join-Path $PSScriptsPath "Output_Terminate.log"

@"
    Starting Terminate...
    Name= $Name
    Data1= $Data1
    Data2= $Data2
    DataStorePath= $DataStorePath
"@ | Out-File -FilePath $filePath

$error.clear()

### Do your terminate script logic here

if ($errorOut -eq $true)
{
    "Error $error" | Out-File -FilePath $filePath -Append
    exit $false
}

"Success" | Out-File -FilePath $filePath -Append
exit $true

Entry Point: IsAlive

Param(
    # Sample properties…
    [Parameter(Mandatory=$true, Position=0)]
    [ValidateNotNullOrEmpty()]
    [string]
    $PSScriptsPath,

    #
    [Parameter(Mandatory=$true, Position=1)]
    [ValidateNotNullOrEmpty()]
    [string]
    $Name,

    #
    [Parameter(Mandatory=$true, Position=2)]
    [ValidateNotNullOrEmpty()]
    [string]
    $Data1,

    #
    [Parameter(Mandatory=$true, Position=3)]
    [ValidateNotNullOrEmpty()]
    [string]
    $Data2,

    #
    [Parameter(Mandatory=$true, Position=4)]
    [ValidateNotNullOrEmpty()]
    [string]
    $DataStorePath
)

$filePath = Join-Path $PSScriptsPath "Output_IsAlive.log"

@"
    Starting IsAlive...
    Name= $Name
    Data1= $Data1
    Data2= $Data2
    DataStorePath= $DataStorePath
"@ | Out-File -FilePath $filePath

$error.clear()

### Do your isalive script logic here

if ($errorOut -eq $true)
{
    "Error $error" | Out-File -FilePath $filePath -Append
    exit $false
}

"Success" | Out-File -FilePath $filePath -Append
exit $true