Avexvss Error 16954 Unable to Initialize Powershell Interface Process Cannot Continue

Today I have been troubleshooting Avamar backups of an Exchange 2013 DAG (Database Availability Groups) setup. The Avamar job would start but no data would be backed up and the job would fail within 15 minutes. Initially VSS was suspected to be the culprit but additional troubleshooting revealed some strange behavior of the Avamar client in which the individual plugins would keep waiting for a status message that would never arrive.

The setup is simple: three Windows 2008 R2 SP1 servers, Exchange 2013 Update 5 with Database Availability Groups and Avamar 7.0.2 as the backup software and target. The error that pops up looks like this:

Or in text:

            2014-12-12 11:43:39 avexvss Info : Number of remote clients: [2] 2014-12-12 11:43:39 avexvss Info : Forwarding [snapup] request to: fe80::708a:69e2:fae6:e741 port:28002 2014-12-12 11:43:39 avexvss Info : Remote client: EXCHANGE01,fe80::708a:69e2:fae6:e741, Status: SENT_SUBWORKORDER 2014-12-12 11:43:40 avexvss Info : Forwarding [snapup] request to: 10.11.11.113 port:28002 2014-12-12 11:43:40 avexvss Info : Remote client: EXCHANGE03,10.11.11.113, Status: SENT_SUBWORKORDER 2014-12-12 11:43:40 avexvss Info : Waiting for back-end clients to start sub-workorders ... 2014-12-12 11:43:57 avexvss Info : ...appending IP address fe80::614f:26fc:ad6f:4502%14. 2014-12-12 11:43:57 avexvss Info : ...appending IP address 169.254.3.159. 2014-12-12 11:43:57 avexvss Info : ...appending IP address 10.11.11.113. 2014-12-12 11:43:57 avexvss Info : Remote client: EXCHANGE01,fe80::708a:69e2:fae6:e741, Status: SUBWORKORDER_STARTED 2014-12-12 11:44:01 avexvss Warning : Primary IP Address wasn't found for fe80::708a:69e2:fae6:e741. 2014-12-12 11:44:01 avexvss Info : Remote client: EXCHANGE01,fe80::708a:69e2:fae6:e741, Status: SUBWORKORDER_STARTED 2014-12-12 11:48:40 avexvss Info : Initiating VSS Snapshot creation on back-end clients. 2014-12-12 11:48:40 avexvss Error : Remote client EXCHANGE03 failed to start its subworkorder. 2014-12-12 11:48:40 avexvss Info : Forwarding [cancel] request to: 10.11.11.113 port:28002 2014-12-12 11:48:41 avexvss Info : Remote client: EXCHANGE03,10.11.11.113, Status: SENT_CANCEL_REQUEST 2014-12-12 11:48:41 avexvss Info : Initiating VSS Snapshot creation on remote client EXCHANGE01. 2014-12-12 11:48:42 avexvss Info : Forwarding [update_fc_status] request to: fe80::708a:69e2:fae6:e741 port:28002 2014-12-12 11:48:42 avexvss Info : Remote client: EXCHANGE01,fe80::708a:69e2:fae6:e741, Status: VSS_SYNCHRONIZATION 2014-12-12 11:48:42 avexvss Info : Waiting for back-end clients to create VSS Snapshot of their components ... 2014-12-12 11:48:55 avexvss Warning : Primary IP Address wasn't found for fe80::708a:69e2:fae6:e741. 2014-12-12 11:48:55 avexvss Info : Remote client: EXCHANGE01,fe80::708a:69e2:fae6:e741, Status: CREATED_VSS_SNAPSHOT 2014-12-12 11:51:41 avexvss Error : The plugin on remote client EXCHANGE01 (fe80::708a:69e2:fae6:e741) terminated with error code 10019: Externally cancelled by Administrator. Please examine the logs on client EXCHANGE01 for details.          

During the backup the Avamar connects to the DAG, lists the databases, sends workorders to the individual clients (SENT_SUBWORKORDER). You can see that only EXCHANGE01 returns a SUBWORKORDER_STARTED; EXCHANGE03 doesn't respond.  After a while Avamar notices EXCHANGE03 doesn't respond and sends a cancel request to Exchange03. Exchange01 also cancels a while later. At this point the VSSWriter on EXCHANGE01 shows a Stable state with a Retryable error. The EXCHANGE03 VSS writer doesn't show an error.

You might be tempted to troubleshoot the server with the VSS error, except there isn't really an error on that server! See the drilldown Avamar log of EXCHANGE01:

            2014-12-12 11:48:55 avexvss Info : Waiting for a wakeup message from the calling agent to proceed with backup. 2014-12-12 11:53:55 avexvss Error : Timed out while waiting for the update status message. 2014-12-12 11:53:56 avexvss Error : ERROR: Selected writer 'Microsoft Exchange Writer' reported an error! - Status: 1 (VSS_WS_STABLE) - Writer Failure code: 0x800423f3 (VSS_E_WRITERERROR_RETRYABLE) - Writer ID: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7} - Instance ID: {b68d345a-b123-43fa-a3f9-581b74bbdf29} 2014-12-12 11:53:56 avexvss Error : Exception occurred in BackupCompleteEx 2014-12-12 11:53:56 avexvss Error : Unable to complete backup.          

And on the "unresponsive server" EXCHANGE03:

            2014-12-12 11:43:56 avexvss Info : Waiting for a VSS snapshot synchronization event from the initiating plugin. 2014-12-12 11:48:40 avexvss Info : Cancel request received          

We were a bit puzzled by the IP addresses displayed in the initial Avamar log: IPv6 is disabled on the servers and the APIPA addresses didn't make any sense either. So while I confirmed some settings my colleague Daniel Hass was googling these errors and ended up on this very helpful blogpost from Dan Anstis. We ran these suggestions by EMC Support whom confirmed we had to adjust the timeout values. We ended up with the following avexvss.cmd file on the "shared /var" location created during the initial DAG client configuration.

            --federated --vss-snapshot-timeout=400 --subworkorder-timeout=400 --clusternode=EXCHANGE01(10.11.11.111) --clusternode=EXCHANGE02(10.11.11.112) --clusternode=EXCHANGE03(10.11.11.113)          

We also had to create identical avexvss.cmd files in the "local /var" folder (e.g. C:\Program Files\avs\var\) that only contained the –vss-snapshot-timeout and –subworkorder-timeout switches. Seems redundant to me but I didn't have time to wait for the back-up to complete and then try it again. So give it a whirl without if you want to try the leanest possible solution…

The above config file fixed the issue: no longer did the Avamar client try to use non-existing or loopback IP addresses and the new configured timeouts gave the subworkorder enough time to start and report back. VSS got to work, wasn't shot down by the parent Avamar process and soon after the consistency checks started running, followed by the actual backup.

Let me know if you run into the same issue and if this solves it for you!

Jon

Jon is a senior consultant in the IT infrastructure and cloud domain, specializing in storage, SAN, backup and virtualization. Outside of IT you can find him on a snowboard or tinkering with technology.

162 posts

coxgenes1951.blogspot.com

Source: https://faststorage.eu/avamar-exchange-2013-dag-backup-fails-waiting-for-vss/

Share this post