Lync 2013 CMS failover in coexistence scenario with Skype for business 2015

I had planned a CMS failover to avoid any Lync/Skype issues during weekend SAN maintenance since our Lync 2013 infrastructure was hosted on the Hyper-V platform. 

As we have during our DR tests many times before, I ran the invoke-csmanagementserverfailover cmdelt from the target pool as documented by Microsoft. 

#CMS failover cmdlet

Invoke-csManagementServerFailover -BackupSQLServerFqdn “Lyncpool31.companyname.com” -BackupSQLInstanceName "rtc"  -Force"$false - verbose 

The cmdlet failed and the log showed a strange message, "Error: An error occurred: "System.FormatException" ""1" error categories reported in topology document."

I also noticed I could not use the invoke-csreplication cmdlet due to the error below:

Invoke-CsManagementStoreReplication : ###50020:XdsForceReplication:This central management store is being moved to

another location. No changes can be made until this move is completed.

At line:1 char:1

+ Invoke-CsManagementStoreReplication -ReplicaFqdn servername.com....

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Invoke-CsManagementStoreReplication], SqlException

    + FullyQualifiedErrorId : System.Data.SqlClient.SqlException,Microsoft.Rtc.Management.Xds.InvokeOcsManagementStore

   ReplicationCmdlet

Knowing that the SAN maintenance window was quickly approaching, I opened a SEV A Microsoft Premier case. With Microsoft assistance and roughly 2.5 hours later, it was determined that the cause of our degraded CMS state was due to version mismatch. 

Solution:

The Invoke-csManagementServerFailover cmdlet was run from a Skype server (along with Enable-Cstopology) to force the failover of CMS to the desired Lync 2013 backup pool. 

I hope this post saves someone a few hours of work for something, in retrospect, that is simple.  

McAfee agent 5.x causes network stability issues which impacts Skype

We experienced some intermittent network stability issues recently in Skype. The issues reported were random client disconnections, mainly impacting wireless users, but some wired users were affected to a lessor extent.  Users would experience either poor call quality or a dropped call when the issue would arise.

Our network team found that the root cause was related to spikes in ARP traffic every 30 minutes, resulting in the poor Skype call experience.  It was determined that that a new setting in the recently distributed McAfee agent 5.x was the source of this ARP traffic.

Installing the McAfee 5.x agent management modules into ePO console introduces a new feature called Peer-to-Peer.

•             The two settings in this tab are enabled by Default.

•             The result was all ePO managed systems running the 5.x agent would attempt to reach other managed systems using the default ePO UDP broadcast port 8082, which in turn created ARP broadcast storms.

McAfee advises that the Peer-to-Peer service should not be enabled on networked laptops, yet it is enabled by Default in the new 5.x version.  McAfee support has fielded many complaints from customers who have experienced this issue.  The simple fix is to disable the setting to avoid any network disruption caused by the ARP broadcasts.

Server Side Conversation History and Exchange

Symptom:

After migrating users from Lync Server 2013 to Skype for Business 2015, we noticed High CPU and memory exhaustion and slow server performance on our Front-Ends. In some cases, IM and Presence would fail and calls could not be completed.

Root Cause:
Microsoft observed that there were an excessive number of records in the Front-End local SQL Express LyncLocal\LYSS database caused by our current client policy configuration.

We discovered that the client policies need to be configured correctly when Skype for Business Server is installed.  The server side EnableServerConversationHistory policy set to True, but should be set to FALSE since we do not yet have the Exchange Partner configuration in place. This caused performance problems and has been attributed to a Skype Server system failures we experienced during user migrations.

Also, some clients are using IM archiving, SQL method.  This client policy should be set as follows. ExchangeArchivingPolicy: UseLyncArchivingPolicy

lyss: Lync Storage Service is used to maintain HA within a FE pool. It is a blob database with abstract writes to back-end database. It maintains a copy of the data within Front End servers in the pool temporarily, and delete that data once it has been delivered to the final database server. It is a replacement for MSMQ which was used in previous version of Lync. Therefore, it is part of the Front End servers and it is located under LYNCLOCAL named instance. It is also used for Archiving integration and Unified Contact Center. Currently, it supports Exchange and SQL for archiving.

Resolution

Set the EnableServerConversationHistory client policy to FALSE. Flush the database and restart the server using specific guidance from Microsoft.

Configure Exchange Partner configuration with Skype Server with OAUTH and test before turning on the EnableServerConversationHistory policy.