Lync 2013 CMS failover in coexistence scenario with Skype for business 2015

I had planned a CMS failover to avoid any Lync/Skype issues during weekend SAN maintenance since our Lync 2013 infrastructure was hosted on the Hyper-V platform. 

As we have during our DR tests many times before, I ran the invoke-csmanagementserverfailover cmdelt from the target pool as documented by Microsoft. 

#CMS failover cmdlet

Invoke-csManagementServerFailover -BackupSQLServerFqdn “Lyncpool31.companyname.com” -BackupSQLInstanceName "rtc"  -Force"$false - verbose 

The cmdlet failed and the log showed a strange message, "Error: An error occurred: "System.FormatException" ""1" error categories reported in topology document."

I also noticed I could not use the invoke-csreplication cmdlet due to the error below:

Invoke-CsManagementStoreReplication : ###50020:XdsForceReplication:This central management store is being moved to

another location. No changes can be made until this move is completed.

At line:1 char:1

+ Invoke-CsManagementStoreReplication -ReplicaFqdn servername.com....

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Invoke-CsManagementStoreReplication], SqlException

    + FullyQualifiedErrorId : System.Data.SqlClient.SqlException,Microsoft.Rtc.Management.Xds.InvokeOcsManagementStore

   ReplicationCmdlet

Knowing that the SAN maintenance window was quickly approaching, I opened a SEV A Microsoft Premier case. With Microsoft assistance and roughly 2.5 hours later, it was determined that the cause of our degraded CMS state was due to version mismatch. 

Solution:

The Invoke-csManagementServerFailover cmdlet was run from a Skype server (along with Enable-Cstopology) to force the failover of CMS to the desired Lync 2013 backup pool. 

I hope this post saves someone a few hours of work for something, in retrospect, that is simple.  

McAfee agent 5.x causes network stability issues which impacts Skype

We experienced some intermittent network stability issues recently in Skype. The issues reported were random client disconnections, mainly impacting wireless users, but some wired users were affected to a lessor extent.  Users would experience either poor call quality or a dropped call when the issue would arise.

Our network team found that the root cause was related to spikes in ARP traffic every 30 minutes, resulting in the poor Skype call experience.  It was determined that that a new setting in the recently distributed McAfee agent 5.x was the source of this ARP traffic.

Installing the McAfee 5.x agent management modules into ePO console introduces a new feature called Peer-to-Peer.

•             The two settings in this tab are enabled by Default.

•             The result was all ePO managed systems running the 5.x agent would attempt to reach other managed systems using the default ePO UDP broadcast port 8082, which in turn created ARP broadcast storms.

McAfee advises that the Peer-to-Peer service should not be enabled on networked laptops, yet it is enabled by Default in the new 5.x version.  McAfee support has fielded many complaints from customers who have experienced this issue.  The simple fix is to disable the setting to avoid any network disruption caused by the ARP broadcasts.

Server Side Conversation History and Exchange

Symptom:

After migrating users from Lync Server 2013 to Skype for Business 2015, we noticed High CPU and memory exhaustion and slow server performance on our Front-Ends. In some cases, IM and Presence would fail and calls could not be completed.

Root Cause:
Microsoft observed that there were an excessive number of records in the Front-End local SQL Express LyncLocal\LYSS database caused by our current client policy configuration.

We discovered that the client policies need to be configured correctly when Skype for Business Server is installed.  The server side EnableServerConversationHistory policy set to True, but should be set to FALSE since we do not yet have the Exchange Partner configuration in place. This caused performance problems and has been attributed to a Skype Server system failures we experienced during user migrations.

Also, some clients are using IM archiving, SQL method.  This client policy should be set as follows. ExchangeArchivingPolicy: UseLyncArchivingPolicy

lyss: Lync Storage Service is used to maintain HA within a FE pool. It is a blob database with abstract writes to back-end database. It maintains a copy of the data within Front End servers in the pool temporarily, and delete that data once it has been delivered to the final database server. It is a replacement for MSMQ which was used in previous version of Lync. Therefore, it is part of the Front End servers and it is located under LYNCLOCAL named instance. It is also used for Archiving integration and Unified Contact Center. Currently, it supports Exchange and SQL for archiving.

Resolution

Set the EnableServerConversationHistory client policy to FALSE. Flush the database and restart the server using specific guidance from Microsoft.

Configure Exchange Partner configuration with Skype Server with OAUTH and test before turning on the EnableServerConversationHistory policy.

DNS configuration is preventing you from presenting content

Our organization is currently in the middle of migrating 160K+ users to Skype for Business 2015 from Lync 2013. We have had several users report an issue while trying to view presented content during a meeting (See pic below).

We resolved the error by removing the Skype4B client EndPointConfiguration.cache file and restarting the client. 

Sometimes this file is not automatically refreshed by the system and when there are DNS infrastructure changes, users could be negatively impacted in a few cases.

Procedure to resolve error:

1. Sign out of Skype for Business

2.  Click File Exit

3. Open File Explorer

4. Type in C:\Users\<username>\AppData\Local\Microsoft\Office\15.0\Lync\sip_<username>@companydomain.com 

5. Press Enter

6. Select File: EndpointConfiguratation.cache and delete it.

7.  Launch Skype for Business client and sign in.

There is no Service-Route to copy

Today I received a ticket from a branch location that stated calls to the PSTN were failing.

Looking at the snooper message log I saw a 487 Request Terminated error.



Digging for more clues, I examined the trace and found, "There is no Service-Route to copy"



Next I viewed the SBS Event log and found Event 25051 below.



Puzzled, I began asking questions and found out that the local DNS server was recently changed to a new server. Calls began working fine after I updated the primary DNS server on the NIC adapter settings for the SBS along with the static DNS entry in the AudioCodes gateway config.

Audicodes Lync 2013 IVR manual activation

Hello,

This post is to clear up some confusion concerning the manual deployment for the Lync 2013 AudioCodes IVR.

#Background

I work at a Global company where topology changes can only be performed on designated days. The external vendor that was assisting with the IVR deployment insisted that we had to run the script that auto-provisioned the IVR. The issue with the script is that it invokes the Enable-Cstopology cmdlet which is a big NO in our environment. After dissecting the script, I realized it was just creating a trusted application pool, application, and endpoint.

The steps for manual activation are listed below:


1. Create the trusted app pool (See sample code):

#Creates IVR pool

New-CsTrustedApplicationPool -Identity ivr-pool.mydomain.com -Site 1 -Registrar sba1.mydomain.com -ComputerFqdn ivr1.mydomain.com

#Creates IVR application

New-CsTrustedApplication -ApplicationId "urn:application:ivr" -TrustedApplicationPoolFqdn ivr-pool.mydomain.com  -Port 15001 (See update below)

Enable-CsTopology

#Creates IVR endpoint AD object

New-CsTrustedApplicationEndpoint -ApplicationId ivr -SipAddress "sip:ivr-pool1.mydomain.com " -TrustedApplicationPoolFqdn ivr-pool.mydomain.com  -DisplayName "Audiocodes IVR application endpoint"

2. Create DNS A record for the IVR computer. Following the example, I would create the host (A) record for ivr1.mydomain.com. NOTE: This step was not needed for the Lync 2010 IVR

3. Ensure that network firewalls are allowing TCP 445 for the new IVR server (ivr1.mydomain.com) to communicate with Central Management Store (CMS)

4. Install the Central Management Store replication service, Enable Central Management store replication, and Set certificate. Steps found here, 

https://msdn.microsoft.com/EN-US/library/office/dn466123.aspx

From my experience, the IVR service will not activate until CMS replication is TRUE. 

I hope this helps!!!

UPDATE 1-26-2017: You might want to think about a different port to use for the trusted application other than the default 15001. I had an SBA fail that was also using an IVR server hosted locally, at a branch location. Users reported immediately that inbound calls were not working. The issue was the the parent site, where the users failed over to, was behind a GWAN firewall that did not contain an allow rule for TCP port 15001 as our security team didn't deem it a standard Lync/Skype port. I had to request an emergency firewall rule to allow the port temporarily in order to restore inbound call functionality. Needless to say, we now use TCP port 5061 as the port for the IVR Trusted Application.

Site not found in topology warning

Have you ever published topology only to receive a warning that states a previously removed site was not found in topology?



Fear not as the solution is simple. Open the Enable-CsTopology.xml log file and you will see a clue to the root cause.



Executing the Get-CsKerberosAccountAssignment cmdlet displays the culprit.



Removing the offending kerberos account fixes the issue (i.e., Remove-csKerberosAccountAssignment -identity "site:19").