WSUS Replicas Syncing as Autonomous

hey everybody. 

we run master / replica deployment of wsus, 1 master server , around 40 downstream replica servers on each of our satellite sites. of around week ago, there have been issues sync. downstream replica servers reporting have synced correctly , no errors, if @ console on master server no clients have reported status since issue occurred. 

looking @ downstream servers option in master mmc, downstream servers have mode set 'unknown' in list of servers. details of each server entry show mode: autonomous , last rollup: not yet rolled up. 

settings both , downstream servers when solution working properly. can confirm: 

- each downstream server has 'this server replica of upstream server' checked

- 'roll status replica downstream servers' checked. 

workstations reporting in status downstream servers fine. issue status being rolled master server. there no nlb, , database external sql 2005 db. 

so far, fix, we've tried:

• rolled installed patches no difference.
• reindexed database, , run server clean wizard.
• set downstream server standalone replica
• we went on rebuilt entire upstream wsus server - fresh database , everything. approvals , computer sets replicated setting downstream of 1 of existing servers setting master. same issue. 

honestly can't think of has changed in environment have caused this. softwaredistribution.log shows errors, searching them unfortunately isn't bearing fruit. can understand please?

2012-10-24 14:19:22.723 utc warning w3wp.5 soaputilities.createexception throwexception: actor = http://downstreamserver/clientwebservice/client.asmx, id=207d01e3-2db6-4aa9-beea-cddab0da3340, errorcode=configchanged, message=, client=dbbce75e-06cc-4987-a03e-25fefd04d349
2012-10-24 14:19:28.739 utc warning w3wp.8 soaputilities.createexception throwexception: actor = http://downstreamserver/clientwebservice/client.asmx, id=2cffd9a3-590a-41f4-8696-081ee88a9fbe, errorcode=configchanged, message=, client=289d39f4-9734-441d-89dd-7461073a65f7
2012-10-24 14:19:33.270 utc info w3wp.1 threadentry isapiruntime.processrequest
2012-10-24 14:19:33.286 utc change w3wp.1 admindataaccess.startsubscriptionmanually synchronization manually started
2012-10-24 14:19:34.708 utc info w3wp.19 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-10-24 14:19:34.708 utc info w3wp.26 threadentry threadhelper.threadstart
2012-10-24 14:19:34.708 utc info w3wp.26 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-10-24 14:19:34.708 utc info w3wp.26 changenotificationdispatcher.internaleventhandler get event configurationchange dispatchmanager
2012-10-24 14:19:34.708 utc info w3wp.20 client.onconfigurationchange creating new clientimplementation because db configuration changed
2012-10-24 14:19:34.708 utc info w3wp.20 clientimplementation..ctor initializing clientwebservice processid = 3324, process start time = 24/10/2012 13:55:09, product version = 3.2.7600.251
2012-10-24 14:19:34.911 utc info w3wp.11 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-10-24 14:19:34.911 utc info w3wp.24 threadentry threadhelper.threadstart
2012-10-24 14:19:34.911 utc info w3wp.24 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-10-24 14:19:34.911 utc info w3wp.24 revisionidcachechangenotificationdispatcher.internaleventhandler get event configurationchange dispatchmanager
2012-10-24 14:19:34.989 utc info wsusservice.7 suseventdispatcher.triggerevent triggerevent called notificationeventname: catalogsyncagent, eventinfo: 
2012-10-24 14:19:34.989 utc info wsusservice.23 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: catalogsyncagent
2012-10-24 14:19:34.989 utc info wsusservice.7 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-10-24 14:19:34.989 utc info wsusservice.23 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-10-24 14:19:35.020 utc info wsusservice.17 threadentry threadhelper.threadstart
2012-10-24 14:19:35.020 utc info wsusservice.17 eventlogeventreporter.reportevent eventid=382,type=information,category=synchronization,message=a manual synchronization started.
2012-10-24 14:19:35.051 utc info wsusservice.17 catalogsyncagentcore.executesyncprotocol server id 80e4e89e-f75d-4ae5-a6ff-b6a8e43fa901
2012-10-24 14:19:35.067 utc info wsusservice.23 webservicecommunicationhelper.configchangedhandler update server configuration has changed. reporting service: stats.update.microsoft.com, syncfrommu: false
2012-10-24 14:19:35.629 utc change wsusservice.17 admindataaccess.executespsetconfiguration wsus configuration has been changed
2012-10-24 14:19:35.645 utc info wsusservice.17 catalogsyncagentcore.syncconfigupdatesfromuss category sync: filter: <filter syncanchor="175814,2012-10-24 11:59:25.291" getconfig="1"></filter>
2012-10-24 14:19:35.770 utc info wsusservice.17 catalogsyncagentcore.syncconfigupdatesfromuss need 0 config updates, 0 new
2012-10-24 14:19:35.770 utc info wsusservice.17 catalogsyncagentcore.syncconfigupdatesfromuss category sync: new config anchor: 175814,2012-10-24 14:19:35.678
2012-10-24 14:19:35.770 utc info wsusservice.17 catalogsyncagentcore.executesyncprotocol catalog sync: filter: <filter syncanchor="175814,2012-10-24 11:59:26.494"><languagelist><language delta="1" value="1033" /><language delta="1" value="1034" /></languagelist></filter>
2012-10-24 14:19:35.801 utc info w3wp.12 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-10-24 14:19:35.801 utc info w3wp.21 threadentry threadhelper.threadstart
2012-10-24 14:19:35.801 utc info w3wp.21 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-10-24 14:19:35.895 utc info wsusservice.17 catalogsyncagentcore.executesyncprotocol catalog sync: new anchor: 175814,2012-10-24 14:19:35.944
2012-10-24 14:19:35.895 utc info wsusservice.17 catalogsyncagentcore.executesyncprotocol need 0 updates, 0 new
2012-10-24 14:19:35.895 utc info wsusservice.17 catalogsyncagentcore.executesyncprotocol total inserted updates: 0
2012-10-24 14:19:35.895 utc info wsusservice.17 catalogsyncagentcore.replicasync deploymentsync: starting delta sync
2012-10-24 14:19:35.911 utc info wsusservice.17 catalogsyncagentcore.replicasync deploymentsync: anchors: "175814,2012-10-24 11:59:26.494" "175814,2012-10-24 14:19:35.944"
2012-10-24 14:19:36.458 utc info wsusservice.17 catalogsyncagentcore.replicasync deploymentsync: new anchor: 175814,2012-10-24 14:19:35.944
2012-10-24 14:19:36.473 utc info wsusservice.17 catalogsyncagentcore.replicasync deploymentsync: target groups count: 814
2012-10-24 14:19:36.473 utc info wsusservice.17 catalogsyncagentcore.replicasync deploymentsync: deployments count: 0
2012-10-24 14:19:36.473 utc info wsusservice.17 catalogsyncagentcore.replicasync deploymentsync: deleted deployments count: 0
2012-10-24 14:19:36.473 utc info wsusservice.17 catalogsyncagentcore.replicasync deploymentsync: accepted eula count: 18
2012-10-24 14:19:36.473 utc info wsusservice.17 catalogsyncagentcore.replicasync deploymentsync: hidden update count: 4050
2012-10-24 14:19:36.473 utc info wsusservice.17 catalogsyncagentcore.replicasync deploymentsync: processing target groups
2012-10-24 14:19:36.489 utc info w3wp.18 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-10-24 14:19:36.489 utc info wsusservice.17 catalogsyncagentcore.replicasync deploymentsync: processing accepted eulas
2012-10-24 14:19:36.489 utc info w3wp.14 threadentry threadhelper.threadstart
2012-10-24 14:19:36.489 utc info w3wp.14 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-10-24 14:19:36.489 utc info w3wp.17 suseventdispatcher.registereventhandler registereventhandler called notificationeventname: configurationchange
2012-10-24 14:19:36.536 utc info wsusservice.17 catalogsyncagentcore.replicasync deploymentsync: processing deleted deployments
2012-10-24 14:19:36.536 utc info wsusservice.17 catalogsyncagentcore.replicasync deploymentsync: processing new deployments
2012-10-24 14:19:36.536 utc info wsusservice.17 catalogsyncagentcore.replicasync deploymentsync: processing hidden updates
2012-10-24 14:19:39.130 utc info wsusservice.17 catalogsyncagentcore.replicasync deploymentsync: completed
2012-10-24 14:19:39.161 utc change wsusservice.17 admindataaccess.executespresumealldownloads downloading retried
2012-10-24 14:19:39.176 utc info wsusservice.17 catalogsyncagentcore.updatesyncresultandgeneratereportingevent catalogsyncthreadprocess: report subscription succeeded
2012-10-24 14:19:39.176 utc info wsusservice.17 eventlogeventreporter.reportevent eventid=384,type=information,category=synchronization,message=synchronization completed successfully.
2012-10-24 14:19:39.192 utc info wsusservice.26 catalogsyncagent.waituntilsyncfinishedorcancelled agent signalled done.
2012-10-24 14:19:39.192 utc info wsusservice.26 catalogsyncagent.setsubscriptionstatewithretry firing event syncfinish...
2012-10-24 14:19:39.192 utc info wsusservice.26 catalogsyncagent.wakeupworkerthreadproc found no more jobs. catalogsyncagent quits run rollup before terminating ...
2012-10-24 14:19:39.208 utc info wsusservice.26 catalogsyncagent.updateserverhealthstatusbasedonerror serverhealth: updating server health component: catalogsyncagent, marking not running
2012-10-24 14:19:39.708 utc info w3wp.19 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-10-24 14:19:39.708 utc info w3wp.28 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-10-24 14:19:39.708 utc info w3wp.28 changenotificationdispatcher.internaleventhandler get event configurationchange dispatchmanager
2012-10-24 14:19:39.723 utc info w3wp.20 client.onconfigurationchange creating new clientimplementation because db configuration changed
2012-10-24 14:19:39.723 utc info w3wp.20 clientimplementation..ctor initializing clientwebservice processid = 3324, process start time = 24/10/2012 13:55:09, product version = 3.2.7600.251
2012-10-24 14:19:39.911 utc info w3wp.11 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-10-24 14:19:39.911 utc info w3wp.26 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-10-24 14:19:39.911 utc info w3wp.26 revisionidcachechangenotificationdispatcher.internaleventhandler get event configurationchange dispatchmanager
2012-10-24 14:19:40.005 utc info wsusservice.7 suseventdispatcher.triggerevent triggerevent called notificationeventname: rollupagent, eventinfo: 
2012-10-24 14:19:40.005 utc info wsusservice.13 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: rollupagent
2012-10-24 14:19:40.083 utc info wsusservice.7 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-10-24 14:19:40.083 utc info wsusservice.13 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-10-24 14:19:40.145 utc info wsusservice.13 webservicecommunicationhelper.configchangedhandler update server configuration has changed. reporting service: stats.update.microsoft.com, syncfrommu: false
2012-10-24 14:19:40.801 utc info w3wp.12 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-10-24 14:19:40.801 utc info w3wp.29 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-10-24 14:19:41.489 utc info w3wp.18 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-10-24 14:19:41.489 utc info w3wp.30 threadentry threadhelper.threadstart
2012-10-24 14:19:41.489 utc info w3wp.30 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-10-24 14:19:41.489 utc info w3wp.17 suseventdispatcher.registereventhandler registereventhandler called notificationeventname: configurationchange
2012-10-24 14:19:41.505 utc info w3wp.17 suseventdispatcher.registereventhandler registereventhandler called notificationeventname: configurationchange
2012-10-24 14:19:43.098 utc warning wsusservice.28 dbconnection.onreceivinginfomessage null value eliminated aggregate or other set operation.
2012-10-24 14:19:43.255 utc warning w3wp.5 soaputilities.createexception throwexception: actor = http://downstreamserver/clientwebservice/client.asmx, id=2886b9cc-fd2c-4155-b399-baaff8eb230d, errorcode=configchanged, message=, client=289d39f4-9734-441d-89dd-7461073a65f7
2012-10-24 14:19:43.301 utc info wsusservice.28 rollupeventreporter.senddownstreamserverrollupchunkwithretry rollupdownstreamservers failed. retrying...
2012-10-24 14:19:43.301 utc error wsusservice.28 soaputilities.logexception uss throwexception: actor = , method = ?, id=00000000-0000-0000-0000-000000000000, errorcode=internalservererror, message=
   at microsoft.updateservices.internal.soaputilities.logexception(soapexception e)
   at microsoft.updateservices.internal.webservicecommunicationhelper.processwebserviceproxyexception(soaphttpclientprotocol& webserviceobject, exception exceptioninfo)
   at microsoft.updateservices.internal.reporting.rollup.rollupeventreporter.senddownstreamserverrollupchunkwithretry(cookie cookie, downstreamserverrollupinfo[] chunk)
   at microsoft.updateservices.internal.reporting.rollup.rollupeventreporter.rollupdownstreamservers(downstreamserverrollupinfo[] downstreamserversinfo, int32 maxchunksize)
   at microsoft.updateservices.internal.reporting.rollup.updateservicesrollup.dodownstreamserversinformationrollup()
   at microsoft.updateservices.internal.reporting.rollup.updateservicesrollup.dorollup()
   at microsoft.updateservices.internal.reporting.rollup.rollupagent.wakeupworkerthreadproc()
   at system.threading.threadhelper.threadstart_context(object state)
   at system.threading.executioncontext.runtrycode(object userdata)
   at system.runtime.compilerservices.runtimehelpers.executecodewithguaranteedcleanup(trycode code, cleanupcode backoutcode, object userdata)
   at system.threading.executioncontext.runinternal(executioncontext executioncontext, contextcallback callback, object state)
   at system.threading.executioncontext.run(executioncontext executioncontext, contextcallback callback, object state)
   at system.threading.threadhelper.threadstart()
2012-10-24 14:19:43.301 utc error wsusservice.28 rollupeventreporter.senddownstreamserverrollupchunkwithretry rollupdownstreamservers cant retried. exception: server unable process request. ---> error: invalid lastrolluptime passed in dss rollup
   at microsoft.updateservices.internal.reporting.rollup.rollupeventreporter.senddownstreamserverrollupchunkwithretry(cookie cookie, downstreamserverrollupinfo[] chunk)
   at microsoft.updateservices.internal.reporting.rollup.rollupeventreporter.rollupdownstreamservers(downstreamserverrollupinfo[] downstreamserversinfo, int32 maxchunksize)
   at microsoft.updateservices.internal.reporting.rollup.updateservicesrollup.dodownstreamserversinformationrollup()
   at microsoft.updateservices.internal.reporting.rollup.updateservicesrollup.dorollup()
   at microsoft.updateservices.internal.reporting.rollup.rollupagent.wakeupworkerthreadproc()
   at system.threading.threadhelper.threadstart_context(object state)
   at system.threading.executioncontext.runtrycode(object userdata)
   at system.runtime.compilerservices.runtimehelpers.executecodewithguaranteedcleanup(trycode code, cleanupcode backoutcode, object userdata)
   at system.threading.executioncontext.runinternal(executioncontext executioncontext, contextcallback callback, object state)
   at system.threading.executioncontext.run(executioncontext executioncontext, contextcallback callback, object state)
   at system.threading.threadhelper.threadstart()
2012-10-24 14:19:43.301 utc error wsusservice.28 updateservicesrollup.dodownstreamserversinformationrollup failed rollup server information child servers.
   at microsoft.updateservices.internal.reporting.rollup.updateservicesrollup.dodownstreamserversinformationrollup()
   at microsoft.updateservices.internal.reporting.rollup.updateservicesrollup.dorollup()
   at microsoft.updateservices.internal.reporting.rollup.rollupagent.wakeupworkerthreadproc()
   at system.threading.threadhelper.threadstart_context(object state)
   at system.threading.executioncontext.runtrycode(object userdata)
   at system.runtime.compilerservices.runtimehelpers.executecodewithguaranteedcleanup(trycode code, cleanupcode backoutcode, object userdata)
   at system.threading.executioncontext.runinternal(executioncontext executioncontext, contextcallback callback, object state)
   at system.threading.executioncontext.run(executioncontext executioncontext, contextcallback callback, object state)
   at system.threading.threadhelper.threadstart()
2012-10-24 14:19:43.317 utc error wsusservice.28 updateservicesrollup.dorollup failed rollup dss information. exception=system.web.services.protocols.soapexception: server unable process request. ---> error: invalid lastrolluptime passed in dss rollup
   at system.web.services.protocols.soaphttpclientprotocol.readresponse(soapclientmessage message, webresponse response, stream responsestream, boolean asynccall)
   at system.web.services.protocols.soaphttpclientprotocol.invoke(string methodname, object[] parameters)
   at microsoft.updateservices.reporting.webservice.rollupdownstreamservers(cookie cookie, datetime clienttime, downstreamserverrollupinfo[] downstreamservers)
   at microsoft.updateservices.internal.reporting.rollup.rollupeventreporter.senddownstreamserverrollupchunkwithretry(cookie cookie, downstreamserverrollupinfo[] chunk)
   at microsoft.updateservices.internal.reporting.rollup.rollupeventreporter.rollupdownstreamservers(downstreamserverrollupinfo[] downstreamserversinfo, int32 maxchunksize)
   at microsoft.updateservices.internal.reporting.rollup.updateservicesrollup.dodownstreamserversinformationrollup()
   at microsoft.updateservices.internal.reporting.rollup.updateservicesrollup.dorollup(). rollup terminated.
   at microsoft.updateservices.internal.reporting.rollup.updateservicesrollup.dorollup()
   at microsoft.updateservices.internal.reporting.rollup.rollupagent.wakeupworkerthreadproc()
   at system.threading.threadhelper.threadstart_context(object state)
   at system.threading.executioncontext.runtrycode(object userdata)
   at system.runtime.compilerservices.runtimehelpers.executecodewithguaranteedcleanup(trycode code, cleanupcode backoutcode, object userdata)
   at system.threading.executioncontext.runinternal(executioncontext executioncontext, contextcallback callback, object state)
   at system.threading.executioncontext.run(executioncontext executioncontext, contextcallback callback, object state)
   at system.threading.threadhelper.threadstart()
2012-10-24 14:19:50.520 utc warning w3wp.1 soaputilities.createexception throwexception: actor = http://downstreamserver/clientwebservice/client.asmx, id=5cc02433-9b42-435a-97b3-ade7cdab3961, errorcode=configchanged, message=, client=d9e8cd1d-33a3-4341-bf38-a3f0d4b68195

"autonomous" != "replica", , reporting rollup works replica servers.

aye, symptom of problem. downstream servers set replicas, yet looking @ upstream server mmc under 'downstream servers' mode showing 'unknown' in 'mode' column of server list. in detail pane, 'mode' showing 'autonomous'. neither of configured...

delete downstream server registration using console of upstream server, , initiate server synchronization downstream server, build new downstream server registration.

---

lawrence garvin, m.s., mcitp:ea, mcdba, mcsa
solarwinds head geek
microsoft mvp - software distribution (2005-2012)
mvp profile: http://mvp.support.microsoft.com/profile/lawrence.garvin

Windows Server  >  WSUS