Tuesday, October 30, 2018

SCCM - Pull Distribution Point - failed to get IsDPSiteSystemCertInstalled


One more problem, 1 PullDP does not seems to work correctly

Analysing DistMgr.log we can see:
~Sleep 65 seconds... 
Failed to open certificate store "MYPULLDP.MYDOMAIN.NET\SMS" (0x80070035) 
Failed to get IsDPSiteSytemCertInstalled method 
Failed to check if the sitesystemcert installation state for ["Display=\\MYPULLDP.MYDOMAIN.NET\"]MSWNET:["SMS_SITE=XXX"]\\MYPULLDP.MYDOMAIN.NET\, Error - 0x80041002  
InstallDPSiteSystemCert failed to install the sitesystem certificate for ["Display=\\MYPULLDP.MYDOMAIN.NET\"]MSWNET:["SMS_SITE=XXX"]\\MYPULLDP.MYDOMAIN.NET\, Error - 0x80041002   
~Currently using 0 out of 20 allowed package processing threads. 
~Sleep 60 seconds... 



















So I use ApiMonitor to find error fastly











We can see SMS_DistributionPoint is object with problem because Method IsDPSiteSystemCertInstalled seems not found ( 0x80041002)


Comparing PullDP with problem and  a PullDP working as below we can see some Methods missing






































$smsdp_ERROR = Get-WmiObject -ComputerName PULLDP_WITH_ERRORS -Namespace root\sccmdp -List SMS_DistributionPoint
$smsdp_ERROR.Methods | select name | sort name

Name
----
AddDeltaFile
AddFile
AddFiles
CancelPullDPJob
CleanupContentLib
ConfigurePXE
CreatePackageShare
CreateRegistryKey
DeleteDPHealthMonTask
DisableDPUsageStatsGathering
EnableDPUsageStatsGathering
ExpandContent
ExpandPXEImage
FinalizeContent
GetContentHash
ImportToSIS
InstallPullDP
InstallPullDPEx
InstallPXE
IsClientUpgradeRequired
NotifyPullDP
RegisterDPHealthMonTask
RemoveContent
SetDefaultAccessPermission
StartDPHealthMonTask
UninstallPullDP
UpdatePackage
UpdatePackageContentReferences

$smsdp_OK = Get-WmiObject -ComputerName PULLDP_OK -Namespace root\sccmdp -List SMS_DistributionPoint
$smsdp_OK.Methods | select name | sort name

Name
----
AddDeltaFile
AddFile
AddFiles
CancelPullDPJob
CleanupContentLib
ConfigureIISModules
ConfigurePXE
CreatePackageShare
CreateRegistryKey
DeleteDPHealthMonTask
DisableDPUsageStatsGathering
EnableDPUsageStatsGathering
ExpandContent
ExpandPXEImage
FinalizeContent
GetContentHash
ImportToSIS
InstallDPSiteSystemCert
InstallPullDP
InstallPullDPEx
InstallPXE
IsClientUpgradeRequired
IsDPSiteSytemCertInstalled
NotifyPullDP
RegisterDPHealthMonTask
RemoveContent
SetDefaultAccessPermission
StartDPHealthMonTask
UninstallPullDP
UpdateIISBinding
UpdatePackage
UpdatePackageContentReferences




Verifying SMS_DP$\sms\bin\smsdpprov.mof file on PullDP we can see these methods exist.

We can also see all files in SMS_DP$\sms\bin are last Distribution Point binaries comparing to another working DP. Problem seems smsdpprov.mof have not been compiled OR wmi have been recompiled WITHOUT compiling this .mof. To verify that we can do it manually using command:

mofcomp smsdpprov.mof  (french computer so display in french below but it exactly same in english)






















below before and after mofcomp. After mof compilation new methods appears






































Just after recompilation you have some failed but now "package thread limit: 20" and some minutes after no more problem (just wait )

Failed to open certificate store "MYPULLDP.MYDOMAIN.NET\SMS" (0x80070035) 
~Package Thread Limit: 20 






















1 more problem solved



SCCM Recover Site - Lines displayed in "Show Install Status"

Just for information below lines taken from SCCM "Show Install Status" after I do a recover (\Administration\Overview\Site Configuration\Sites / right click / "Show Install Status")
























[Passed]:Begin prerequisite checks
[Passed]:Verifies that the site server computer account has administrative rights on the SQL Server and management point.
[Passed]:Check Server Service (LanmanServer) is running.
[Passed]:Verifies that the site server operating system meets the minimum requirement for site server installation. https://go.microsoft.com/fwlink/?linkid=841654
[Passed]:Verifies that the computer specified for installation is a member of a Windows domain.
[Passed]:Checks that the site server computer has sufficient available disk space to install the site server.
[Passed]:Checks if a system restart is pending.
[Passed]:Checking unsupported Read-Only Domain Controller on site server.
[Passed]:Checking Site Server FQDN Length.
[Passed]:Verifies that the Microsoft Core XML Services (MSXML) version 6.0 or later libraries are installed.
[Passed]:Verifies that the Microsoft Remote Differential Compression (RDC) library is registered on the computer specified for Configuration Manager site server installation.
[Passed]:Checking Windows Installer Version >= 4.5.
[Passed]:Checks if SQL Server Express can be successfully installed on a secondary site.
[Passed]:Checks if the target site server computer already has existing Configuration Manager server components installed.
[Passed]:Checks if the Windows Defender Firewall is disabled or if a relevant Windows Defender Firewall exception exists for SQL Server.
[Passed]:Check SQL Server service running account.
[Passed]:Check to see if the selected SQL Server instance is already in use by another Configuratin Manager site
[Warning]:SQL Index create memory is not configured as default value of 0 and might hit issue
[Passed]:Verifies that the site server's database collation matches the database collation of its parent site.
[Passed]:Verifies that the Microsoft .NET Framework version 3.5 is installed on Configuration Manager central administration site servers, primary site servers, and secondary site servers.
[Passed]:Verifies that the Microsoft .NET Framework version 4.0 is installed on Configuration Manager Secondary site computers for installing SQL Server Express edition.
[Passed]:Check product version in source folder for secondary site installation.
[Passed]:Check machine account of secondary site access to setup source folder.
[Passed]:Check SQL Server in the secondary site machine.
[Passed]:Checks if the FQDN provided for the site system uses the primary DNS hostname for the computer.
[Warning]:The site server might be unable to publish to Active Directory. The computer account for the site server must have Full Control permissions to the System Management container in its Active Directory domain. You can ignore this warning if you have manually verified these permissions. For more information about your options to configure required permissions, see https://go.microsoft.com/fwlink/p/?LinkId=233190.
[Passed]:Check remote connection to WMI on secondary site.
[Passed]:Verifies that the SQL Server instance and Configuration Manager site database (if present) are configured to use a supported collation.
[Passed]:Checks if the SQL Server Express version on the secondary site is at least SQL Server 2008 R2 Service Pack 1 (version 10.51.2500.0).  If Configuration Manager did not previously install SQL Server Express (existing instance is not CONFIGMGRSEC), then Setup skips this check.
[Passed]:Verifies that SUM is not using any virtual locations for active SUPs.
[Passed]:Verifies that the user account running Configuration Manager Setup has been granted sysadmin SQL Server role permissions on the SQL Server instance selected for site database installation. SQL Server sysadmin role permissions are required in order to create the site database and configure necessary database role and login permissions for Configuration Manager sites.
[Passed]:Verifies that the user account running Configuration Manager Setup has been granted sysadmin SQL Server role permissions on the SQL Server instance targeted for site database installation. SQL Server sysadmin role permissions are required in order to create the site database and configure necessary database role and login permissions for Configuration Manager sites.
[Passed]:Verifies that SQL Server is configured for Windows authentication security.
[Passed]:Verifies that the version of Microsoft SQL Server installed on the computer selected to host the site database meets the minimum requirements.
[Passed]:Configuration Manager sites require a supported SQL Server version. For more information, see https://go.microsoft.com/fwlink/p/?LinkID=232936.
[Passed]:Checking the site SQL Server is not Express Edition.
[Passed]:Checking the site SQL Server Tcp is enabled and set to Static port.
[Warning]:SQL Server is configured for unlimited memory usage. You should configure SQL Server memory to have a maximum limit.
[Warning]:Configuration Manager requires SQL Server to reserve a minimum of 8 gigabytes (GB) of memory for the central administration site and primary site and a minimum of 4 gigabytes (GB) for the secondary site. This memory is reserved by using the Minimum server memory setting under Server Memory Options and is configured by using SQL Server Management Studio. For more information about how to set a fixed amount of memory, see https://go.microsoft.com/fwlink/p/?LinkId=233759.
[Passed]:Checks if the SQL Server hosting the Configuration Manager site database is using a case-insensitive collation.
[Passed]:Check that the specified FQDN for the SQL Server computer is valid.
[Passed]:Verifies that the required administrative shares are present on the site system computer.
[Passed]:Verifies that a valid Service Principal Name (SPN) is registered in Active Directory Domain Services for the account configured to run the SQL Server service.
[Passed]:Verifies that the site server operating system meets the minimum requirement for site server installation. https://go.microsoft.com/fwlink/?linkid=841654
[Passed]:Check target management point and distribution point machine is not Windows Cluster Node.
[Passed]:Verifies that Internet Information Services (IIS) is running.
[Passed]:Verifies that Background Intelligent Transfer Service (BITS) is installed in Internet Information Services (IIS).
[Passed]:Verifies that Background Intelligent Transfer Service (BITS) is enabled in Internet Information Services (IIS).
[Passed]:Verifies that Internet Information Services (IIS) is configured for HTTPS communication protocol.
[Passed]:Checks if the Windows Defender Firewall is disabled or if a relevant Windows Defender Firewall exception exists for SQL Server.
[Passed]:Verifies that the machine account of site server has administrative rights on the management point and distribution point computer.
[Passed]:Checks to see if there is an earlier version of the Configuration Manager client installed on the targeted management point computer.
[Passed]:Verifies that the operating system meets the minimum requirement of Windows Server 2003 for distribution point installation.
[Passed]:Verifies that the Microsoft Core XML Services (MSXML) version 6.0 or later libraries are installed.
[Passed]:Verifies that Internet Information Services (IIS) is configured for HTTPS communication protocol.
[Passed]:Verifies that the user running Setup has local administrator rights on the distribution point computer.
[Passed]:Prerequisite checks complete
[Passed]:Initiate secondary site server bootstrap installation service
[Passed]:Secondary site server bootstrap installation service initiation succeeded
[Passed]:Decompress secondary site server installation files
[Passed]:Secondary site server installation file decompression succeeded
[Passed]:Secondary site server bootstrap installation service initiating setup
[Passed]:Secondary site server bootstrap installation service initiating setup succeeded
[Passed]:ConfigMgr Setup - Recovering ConfigMgr Secondary site...
[Passed]:ConfigMgr Setup - Drop existing database on the secondary site.
[Passed]:Begin downloading and verifying installation files
[Passed]:Downloaded and verified installation files
[Passed]:Begin evaluation of secondary site server installation
[Passed]:Evaluation of secondary site server installation succeeded
[Passed]:Begin creation of secondary site server database
[Passed]:Creation of secondary site server database succeeded
[Passed]:Begin transfer of secondary site server installation files from parent primary site server
[Passed]:Transfer of secondary site server installation files from parent primary site server succeeded
[Passed]:Register secondary site server controls
[Passed]:Secondary site server control registration succeeded
[Passed]:Begin replication of secondary site server data from primary site server
[Passed]:Replication of secondary site server data from primary site server succeeded
[Passed]:Begin installation of secondary site server component manager
[Passed]:Installation of secondary site server component manager succeeded
[Passed]:Begin installation of secondary site server component manager service
[Passed]:Installation of secondary site server component manager service succeeded
[Passed]:Waiting for Database Replication Link State to be active.
[Passed]:Database Replication Link between primary and seconary site is active.
[Passed]:Start to validate content on the secondary site.
[Passed]:Content validation is in progress on the secondary site.
[Passed]:Content is validated on the secondary site.
[Passed]:ConfigMgr Setup - Recovered ConfigMgr secondary site successfully.
[Passed]:Secondary site server bootstrap installation service succeeded








Wednesday, October 24, 2018

SCCM Secondary Site stuck in recovering - "Recover Secondary Site" grayed (edit 2019-01: + Upgrade grayed)

[EDIT 2019-01-27 begin]
I complete my article:
It exist a stored procedure spUpdateSiteStatus permitting to un-gray by changing status.(Thanks Alban for this tip)



For this situation for example: here Upgrade stay grayed after some hours
 but prerequisite KO (moreover you can see c:\ConfigMgrSetup.log , c:\ConfigMgrSetupWizard.log and X:\SMS_BOOTSTRAP.log NOT modified since 4 months ago since last upgrade)

You must execute in SQL (with  XXX = site code)

exec dbo.spUpdateSiteStatus 'XXX', 3,2



now you can try to upgrade again













 Checking stored proc code:
 3, 2  = Install Successed
 5,2   =  Upgrade Successed
 6,2 = Recover Successed

-->
below my old article
[EDIT 2019-01-27 end]





Problem in one of secondary site stuck in recovering

 


And recovering does not seems to be cancellable and "Recover Secondary Site" grayed





















In "Show Install Status" log stay at
[Passed]:Waiting for Database Replication Link State to be active.

















but in C:\ConfigMgrSetup.log secondary site seems to be active (PRI = Primary MXX=Secondary site code, secondary computer is mysecondary.mydomain.net)

INFO: Verified that iphlpsvc is running on \\mysecondary.mydomain.net.
INFO: Setup will not check the CcmExec service .
INFO: Configuring SQL Server service broker... 
INFO: Reading Primary SQL Server certificate, Primary sitecode is PRI.
INFO: SQL Connection succeeded. Connection: SMS ACCESS, Type: Secure
INFO: Stored Parent Site [PRI] SQL Certificate to registry. 
INFO: SQL Connection succeeded. Connection: mysecondary.mydomain.net CONFIGMGRSEC\CM_MXX, Type: Secure
INFO: Service Broker configuration complete.
CPublicKeyLookup::Initialize("E:\Program Files\Microsoft Configuration Manager\inboxes\hman.box\pubkey") 
CPublicKeyLookup::Initialize()  Initializing the Public Key Store Path to E:\Program Files\Microsoft Configuration Manager\inboxes\hman.box\pubkey~ 
CPublicKeyLookup::UpdateCurrentKey("PRI", "0602000000A40XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")
CPublicKeyLookup::UpdateCurrentKey() Checking E:\Program Files\Microsoft Configuration Manager\inboxes\hman.box\pubkey\PRI.pkp for Key0~ 
CPublicKeyLookup::UpdateCurrentKey() Matching Key found in iteration 0.  Updating Date.~ 
INFO: Registered type SMS_ACCESS for mysecondary.mydomain.net CONFIGMGRSEC\CM_MXX 
INFO: SQL Connection succeeded. Connection: SMS_ACCESS, Type: Secure 
INFO: SiteStatus set to 105 on mysecondary.mydomain.net 
INFO: send message 0x00030051 to parent 
INFO: SQL Connection succeeded. Connection: SMS ACCESS, Type: Secure 
INFO: wait for secondary site replication state to be active. 
INFO: send message 0x00060007 to parent 
INFO: send message 0x00060007 to parent 
INFO: send message 0x00060007 to parent 
INFO: send message 0x00060007 to parent 
INFO: send message 0x00060007 to parent 
INFO: secondary site is now active 
INFO: send message 0x00060008 to parent 
INFO: send message 0x00060004 to parent 
INFO: verifying content metadata (1 processed)...  
INFO: send message 0x00060006 to parent 
INFO: Some content does not exist on secondary site. Make call to ResetMappedPkgStatus() 
Content can be identified with SQL query on Secondary Site DB: SELECT * From  PkgStatus_G where SourceVersion = 0 AND Type=1 AND SiteCode=dbo.fnGetSiteCode()
INFO: send message 0x00060005 to parent 
INFO: send message 0x00060002 to parent 
INFO: Removing byte order marker after reading from file (E:\Program Files\Microsoft Configuration Manager\bin\X64\secondarysiteupdatepackage.xml) 
Successfully update secondary site update pacakge status from file E:\Program Files\Microsoft Configuration Manager\bin\X64\secondarysiteupdatepackage.xml 
INFO: Successfully begin Automatic Updates detection task 
Not recovery mode or not top level site. Skip restoring client piloting packages. 
~~===================== Completed Configuration Manager Server Setup ===================== 





Waiting some hours, but Secondary site was always in "recovering mode" . I find problem could be in SCCM SQL table Sites


After some internet search, I only find information about something really interesting in Release notes for System Center Configuration Manager
which give me information that table Sites AND  SC_SiteDefinition_Property  tables are important


I copy some article line in case of article deletion:

Recovery options for a secondary site are not available in the console

After recovery of a secondary site fails, the option Recover Secondary Site might no longer be available in the Configuration Manager console.
This issue affects System Center Configuration Manager version 1511 and 1602, and is expected to be resolved in a future update.
Workaround: Use one of the following methods to recover (reinstall) the secondary site:
  • Use Preinst.exe and the /delsite command to remove the secondary site, and then reinstall the secondary site. For information about preinst.exe, see Hierarchy Maintenance Tool (Preinst.exe) for System Center Configuration Manager
  • Run the following script to start the secondary site recovery. You run this script on the database at the primary parent site of the secondary site you want to recover:

    declare @SiteCode NVARCHAR(3)=N''  

    UPDATE Sites SET Status = 9 
                    , DetailedStatus = 3 
    FROM Sites WHERE SiteCode = @SiteCode 

    UPDATE SCP SET SCP.Value1 = 9 
                    , SCP.Value2 = N'3' 
    FROM SC_SiteDefinition_Property SCP INNER JOIN SC_SiteDefinition SC ON SC.SiteNumber = SCP.SiteNumber 
    WHERE SC.SiteCode = @SiteCode AND SCP.[Name] = N'Requested Status'   


I DON'T WANT delete my secondary site. Too much configuration to do, and Microsoft recommend to use another SiteCode
So I verify that using some SQL queries

SELECT * FROM Sites




We can see "Status"  and "DetailedStatus" field different in MXX from others Site codes



 SELECT * FROM SC_SiteDefinition_Property
 WHERE SiteNumber
IN (
SELECT Sitenumber FROM SC_SiteDefinition
WHERE SiteCode = 'MXX'

)
As you can see field Named "Requested Status" has Value1 corresponding to "Status" Sites table and Value2 corresponding to "DetailedStatus" Sites table


Information about theses information in SMS_Site Server WMI Class

RequestedStatus
Value indicating a request for secondary site status. Possible values are listed below. The default value is 1001.
1001 Create a secondary site; the primary site will send down the installation media.
1002 Create a secondary site using the installation media already available on the secondary site.
1003 Secondary site creation has started.
1004 Upgrade a secondary site; the primary site will send down the installation media.
1005 Upgrade a secondary site using the installation media already available on the secondary site.
1006 Secondary site upgrade has started.
1007 Deinstall a secondary site.
1008 Secondary site deinstall has started.
1009 Delete a secondary site.
1010 Secondary site deletion has started.
1011 Recover a secondary site; the primary site will send down the installation media.
1012 Recover a secondary site; the installation media is already available on the secondary site.
1013 Secondary site recovery has started.

Status
Current status of the site. Possible values are listed below. The default value is ACTIVE (1).
1 ACTIVE
2 PENDING
3 FAILED
4 DELETED
5 UPGRADE
6 Failed to delete or deinstall the secondary site.
7 Failed to upgrade the secondary site.
8 Secondary site recovery is in progress.
9 Failed to recover secondary site.

[EDIT 2019-01-27 begin]


From https://blogs.technet.microsoft.com/umairkhan/2014/02/17/configmgr-2012-data-replication-service-drs-unleashed/  which seems to complete DetailedStatus

SiteStatus
Mode
100
105
110
115
120
125
130
135
199
200
205
210
215
220
225
230
250
255

'SITE_INSTALLING'
'SITE_INSTALL_COMPLETE'
'INACTIVE'
'INITIALIZING'
'MAINTENANCE_MODE'
'ACTIVE'
'DETACHING'
'READY_TO_DETACH'
'STATUS_UNKNOWN'
'SITE_RECOVERED'
'SITE_PREPARE_FOR_RECOVERY'
'SITE_PREPARED_FOR_RECOVERY'
'REPLCONFIG_REINITIALIZING'
'REPLCONFIG_REINITIALIZED'
'RECOVERY_IN_PROGRESS'
'RECOVERING_DELTAS'
'RECOVERY_RETRY'
'RECOVERY_FAILED'
[EDIT 2019-01-27 end]


So I modify  request as


declare @SiteCode NVARCHAR(3)=N'MXX' 

UPDATE Sites SET Status = 1, DetailedStatus = 125
FROM Sites WHERE SiteCode = @SiteCode

UPDATE SCP SET SCP.Value1 = 1 , SCP.Value2 = N'125'
FROM SC_SiteDefinition_Property SCP INNER JOIN SC_SiteDefinition SC ON SC.SiteNumber = SCP.SiteNumber
WHERE SC.SiteCode = @SiteCode AND SCP.[Name] = N'Requested Status' 



WARNING: modifying database is NOT supported by Microsoft. Please use it at your own risk




now Sites and SC_SiteDefinition_Property tables are ok


SELECT * FROM Sites



 SELECT * FROM SC_SiteDefinition_Property
 WHERE SiteNumber
IN (
SELECT Sitenumber FROM SC_SiteDefinition
WHERE SiteCode = 'MXX'




 and now "Recover Secondary Site" no more grayed. I can recover secondary site if it's necessary.