For the past few days I've been struggling to wrap my head around what may have happened that resulted in distribution errors like this in distmgr.log:
Sleep 30 minutes... SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:35 PM 6584 (0x19B8) Found notification for package 'AP2000D9' SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:40 PM 5288 (0x14A8) Used 0 out of 5 allowed processing threads. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:40 PM 5288 (0x14A8) Starting package processing thread, thread ID = 0x1BA8 (7080) SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 5288 (0x14A8) Sleep 3600 seconds... SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 5288 (0x14A8) STATMSG: ID=2300 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_DISTRIBUTION_MANAGER" SYS=AP2SCCM01.F.Q.D.N SITE=AP2 PID=11276 TID=7080 GMTDATE=Thu Apr 24 18:06:41.487 2014 ISTR0="Wireless (Internal)" ISTR1="AP2000D9" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=1 AID0=400 AVAL0="AP2000D9" SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 7080 (0x1BA8) No action specified for the package AP2000D9. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 7080 (0x1BA8) Start adding package to server ["Display=\\svcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\svcm01.F.Q.D.N\... SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 7080 (0x1BA8) Attempting to add or update a package on a distribution point. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 7772 (0x1E5C) Start adding package to server ["Display=\\brcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\brcm01.F.Q.D.N\... SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 7080 (0x1BA8) Attempting to add or update a package on a distribution point. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 11496 (0x2CE8) Start adding package to server ["Display=\\sfcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\sfcm01.F.Q.D.N\... SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 7080 (0x1BA8) Will wait for 3 threads to end. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 7080 (0x1BA8) Thread Handle = 0000000000001D40 SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 7080 (0x1BA8) Attempting to add or update a package on a distribution point. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 6680 (0x1A18) STATMSG: ID=2342 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_DISTRIBUTION_MANAGER" SYS=AP2SCCM01.F.Q.D.N SITE=AP2 PID=11276 TID=11496 GMTDATE=Thu Apr 24 18:06:41.602 2014 ISTR0="Wireless (Internal)" ISTR1="["Display=\\brcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\brcm01.F.Q.D.N\" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=2 AID0=400 AVAL0="AP2000D9" AID1=404 AVAL1="["Display=\\brcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\brcm01.F.Q.D.N\" SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 11496 (0x2CE8) STATMSG: ID=2342 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_DISTRIBUTION_MANAGER" SYS=AP2SCCM01.F.Q.D.N SITE=AP2 PID=11276 TID=6680 GMTDATE=Thu Apr 24 18:06:41.610 2014 ISTR0="Wireless (Internal)" ISTR1="["Display=\\sfcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\sfcm01.F.Q.D.N\" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=2 AID0=400 AVAL0="AP2000D9" AID1=404 AVAL1="["Display=\\sfcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\sfcm01.F.Q.D.N\" SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 6680 (0x1A18) Get site system FQDN and account information from DB for ["Display=\\sfcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\sfcm01.F.Q.D.N\ SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 6680 (0x1A18) for ["Display=\\sfcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\sfcm01.F.Q.D.N\, no connection account is available SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 6680 (0x1A18) STATMSG: ID=2342 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_DISTRIBUTION_MANAGER" SYS=AP2SCCM01.F.Q.D.N SITE=AP2 PID=11276 TID=7772 GMTDATE=Thu Apr 24 18:06:41.655 2014 ISTR0="Wireless (Internal)" ISTR1="["Display=\\svcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\svcm01.F.Q.D.N\" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=2 AID0=400 AVAL0="AP2000D9" AID1=404 AVAL1="["Display=\\svcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\svcm01.F.Q.D.N\" SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 7772 (0x1E5C) Thread Handle = 0000000000002634 SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 7080 (0x1BA8) Thread Handle = 00000000000024E4 SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:41 PM 7080 (0x1BA8) The current user context will be used for connecting to ["Display=\\brcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\brcm01.F.Q.D.N\. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 11496 (0x2CE8) The current user context will be used for connecting to ["Display=\\sfcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\sfcm01.F.Q.D.N\. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 6680 (0x1A18) The current user context will be used for connecting to ["Display=\\svcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\svcm01.F.Q.D.N\. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 7772 (0x1E5C) Failed to make a network connection to \\brcm01.F.Q.D.N\ADMIN$ (0x5). SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 11496 (0x2CE8) Cannot establish connection to ["Display=\\brcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\brcm01.F.Q.D.N\. Error = 5 SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 11496 (0x2CE8) Failed to make a network connection to \\sfcm01.F.Q.D.N\ADMIN$ (0x5). SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 6680 (0x1A18) STATMSG: ID=2323 SEV=E LEV=M SOURCE="SMS Server" COMP="SMS_DISTRIBUTION_MANAGER" SYS=AP2SCCM01.F.Q.D.N SITE=AP2 PID=11276 TID=11496 GMTDATE=Thu Apr 24 18:06:42.260 2014 ISTR0="5" ISTR1="3" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=2 AID0=400 AVAL0="AP2000D9" AID1=404 AVAL1="["Display=\\brcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\brcm01.F.Q.D.N\" SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 11496 (0x2CE8) Failed to make a network connection to \\svcm01.F.Q.D.N\ADMIN$ (0x5). SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 7772 (0x1E5C) Error occurred. Performing error cleanup prior to returning. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 11496 (0x2CE8) STATMSG: ID=2323 SEV=E LEV=M SOURCE="SMS Server" COMP="SMS_DISTRIBUTION_MANAGER" SYS=AP2SCCM01.F.Q.D.N SITE=AP2 PID=11276 TID=6680 GMTDATE=Thu Apr 24 18:06:42.329 2014 ISTR0="5" ISTR1="3" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=2 AID0=400 AVAL0="AP2000D9" AID1=404 AVAL1="["Display=\\sfcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\sfcm01.F.Q.D.N\" SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 6680 (0x1A18) Cannot establish connection to ["Display=\\sfcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\sfcm01.F.Q.D.N\. Error = 5 SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 6680 (0x1A18) DP thread with array index 1 ended. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 7080 (0x1BA8) Cannot establish connection to ["Display=\\svcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\svcm01.F.Q.D.N\. Error = 5 SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 7772 (0x1E5C) Error occurred. Performing error cleanup prior to returning. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 7772 (0x1E5C) STATMSG: ID=2323 SEV=E LEV=M SOURCE="SMS Server" COMP="SMS_DISTRIBUTION_MANAGER" SYS=AP2SCCM01.F.Q.D.N SITE=AP2 PID=11276 TID=7772 GMTDATE=Thu Apr 24 18:06:42.356 2014 ISTR0="5" ISTR1="3" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=2 AID0=400 AVAL0="AP2000D9" AID1=404 AVAL1="["Display=\\svcm01.F.Q.D.N\"]MSWNET:["SMS_SITE=AP2"]\\svcm01.F.Q.D.N\" SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 7772 (0x1E5C) Error occurred. Performing error cleanup prior to returning. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 6680 (0x1A18) DP thread with thread handle 0000000000002634 and thread ID 11496 ended. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 7080 (0x1BA8) Will wait for 2 threads to end. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 7080 (0x1BA8) Thread Handle = 0000000000001D40 SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 7080 (0x1BA8) Thread Handle = 00000000000024E4 SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 7080 (0x1BA8) DP thread with array index 1 ended. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 7080 (0x1BA8) DP thread with thread handle 00000000000024E4 and thread ID 6680 ended. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 7080 (0x1BA8) Will wait for 1 threads to end. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 7080 (0x1BA8) Thread Handle = 0000000000001D40 SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 7080 (0x1BA8) DP thread with array index 0 ended. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 7080 (0x1BA8) DP thread with thread handle 0000000000001D40 and thread ID 7772 ended. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:42 PM 7080 (0x1BA8) Created policy provider trigger for ID AP2000D9 SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:44 PM 7080 (0x1BA8) Package AP2000D9 does not have a preferred sender. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:44 PM 7080 (0x1BA8) A program for package AP2000D9 has changed, therefore it needs to be replicated to all child sites. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:44 PM 7080 (0x1BA8) Package AP2000D9 is new or has changed, replicating to all applicable sites. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:44 PM 7080 (0x1BA8) CDistributionSrcSQL::UpdateAvailableVersion PackageID=AP2000D9, Version=1, Status=2301 SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:44 PM 7080 (0x1BA8) StoredPkgVersion (1) of package AP2000D9. StoredPkgVersion in database is 1. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:44 PM 7080 (0x1BA8) SourceVersion (1) of package AP2000D9. SourceVersion in database is 1. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:44 PM 7080 (0x1BA8) STATMSG: ID=2302 SEV=E LEV=M SOURCE="SMS Server" COMP="SMS_DISTRIBUTION_MANAGER" SYS=AP2SCCM01.F.Q.D.N SITE=AP2 PID=11276 TID=7080 GMTDATE=Thu Apr 24 18:06:44.252 2014 ISTR0="Wireless (Internal)" ISTR1="AP2000D9" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=1 AID0=400 AVAL0="AP2000D9" SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:44 PM 7080 (0x1BA8) Failed to process package AP2000D9 after 0 retries, will retry 3 more times SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:44 PM 7080 (0x1BA8) Exiting package processing thread. SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:44 PM 7080 (0x1BA8) Sleep 30 minutes... SMS_DISTRIBUTION_MANAGER 4/24/2014 2:06:45 PM 6584 (0x19B8)
The log is huge so I invite you to check out the 'FailedDistributionLog.log' log in the 'FailedDistributionError5' folder of my OneDrive.
The hierarchy is simple:
- single primary site with a local SQL installation, management point role (among other roles)
- five other servers acting as the distribution points: 1 local, 4 remote
Something has changed recently because distribution was working fine up until I believe this week. I suspect some GPO is in place (or not in place) which is causing trouble, but I have not been successful in confirming that or isolating the issue to anything other than "it affects my 2008 R2 servers".
What I've checked so far:
- the site server (2008 R2 Standard) has no issues during distribution to itself
- Local DP win7dp01 has no issues during distribution (this is a Windows 7 DP just for testing/proof of concept purposes)
- Remote DP 2003dp01 (Server 2003) has no issues during distribution
- Remote DP's 2008dp01, 2008dp02 & 2008dp03 (all 2008 R2 Standard) are producing the errors above
- Site System Installation Account is set to 'Use the site server's computer account to install this site system'
- All servers are joined to the same domain
- All servers are members of an SCCM Servers AD security group
- All servers have the above SCCM Servers AD security group added to the local administrators group
- As part of troubleshooting, I've added the Site Server by name (domain\servername) to the local administrators group, in addition to the AD security group which its already a member of, to the three 2008 DP's that are failing. (The Windows 7 and 2003 DP's don't have this setup because they seem to be working fine with the AD security group.)
- The firewall is disabled [via GPO] on all these servers & DP's
- The ADMIN$ share is available & accessible on the three 2008 DP's
- The three 2008 DP's have NO_SMS_ON_DRIVE.SMS only on the root drive (not the data drive)
- The three 2008 DP's have Data drives which house the SCCMContentLib, SMSPKG$, SMS_DP$, SMSPKGSIG$ & RemoteInstall directories.
- I checked the Share & NTFS permissions for the SCCMContentLib, SMSPKG$, SMS_DP$, SMSPKGSIG$ & REMINST shares, and local Administrators have full access/full control
- The three 2008 DP's have at least 100GB free; this package is under 1MB.
- The file 2 files in this package (one XML & one CMD) have very short file name lengths
- The files in this package are not currently in use. (I've edited, renamed & moved them to be sure)
- RSOP of the server confirms firewall is disabled (I checked wf.msc manually anyway)
Under Component Status, SMS_DISTRIBUTION_MANAGER is showing critical errors for the three 2008 DP's. I've copied up the relevant entries from the log up to my OneDrive as 'SMS_DISTRIBUTION_MANAGERLog.log'. In short there are two error codes:
2323 with a description of:
Distribution Manager failed to connect to the distribution point. Possible cause: Distribution Manager cannot access the distribution point machine because of access permissions issues. Solution: Make sure that the site server machine account or Site System Installation account has administrative permissions on the distribution point machine. Retry Interval is 5 minutes, number of retries left is 3.
2302 with a description of:
Distribution Manager failed to process package "My (Package)" (package ID = PRI000D9). Possible cause: Distribution Manager does not have access to either the package source directory or the distribution point. Solution: Verify that distribution manager can access the package source directory/distribution point. Possible cause: The package source directory contains files with long file names and the total length of the path exceeds the maximum length supported by the operating system. Solution: Reduce the number of folders defined for the package, shorten the filename, or consider bundling the files using a compression utility. Possible cause: There is not enough disk space available on the site server computer or the distribution point. Solution: Verify that there is enough free disk space available on the site server computer and on the distribution point. Possible cause: The package source directory contains files that might be in use by an active process. Solution: Close any processes that maybe using files in the source directory. If this failure persists, create an alternate copy of the source directory and update the package source to point to it.
I stood up 1 more DP on a 2003 servers which completed successfully. Its already received content error-free. I'll enable PXE & verify that works before standing up my final 2003 DP. I don't have any other 2008 servers I could test this on which complicates this a bit.
Before I go standing up more [2008] servers, what else could I be testing/validating?