SCCM Updates: Go for "Autopilot" not "Total Automation"

I've been getting lot of questions on a previous post regarding my approach to software update structure with SCCM. I replied to a few and this generated even more questions so I felt the need to do a more lengthy response about why I do things the way I do them and what led to my approach and that led to this post. I'll start by answering the last question posted there. Yes, I clean up superseded and expired updates by hand (for now at least) and there are a few very good reasons why I do it that way. When I was first attempting to implement software updates with SCCM I feel that my big initial mistake was to chase after a strategy that focused on "total automation". During my that first approach I literally created an ADR and package for every individual product that I wanted to patch. The ADRs ran monthly and cleaned out the expired and superseded stuff automatically. Sounds great right? In practice what I actually discovered was this approach works great for something simple like a daily Endpoint Protection update (because in that scenario you really only have to consider about 3-4 small patches a day tops). You try to do that with all the security, critical, service pack, etc. updates for multiple Windows versions going back to say 2003 and you run into several problems.

First you have a *lot* more patches to deal with in your software update group(s). Every time your SCCM clients kick off a software update evaluation cycle they have to look through all those updates and determine if any apply to the device upon which they are running. At a Microsoft conference last year I learned that this can cause high CPU utilization on your target machines. Still that’s not too bad because it only has to run once against the software update group provided the update group doe not change after that initial run. Ah but there’s the rub, I also learned that every time the ADR runs it automatically changes the content in your software update group! Do that every Patch Tuesday and your clients have to once again parse or scan that update group which (again) can hit your CPUs pretty hard. To put it in perspective, think about how many times we hear users complain about "slow computers" in our line of work. You don’t want to introduce something hat will hammer CPU utilization on your Servers or User PC multiple times a month.

Second by having a bunch of ADRs (I had a total of 12) that were constantly changing and updating the content in my software update groups I discovered I was putting a lot of load on my primary SCCM server in the weeks following patch Tuesday. The reason there should be obvious as it was constantly downloading patches and cleaning up software update groups based on those ADRs. So by chasing total automation I was (in fact) causing lot of performance issues on both my clients and my servers. Also it really made no sense to have ADRs that constantly ran against groups with updates that go way back in the past (they almost never change unless Microsoft expires or supersedes them). I came to realize that year-based static update groups with a single monthly ADR were the best choice for dealing with the performance and organizational issues I was facing. The reason being is I typically only have to change the older groups once a quarter (sometimes longer).  So (usually) I only have to worry about the updates in my most recent year-based group or my monthly groups (I usually keep monthly groups up to three months in the past before moving the updates from the oldest group into a year-based group).

Finally I learned another valuable lesson about the ADRs. No matter how well you craft your download rules they will inevitably screw something up. Either they will download something you don't want or remove something you want to keep. Its much better to have your monthly ADRs downloading updates into a test group and then (once you've verified what it downloaded) you can then move the contents of that test group into your static update group that targets production machines. Coincidentally this solves the "Expired and Superseded" issue through attrition as it forces you to do a "monthly visual audit" on your update groups when you open the console up after patch Tuesday to see what your ADR downloaded. If your static groups contain superseded or expired updates you'll see that the icon next to their names has changed to reflect that. At that point its child's play to just open the group and remove the older updates by editing their membership.

So that’s a pretty big reply to the questions I was being asked and (honestly) a lot of it is just what I’ve discovered through using SCCM 2012 in the past year. There may be a better way to do it, but this one works for me. I really struggled a lot with this at first and I hope I'm posting this big response in an effort to help others avoid some of my own pain. SCCM is a really complicated animal and I just think its a bad idea to go into it "set it and forget it approach". If it were really that easy we wouldn't need SCCM admins right? The main point I’d like to get across though is sometimes a 100% automated solution just doesn't make sense. SCCM can automate a lot of things for you but even with the advent of the autopilot feature on an airplane we still need pilots at the helm. Focus on design paradigms that automate the most time consuming, tedious or repetitive steps. Then if you find yourself still stuck with a manual step, try to make it something you can do very quickly (like at the touch of a button). I hope that makes sense and that others find it helpful. If there are any counterpoints to this philosophy or if anyone just has better ways of approaching this I'd love to hear about it in the comments section.

UPDATE: Someone requested a screenshot of my SUGs. I've added it below. Like I said we roll about 2-3 months in the past on our monthly deltas and once we achieve around 90% I move those to a year-based group. I've highlighted the delta and year-based groups in the screenshot. The other groups listed there are "special case" groups: