I will grant that elite units within a parent organization are superbly trained and highly skilled in what they do.
But let's take a minute to look a bit closer. I'll use the FBI's HRT as an example, but the comments apply to pretty much any of the other units.
The Hostage Response Team was created for a specific purpose -- to rescue hostages. It is a specialized task towards which all their training is geared. I'm not saying that they are not adept at marksmanship, small-unit tactics, and other military skills that may serve them in an invasion scenario. However, their training and general mindset would make them less effective as a partisan commando than some would believe. I would equate them in capability roughly with SWAT teams and National Guard Military Police units -- better than the typical run of civilian warriors, but not necessarily front-line troops.
The other disadvantage to these guys is that there aren't many of them. I don't know how many HRTs there are, but I know that they are scattered around the country and, even if you could get them all together in one place, I don't think that there is more than a heavy platoon of them.
The only exceptions that I can think of are the US Marine Embassy Guards and the US Army soldiers who guard the Tomb of the Unknowns. In both cases, the men in question are fully-fledged members of their branch of service who rotate the duty assignments with others who have qualified for the special duty. (The 3rd Infantry even has a couple of units deployed right now, if I remember correctly.)