For simplicity, you could use something like this in a button to select all the nested items. Though it won't work in places like collections, libraries, etc.
=return "Select DESELECTNOMATCH FILTERDEF =return Count("""+source+""")+1 < Count(fullpath);";
This other will work for the selected folder only:
=return "Select DESELECTNOMATCH FILTERDEF =return isParent("""+selpath+""",fullpath) && Count("""+selpath+""")<Count(fullpath);";
Also, check out this script add-in, once installed, you could use something like this to expand the selected folders and select all their contents in one go.
ExpandFolders EXPAND=top SELECT
For something more specific, you'd need to use scripting, which basically lets you do exactly what you want.