Is there a way to arrange captions on a vertical, text only toolbar as rotated? Like in Microsoft Word for example (which makes it not quite right, because captions' bottoms are facing outwards the screen).
X.
I don't think you can do that at the moment; the labels are always horizontal.
Do you think it's worth putting in a request (+ arraging icon side to side with caption on a button)?
X.
If you want it request it. 
Filled a request.
X.
[quote="Xyzzy"](+ arraging icon side to side with caption on a button)?
[/quote]
BTW, this one I've found in edit dialog of a button- it's per button setting, not per toolbar.
X.