I missed the other half of the issue. The wildcard escaping probably doesn't matter, in fact.
If your filename is (aaa.txt and you use a pattern of (aaa.txt with IGNOREEXT then that won't work as the pattern doesn't match the filename. It would only work if the file had a name like (aaa.txt.jpg, because it is looking for (aaa.txtbefore the extension.
This will rename (aaa.* to 0005.* for all selected files:
Rename IGNOREEXT PATTERN="(aaa" TO="0005"
(Note that the PATTERN argument, does not need the ( to be escaped. Escaping things with the FROM argument is complicated and depends if only FROM is used or if both FROM and PATTERN are used together, I think.)
Or, if you want the file selection to be ignored and for (aaa.txt to always be the file that gets renamed:
Although there are much more direct ways of doing that, of course:
Rename FROM="(aaa.txt" TO="0005.txt"
If you want to change the file's extension as well then you should not be using IGNOREEXT at all, since the whole purpose of IGNOREEXT is to preserve the existing extension (so that you can ignore it when worrying about your wildcards / regular expressions / etc.)