ModifyDate doesn't seem to be listed here: exiv2.org/tags.html
Are you sure that isn't ExifTool's name for the filesystem modified date? Is that being set as you expect, or is it set to something else (e.g. the current time)?
If you make the change in Opus, then close and re-open the metadata panel, are all the timestamps still showing what you expect them to be?