There isn't a way to selectively use different playback engines for different file extensions. (Anything in an MKV could also be inside an MP4 as well.)
The problem you're seeing is likely due to a shell extension something has installed, which doesn't like when the file is in use, or which does excessive amounts of file access when building the menu, making it slow if the file is in use.
See Crash, exit or high CPU when right-clicking certain files for how to track down the problem component.