I am afraid you are right being concerned about performance. Maybe you can use collections built by Find DUPES
to simplify your caching. For maximum speed and best usability you will probably need your own cache. ExtendedExif uses one and could provide some ideas.