Hey Aaron, thanks for the message and inquiry! Happy to help with optimization and best practices for this:
Generally, when researching at an account/company level, we recommend doing that in the Company Table and the same is true for contacts when researching about them. We do have Lookup actions (quick video) you can leverage to pivot corresponding information from one table to the other as long as a matching/pivot point exists i.e. both tables have the company domain present.
This ensures you don't run into that issue you're describing, and only enriching the specific record once depending on its type. My recommendation is that you do this in the company table instead- Let me know if you have additional questions!