What's probably happening here is your adapter is signaling some button press when you connect it that's popping up the Assistant. Only way to fix that is to get a new adapter since you can't manually disable that input from triggering the Assistant without root.
However, you can completely disable the Google Assistant from appearing at all via:
Settings > Apps > Default Apps > Digital Assistant App > Default Digital Assistant App > None
That should fully disable it in all regards and it'll never pop up again.
Google voice assistant cannot be turned off. How else are they supposed to listen to your conversations and create personalized ads just for your convenience?