The reason for this is how Bluetooth deals with music when your microphone is turned off (MIC OFF) versus when it's turned on (MIC ON). Your phone treats these two situations differently and sets the volume levels separately.
For instance, if you're in a group with your microphone muted (MIC OFF) and you're listening to music, your volume might be set at 60%. But when you press the Unmute mic button (MIC ON), your volume level will change and your music won't be at the same level as before. Then, when you finish talking and mute your mic again (MIC OFF), your volume will go back to the original 60%.
Unfortunately, in both MIC ON and MIC OFF modes, the volume levels won't be the same. However, once you set the volume in both of these modes, they will stay the same.
If you're using an Apple device, you'll need to adjust the volume separately for each of these modes using your volume buttons. On Android, it's a bit easier; you can adjust the volume with the volume buttons on your phone and tap the volume level to access the mixer, which allows you to change the volume for multiple audio sources.