You did most of the things, I have just added the Referrer URLs for Browser API key and you will see View Map button for Address to Address booking only, system will not show that button for Fixed Fares. Now as you are using your own API keys, so you can show the Google Map at the right side in the First Step (as it was earlier few months ago) so that your customers can see Direction Route immediately after pickup and dropoff are selected, they don't have to click on View Map button. But, then your API key will be triggered on each map view so you might be billed higher by Google. If you still want to show the map at right side, then you just need to set Show map in popup = NO and Show on Desktop & Mobile = YES at Backend -> Design settings.
Please check your entire Booking system and let us know if you find any problem.
Thanks & Regards