How long to drive between Pacific Grove / Moneterey to Hearst Castle on Highway 1?

Hi - I am driving from SF to LA end of August over 3 days. I am thinking of spending the 1st night in Pacifc Grove or Monterey, and 2nd night in San Simeon. I would like to visit Hearst Castle so am thinking of doing one of the latest tours (4pm or 4:20pm I believe) on day 2. Therefore how long should I allow to drive from Pacific Grove to Hearst Castle on Highway 1, to reach Hearst Castle for approx 3:30pm? I would still like to have enough time to stop a few times enroute for photos etc? Is this doable, or do I need to replan? Thanks for your help!