The modern city of Olympia is quite small, (perhaps less than 2,000 people?) but does have a few hotels. It is not a great place, not much is going on and it is not that attractive.
Pirgos to Patra is about 105km.
Patra is about 200,000 people and is the major hub for the western end of the country. It's never impressed me as a charming place, but I suppose it has its attractions. Ferries connect there.
Nafpaktos across the impressive Rio bridge is a nice place to rest. East and north of there is Galaxidi and then Delphi.
You can come across the mountains and approach Olympia from the east. It's a very scenic drive and you can visit Dimitsana on the way. But the road can be ... entertaining.
The area around Olympia is served by bus and rail. The rail line west of Athens goes to Patra, then down to Kalamata. In 2007 the timetables shows about 10 trains a day in both directions, with a time from Pirgos to Patra of about 2 hrs. I've never taken this train or bus to Olympia so no promise from me of how it works.