The most practical way is by train.
First station out of Amsterdam with bike rental is Weesp. That's only a few stops and about 15 minutes by train from Amsterdam Central (4 trains per hour till after midnight).
Direct trains from Schiphol Airport take you to Weesp in about 25 minutes (also 4 trains per hour both ways).
See timetables railways : www.ns.nl. On that website you also find information on station facilities, e.g. bicycle rental.
Another good starting point for cycling is the next train station : Naarden-Bussum, or the next couple of stops.
More information on bike rental : where to start.
Every town and village in the region can also be reached by bus. For lines and timetables of all public transport in the Netherlands : www.9292ov.nl.
All public transportation in the Netherlands works with prepaid chip cards. You have to check in correctly, otherwise gates will not open when you try to check out.
Leave Amsterdam to the south-east on motorway A1, direction Hilversum-Amersfoort. There's always dense traffic, so some form of delay is not unlikely.
Just outside Amsterdam, after you crossed the Amsterdam-Rhine Canal, the first exit is Muiden / Weesp. Other places mentioned above at "train" are further down the same motorway.
For Muiden best park at the motorway exit, and walk.
For a visit to Weesp, best park at the train station there. Driving into direction Weesp, you'll see the railroad. Just before entering the town, turn right at a roundabout, street called Leeuwenveldseweg. Then you can't miss the station and a large parking on your left. The bicycle shop is on other side of the station, side centre of town. For the local situation, see this Google map.
Driving your car on the narrow country roads is not always a pleasure, and many places are simply impossible to reach by car. Parking space in the centre of the old towns is limited, and not for free. So best park somewhere outside and walk.
It is perfectly possible to cycle all the way out of Amsterdam. Easiest and shortest route is to follow the mighty Amsterdam-Rhine Canal. There's a bicycle path all along the western side of the canal.
From the centre of Amsterdam to Weesp or Muiden, count 45 minutes to an hour of relaxed pedalling. Double the time, since you do have to get back later in the day, of course!
Following the canal to the south, you can't miss the railway bridge (see pictures above). Go up the bicycle path to the other side of the canal.
High on the bridge, you'll have a nice opening panorama of the region, with Weesp, Muiden and the wide open space of the green polder. On a clear day, you can even see the bell tower of fortified Naarden on the horizon, and vaguely the woods on the low hillocks beyond.
An alternative is to cycle along the IJburg district, a brand new Amsterdam expansion built in this part of the former Zuyder Sea. Following the old sea dike, the Diemerdijk. Keep in mind that the dike stretch between the power plant and Muiden is closed during late autumn and winter months.
For a series of pictures of the region you're about to enter : click here.