Just some random thoughts on Kalymnos:
Avoid the busy season. Shoot for the month of November.
Contact info for the awesome flat I stayed at: stsakrios@gr.com (Skavos and Katina)
Flying into Kalymnos from Athens wasn’t bad… although I can see the potential for getting screwed by the weather.
There are plenty of walkable crags, a scooter is not a strict requirement. However, if you want to go to less polished crags a scooter is needed.
You can take the bus to the cheaper market in Panoramas (I think it’s called Smalios).
Apparently Olympic Air doesn’t fly regularly to Kalymnos, so plan accordingly.
Update:
Having now done the Kos flight and subsequent ferry I can endorse this as a totally feasible option.