We started by just putting aside compostable (is that a word?) materials -- peels, skins, any leftover vegetables, tea bags, coffee grounds, egg shells etc. into a container on our counter.
We use a small plastic bucket with a plastic grocery bag liner but I know there are some special counter top compost containers available that have filters to help with the smell. For us, if it starts to get smelly, we just carry it outside.
Right now we just have a pile out behind our house (a good distance back into the woods). Eventually I want to construct a container so we can more effeciently turn our scraps into usable compost. We also mix in leaves, pine needles, grass clippings etc. (although I think some people do not mix in grass because they don't want grass seeds in their compost).
We were AMAZED at how much our trash was reduced by composting.
Good luck! Have fun!!