Everyone is different, but for me even early on, my joints start to loosen up; so it is easier to turn my ankle if I step wrong, or sprain (and possibly break) something if I fall. So that is a consideration.
I think it is very unlikely you'd fall in such a way that the baby could be harmed, but if you injure YOURSELF and need an xray, painkillers, (or, heaven forbid, surgery) during pregnancy, THAT would be upsetting. I know it's unlikely but the possibility would probably worry me so much I'd prefer sitting it out and watching.
That said, my sister broke her foot coming down some stairs and tripping, in her last trimester, and had to cope with that, so I suppose it could be said that doing ANYTHING is a risk, but roller skating probably puts you at slightly HIGHER risk. My sister's baby was breech position and she had a c-section, which is possibly coincidental to the ijury or possibly due to walking around in her "boot" the last trimester (hips uneven.)