You can do it. And it sounds like you want to. My situation is very similar to yours, I have a first grader and though i dont have a high needs toddler but i have a 'easy' 4 yo and a baby with feeding issues. An hour a day 4 times a week is MORE than enough at this age. School will always be there, but i would imagine its very hard to homeschool once your family is part of the culture of school.