Unfortunately there are quite a lot of ball joints in the supension and steering. As to whether it's bad to drive it depends on which ball joint and how bad it is. If it's an important one and fails catastrophically your suspension or steering could fail - maybe at high speed. I'd get it fixed ASAP.
Terry, October 2006