you will have to remove driveshaft. sounds like you pulled the three legged bearing out of driveshaft cup and lost all rollers. this makes the shaft longer and not fit in properly. it will become apparent when rubber boot removed on inner joint of driveshaft. you can rebuild it back together but its fiddly and you need all the little rollers. or you could just buy another driveshaft and fit that. next time undo the 36mm hub nut so driveshaft is not disturbed too much.
been there done that, May 2010