You shouldn't have to remove the pulley to change the belt.
Remove the deck belt first, then install the drive belt on the small pulley before running it through the idlers and around the transaxle pulley. The double pulley is one unit. It is held on with a bolt up through the bottom of it. They can be tough to remove even with the correct pullers.
RIck, August 2006