The transformer may have been damaged, but some damage to the cables are a distinct, and more likely, possibility.
Carefully inspect the low tention cables. Replace any damaged ones. Make sure that ALL connections are secure and make good contact. If the isolation show any heat damage, it mean that the contact is, or was, bad.
Also, inspect the whiring TO the transformer.
The original transformer surely have some kind of fault detection capability. When you bypassed the breaker, a backup feature reduced the tention provided to prevent any fire hasard. This, in turn, made the lights glow dim.
If you need to replace the transformer, you must make sure that it's rated AT LEAST as 300W, is grounded, have a built in breaker and fault protection.
The cheap transformer could not provide enough current. It's why it got so hot that some whires melted. You are lucky that it did not start a fire...
Electro, April 2013