The crumbly stuff may well be old lime plaster - it you drill it for fixings it will just fall away leaving big holes.
If it were my problem I would take a piece of decent plank wood - cut it generously oversize and fix the wood to the wall with Frame Fixings like this...
http://www.jpbolts.co.uk/imgs/products/lrg/f_f_z_p_m8_m10.jpg
Four of them, one at each corner and long enough to get at least 40mm into the underlying brickwork so probably 60mm long to allow for the wood and the plaster.
I would then screw the TV bracket to the wood using appropriate wood screws.
Peccavi, August 2009