I just tested my HD5450 in an x1 slot, it works perfectly. This is on my x58 UD3R board but it proves that at least on some systems you can use x1 slots for GPU's. Performance did not appear to be down either but then the 5450 is not a great performer. I used a passive PCI x1 to x16 slot converter which just allows you to do this without having to butcher a slot, the card was working at x1 speeds but is a full x16 card.
I would suggest your problem is the card, have you tried the x1 card in the x16 slot? Just to rule it out as a card issue.