Housing loan borrowers who have chosen for fixed rates might have trouble free terms on their finance as the Reserve Bank of India has recommended banks to modify the penalty arrangement on prepayments and charge only on the remaining amount.