You have no income tax liability, but depending upon the size of the remaining balance you may face federal gift tax issues. Your son has no tax liability; your generosity in paying off his mortgage is a gift to him.
Gifts in excess of an $12,000-per-donee annual exclusion are subject to gift tax, but the tax may be offset by a gift tax credit.