Leaving the payment of fidyah without a permissible reason is a sin because it is an obligation that must be fulfilled.

Reference: Al-Syirazi, al-Muhadzdzab, Vol. 2, p. 75.