Fidyah cannot be given to non-Muslims because it falls under the category of charity that is specific to Muslims.

Reference: Al-Syirazi, al-Muhadzdzab, Vol. 2, p. 76.