We report on the occurrence of strong nonlinear acousto-optic interactions in phoxonic structures, that support, simultaneously, acoustic and optical localized resonant modes, under the influence of acoustic losses. Deploying a detailed theoretical investigation of the acousto-optic coupling in the specific case of a one-dimensional phoxonic cavity, realized by homogeneous SiO₂ and Si layers, we demonstrate the possibility for an enhanced modulation of light with sound through multi-phonon exchange mechanisms. A full electrodynamic and elastodynamic multiple scattering approach is employed to describe the optical and acoustic modes, and to account for their mutual interaction and the underlying effects both in time and frequency domains. In particular, we discuss the influence of hypersonic attenuation on the acousto-optic interaction by considering typical acoustic losses in the GHz regime.