We show that, in the presence of a complex spectrum, antiresonances act as a precursor for dephasing enabling the crossover to a fully decoherent transport even within a unitary Hamiltonian description. This general scenario is illustrated here by focusing on a quantum dot coupled to a chaotic cavity containing a finite, but large, number of states using a Hamiltonian formulation. For weak coupling to a chaotic cavity with a sufficiently dense spectrum, the ensuing complex structure of resonances and antiresonances leads to phase randomization under coarse graining in energy. Such phase instabilities and coarse graining are the ingredients for a mechanism producing decoherence and thus irreversibility. For the present simple model one finds a conductance that coincides with the one obtained by adding a ficticious voltage probe within the Landauer-Büttiker picture. This sheds new light on how the microscopic mechanisms that produce phase fluctuations induce decoherence.