Nonlinear frequency chirping of toroidal Alfén eigenmodes (TAE) driven by energetic particles is investigated by kinetic simulations in toroidal plasmas. It is found that the up–down symmetry of the frequency chirping of a TAE is broken due to an anisotropic pitch-angle distribution with dominant co-passing energetic particles. The nonuniform distribution of the free energy associated with the initial energetic particle distribution causes biased driving forces that result in a strongly asymmetric frequency chirping. The evolution of the perturbed distribution function in the phase space shows that a hole–clump pair moves together towards the magnetic axis for the small pitch-angle parameter cases. The downward chirping of the mode frequency is associated with the negative drift of the phase island in the KAM surfaces or the resonance δf structures in the radial direction. On the other hand, the energetic particle distribution with larger pitch-angle parameters leads to upward chirping of the TAE frequency. The upward chirping is due to the drifting of the resonance structure towards the boundary of the simulation region and overlapping of different poloidal resonances in the (Λ, E) phase space at the late stage. The phase space dynamics provides a key mechanism for understanding the wave chirping direction and particle transport process.