We present the first 3D radiation-hydrodynamic simulations of directly driven inertial confinement fusion implosions with an inline package for polarized crossed-beam energy transfer, which were used to assess the impact of the current distributed polarization rotators (DPRs) on OMEGA as well as other known sources of asymmetry. Applied to OMEGA implosions, the simulations predict bang times with no need for ad hoc multipliers, as well as yields—if you separately account for the impacts of imprint and fuel age. The magnitude of the flow is well reproduced when the low mode sources are large, whereas the modeling of the stalk is thought to be required to match the flow magnitude in the remaining cases. For the cases explored in more detail, polarized cross-beam energy transfer (CBET)—the only known systematic drive asymmetry, brought the results closest to the measured flow vectors. The remaining discrepancies are shown to be stemming from the limited knowledge of the laser pointing modes. For typical current levels of beam mispointing, power imbalance, target offset, and asymmetry caused by polarized CBET, low modes degrade the yield by more than 40%. The current strategy of attempting to compensate the mode-1 asymmetry with a preimposed target offset recovers only about one-third of the losses caused by the low modes due to the dynamic nature of the multiple asymmetries and the presence of low modes other than l = 1. Therefore, addressing the root causes of the drive asymmetries is apt to be more beneficial. To that end, one possible solution to the specific issue of polarized CBET (10 µm DPRs) is shown to work well.