The main goal of radiation therapy is to deliver a lethal dose of radiation to the targeted tumor while minimizing the radiation exposure to the surrounding normal tissues and critical organs. Modern cancer therapy has benefited enormously from computer controlled treatment devices with increased precision and capability. However, this increased sophistication also creates new challenges for treatment planning. As the number of parameters in a treatment plan increases, the traditional computational approaches are no longer adequate to fully exploit the potential of the latest treatment devices. This is because at the heart of treatment planning is often a set of substantially non-trivial constrained geometric optimization problems. We introduce a new optimization framework based on Particle Swarm Optimization for planning radiotherapy, and demonstrated its potential by solving an open problem in radiation therapy treatment planning. Specifically, we have applied the new framework to Gamma Knife® radiosurgery and high-dose rate (HDR) prostate brachytherapy. Mathematically, Gamma Knife® radiosurgery is a ball-packing process whose goal is to"pack" some spherical high dose volumes into a tumor volume, while that of HDR brachytherapy is to find the trajectories of some spherical high dose volumes. Both problems are computationally intractable. There is no known algorithm for brachytherapy that can generate the trajectories, which is currently done manually by a physician. The new framework models the spherical high dose volume as kinetic particles and simulates the "swarm" of these particles through a potential field created based on medical constraints and prescriptions. The resulting stable swarm, further refined by least distance programming, is the final treatment plan. We have conducted experiments with real and simulated data. Our results show that the new framework is at least equivalent to current clinical systems, but in the case of HDR prostate brachytherapy it can significantly outperform current clinical systems. We expect that this new framework, due to its clinical advantage, simplicity, and ease of understanding will be widely implemented in clinics making significant impact to Gamma Knife® radiosurgery, HDR brachytherapy, and other radiation therapy modalities.