A number of scientific fields rely on placing permanent magnets in order to produce a desired magnetic field. We have shown in recent work that the placement process can be formulated as sparse regression. However, binary, grid-aligned solutions are desired for realistic engineering designs. We now show that the binary permanent magnet problem can be formulated as a quadratic program with quadratic equality constraints, the binary, grid-aligned problem is equivalent to the quadratic knapsack problem with multiple knapsack constraints, and the single-orientation-only problem is equivalent to the unconstrained quadratic binary problem. We then provide a set of simple greedy algorithms for solving variants of permanent magnet optimization, and demonstrate their capabilities by designing magnets for stellarator plasmas. The algorithms can a-priori produce sparse, grid-aligned, binary solutions. Despite its simple design and greedy nature, we provide an algorithm that compares with or even outperforms the state-of-the-art algorithms while being substantially faster, more flexible, and easier to use.