Tight tolerances have been a leading driver of cost in recent stellarator experiments, so improved definition and control of tolerances can have significant impact on progress in the field. Here we relate tolerances to the shape gradient representation that has been useful for shape optimization in industry, used for example to determine which regions of a car or aerofoil most affect drag, and we demonstrate how the shape gradient can be computed for physics properties of toroidal plasmas. The shape gradient gives the local differential contribution to some scalar figure of merit (shape functional) caused by normal displacement of the shape. In contrast to derivatives with respect to quantities parameterizing a shape (e.g. Fourier amplitudes), which have been used previously for optimizing plasma and coil shapes, the shape gradient gives spatially local information and so is more easily related to engineering constraints. We present a method to determine the shape gradient for any figure of merit using the parameter derivatives that are already routinely computed for stellarator optimization, by solving a small linear system relating shape parameter changes to normal displacement. Examples of shape gradients for plasma and electromagnetic coil shapes are given. We also derive and present examples of an analogous representation of the local sensitivity to magnetic field errors; this magnetic sensitivity can be rapidly computed from the shape gradient. The shape gradient and magnetic sensitivity can both be converted into local tolerances, which inform how accurately the coils should be built and positioned, where trim coils and structural supports for coils should be placed, and where magnetic material and current leads can best be located. Both sensitivity measures provide insight into shape optimization, enable systematic calculation of tolerances, and connect physics optimization to engineering criteria that are more easily specified in real space than in Fourier space.