We formulate a framework for the depolarization of linearly polarized backscattered light based on the concept of geometric phase, {\it i.e} Berry's phase. The predictions of this theory are applied to the patterns formed by backscattered light between crossed or parallel polarizers. This theory should be particularly adapted to the situation in which polarized light is scattered many times but predominantly in the forward direction. We apply these ideas to the patterns which we obtained experimentally with backscattered polarized light from a colloidal suspension.