As one of the most active research areas in computer vision, visual analysis of human motion attempts to detect, track and identify people, and more generally, to interpret human behaviors, from image sequences involving humans. The main concern of this dissertation is the issue of shape analysis of imaging data with application to human behavior analysis. In particular, to filter some undesirable transformations, the shape extracted from the human body and face are represented as elements of a shape space defined as the invariant under the action of groups modeling the undesirable transformations. The main contribution presented in this dissertation is a unified framework for human behavior analysis through multiple manifolds representing different data, with different applications ranging from action recognition to soft-biometrics estimation including facial expression analysis and classification. First, the landmarks issued from the skeleton or facial landmarks were modeled on Kendall shape space where the comparison is invariant to scale, translation and rotation. An intrinsic sparse coding and dictionary learning SCDL on the Kendall Shape Space were performed with application to action and expression recognition using dynamic landmarks. A comparative study to an extrinsic sparse coding is also presented to understand the benefit of each methodology. Second, the facial curves were viewed as points on an infinite-dimensional, dfferentiable manifold and shooting vector along a geodesic representing the deformations between 3D faces has been proposed with application to soft-biometric recognition from 3D faces and expression recognition from 3D dynamic faces. Finally, a framework for 3D parametrized surfaces is presented. We present the algorithms to calculate geodesic paths, distances and intrinsic means. A novel idea based on gauge theory capable to compute the geodesic paths on shape space without any need to filter the re-parameterization group is proposed. Experiments conducted on the main benchmarks of action, facial expression and soft-biometric recognition demonstrate the efficiency of the proposed framework on the task of human behavior understanding.