Due to the popularization of the touch screen devices, nowadays people are used to conduct the human-computer interactions with touch gestures. However, limited by current studies, users can use only simple multi-touch gestures to execute only simple manipulations such as rotation, translation, scaling, with most of time one user even if adapted devices are now available. The work reported here concerns the expanding usage of multi-touch gestures, that make them available at the same time for more shortcut commands (called indirect commands, as copy, past, ... ), more manipulation commands (called direct commands like zoom or rotation) and in the context of multiple users on the same screen. For this purpose, we analyze the shape of the gesture's motion trajectories and the temporal and spatial relations between trajectories in order to characterize a multi-touch gesture. We propose a graph modeling to characterize these motion features and develop a graph based analysis and recognition system. To resolve the conflict between interface manipulation and shortcut command inputs, we study and validate an early recognition strategy for multi-touch gesture. We built a reject option based multi-classifier early recognition system to recognize multi-touch gestures in early stage. To set-up, train and validate our systems, we built MTGSet, a multi-touch gesture dataset formed by 7938 gestures from 41 different classes collected in isolated contexts and MUMTDB a dataset of gestures collected in a real multi-user usage case of diagram drawing. The experimental results prove that our system can well recognize multi-touch gestures in these different situations.