In recent years, deep neural architectures display groundbreaking performances in various speech processing area, including Text-To-Speech (TTS). Models have grown bigger, including more layers and millions of trainable parameters to achieve almost natural synthesis, at the expense of interpretability of computed intermediate representations, called embeddings. However, statistical learning performed by these neural models constitutes a valuable source of information about language. This presentation aims at openning this "black box" to explore intermediate embeddings computed by state-of-the-art TTS models. By identifying phonetic and acoustic features in model representations, the proposed methods help understanding how neural TTS are able to organize speech information on an unsupervised manner and provide new insights on phonetic regularities captured by statistical learning on massive data that are beyond human expertise. This work open the route toward designing more careful control architectures for neural TTS, without the need for additional data nor training process. These results led us to propose an auxiliary module for expressive synthesis called Local Style Tokens (LST), which models local variations in prosody with respect to the type of embeddings to bias.