Establishing temporal order between events and resolving bridging references are crucial for automatic discourse understanding. For that, effective event and mention representations are essential to accurately solve temporal relation classification and bridging resolution. This thesis addresses exactly that and designs novel approaches to obtain more expressive event and mention representations.Contextual and commonsense information are needed for obtaining such effective representations. However, acquiring and injecting them is a challenging task because, on the one hand, it is hard to distinguish useful context itself from bigger paragraphs or documents and also equally difficult to process bigger contexts computationally. On the other hand, obtaining commonsense information like humans acquire, is still an open research question. The earlier attempts of hand engineered event and mention representations are not sufficient for acquiring contextual information. Moreover, most of the approaches are inadequate at capturing commonsense information as they again resorted to hand-picky approaches of acquiring such information from sources like dictionaries, web, or knowledge graphs. In our work, we get rid of these inefficacious approaches of getting event and mention representations.First, we obtain contextual information to improve event representations by providing neighboring n-words of the event. We also use character-based representation of events capture tense, and aspect information from the internal structure of event head words. We also go a step further and learn interactions over these event representations to get rich event-pair representations. We find that our improved event representations demonstrate substantial gains over an approach which relied only on the event head embeddings. Also, our ablation study proves the effectiveness of complex interaction learning as well as the role of character embeddings. Next, we probe transformer language models(e.g. BERT) that are proved to be better at capturing context. We investigate specifically for bridging inference to understand the capacity of these models at capturing bridging information. The purpose of this investigation is to use these understandings for making informed decisions at designing better transformer models to further improve mention representations. For that, we examine the model's internal structure individually and then the whole model. The investigation shows that pre-trained models are surprisingly good at capturing bridging information and these capabilities are highly context dependent, as they perform poorly with distorted contexts. Further, our qualitative analysis shows that BERT is capable of capturing basic commonsense information but fails to capture sophisticated information which is required for bridging resolution. Finally, we combine both contextual and commonsense information for further improving event and mention representations. We inject commonsense information with the use of knowledge graphs for both temporal relation classification and bridging anaphora resolution tasks. We take a principled approach at acquiring such knowledge where we employ graph node embeddings learned over knowledge graphs to capture the overall topology of the graph as a result gaining holistic external information. Specifically, we used graph node embeddings learned over Wordnet and TEMPROB and combined them with text-only embeddings like word2vec, BERT, etc. to obtain knowledge-aware representations. We evaluated our approach over standard datasets like ISNotes, BASHI, and ARRAU for bridging anaphora resolution and MATRES for temporal relation classification. We observe substantial gains in performances over text-only representations on both tasks proving the effectiveness of our approach.