We witnessed, in the last few years, the massive expansion of the Internet of Things (IoT) to multiple everyday devices, in homes, buildings and factories. The inclusion of heterogeneous wireless transceivers in potentially critical appliances and environments is then raising growing security and safety concerns. However, according to the litterature, there is still room for improvement, especially in off-the-shelf devices, often not implementing the different security measures of the protocols they use. As such, numerous works focus on the improvement of the global security and safety of the IoT, and more generally in the wireless communications often used by these devices. The subject of the physical layer security is often disregarded, because of its complexity or the fact that it is less critical in more traditional wired networks. However, the wireless communication medium can have a substantial impact on the attack surface because of its higher availability, since anyone in radio range can listen to communications. It also brings new vulnerabilities, that are specific to this kind of transmissions. Furthermore, the low complexity and computational power of the devices in use often results in simple protocols, making device cloning and injections more easy, and harder to detect from higher layers. Thus, in this thesis, we decided to focus on the impact of the wireless physical layer on network security, and highlight the importance of transversal approaches between signal processing and cybersecurity. We first show the new dimension introduced by the common wireless medium in offensive security, by showing a new attack based on similarities in physical layers of two protocols to break the isolation between them. This attack, Wazabee, allows Bluetooth Low Energy transceivers to communicate seemlessly with Zigbee networks. We also show that the principle behind it could be extended to other pairs of protocols, should the conditions be met. This highlights the importance of a more protocol-agnostic monitoring of the wireless medium, able to detect communications outside the legitimate protocols of the environment, and to not consider co-existing wireless protocols as isolated. Then, we propose two defensive contributions, to help secure wireless networks by analysing the communications from a physical layer perspective. First, we present a low-cost fingerprinting approach for wireless devices, to detect potential identity theft attacks, i.e. attacks where an illegitimate device tries to take the place of a legitimate one in the network. Then, we present an approach for automated wireless protocol audit, allowing to detect and analyze various protocols emitting in a wide range of frequencies, with minimal assumptions on their nature. These two approaches complete higher-layer security measures, to detect potential intruders or covert channels in the environment, and to ease protocol analysis for security experts.