Many multicast overlay networks maintain application-specific performance goals such as bandwidth, latency, jitter and loss rate by dynamically changing the overlay structure using measurement-based adaptation mechanisms. This results in an unstructured overlay where no neighbor selection constraints are imposed. Although such networks provide resilience to benign failures, they are susceptible to attacks conducted by adversaries that compromise overlay nodes. Previous defense solutions proposed to address attacks against overlay networks rely on strong organizational constraints and are not effective for unstructured overlays. In this work, we identify, demonstrate and mitigate insider attacks against measurement-based adaptation mechanisms in unstructured multicast overlay networks. The attacks target the overlay network construction, maintenance, and availability and allow malicious nodes to control significant traffic in the network, facilitating selective forwarding, traffic analysis, and overlay partitioning. We propose techniques to decrease the number of incorrect or unnecessary adaptations by using outlier detection. We demonstrate the attacks and mitigation techniques in the context of a mature, operationally deployed overlay multicast system, ESM, through real-life deployments and emulations conducted on the PlanetLab and DETER testbeds, respectively.