The Indian Medical Association will consider taking back the police complaints lodged against yoga guru Ramdev and also the defamation notice sent to him if he withdraws his remarks against COVID-19 vaccines and modern medicine, IMA national chief Dr JA Jayalal said today.