I have both (work uses Apple, I had Android before). It's not as bad as I expected but you do lose some functionality. I wouldn't switch unless you were having issues or have other Apple products you can sync with (iPad, Macbook).
Even if you had Samsung issues, there are other Android phones out there. Google Pixel and OnePlus are both good.