Lot of variables that could cause that. Giving you a single tip is like giving you a box of bandaids on a gushing wound.
Let's try and narrow down what's going on there. Where is the ball starting before it starts drifting right? Left of target, straight at your target, or right of target?