Electrowetting (EW) of water drops in ambient oil has found a wide range of applications including lab-on-a-chip devices, display screens, and variable focus lenses. The efficacy of all these applications is dependent on the contact angle hysteresis (CAH), which is generally reduced in the presence of ambient oil due to thin lubrication layers. While it is well-known that AC voltage reduces the effective contact angle hysteresis (CAH) for EW in ambient air, we demonstrate here that CAH for EW in ambient oil increases with increasing AC and DC voltage. Taking into account the disjoining pressure of the fluoropolymer-oil-water system, short range chemical interactions, viscous oil entrainment, and electrostatic stresses, we find that this observation can be explained by progressive thinning of the oil layer underneath the drop with increasing voltage. This exposes the droplet to the roughness of the underlying solid and thereby increases hysteresis.