There are 2 portions to the cast time of each skill:
1. Variable cast portion
2. Fixed cast portion
The variable part can be reduced by DEX and INT. Fixed can only be reduced by Sacrament, and similar effects.
The amount of Variable and Fixed cast each spell has depends on the spell itself. For example, Energy Coat is 100% fixed cast, but Soul Expansion is 100% variable. But spells like Chain Lightning lie somewhere in between. Most warlock spells have 0.5-1 second fixed cast, with a few having 2 seconds.
Take Chain Lightning, for example.
Fixed Cast Time: 1 second
Variable Cast Time: (3 + 0.5 * Skill Level) seconds
Skill re-use Delay: None
Global Skill Delay: 3 seconds
So Chain Lightning 5 has a 1 second fixed cast time, and 5.5 seconds variable.
This means, without any effects like Sacrament on you, no matter what your stats are, you cannot get it to cast quicker than 1 second.
With Sacrament and a +9 Bellum Arc Wand, however, you can reduce that fixed portion to 0.4 seconds.
However, if you had no DEX or INT, even with Sacrament and the Bellum Arc Wand, your cast time will still be 5.1 seconds (without any cast % reduction gear on).
It IS possible to do a 100% variable reduction through gear build. In which case, you could get away with 1 DEX.
Warlocks CAN compete in PVP. While they may not be the best, it is definitely fun lol. Though, I personally find Sorcerers more fun if you're gonna be soloing in PVP on a magic character.