Ideal capacitors have zero inductance and zero resistance. Real-world capacitors, unfortunately, have a non-zero inductance (called "parasitic inductance") which limits the speed at which a capacitor can discharge.
There are capacitors made that are described as "low inductance capacitors" or "high-frequency capacitors".
Guides for making home-made high-voltage low-inductance capacitors: