Slightly confusingly, DSP stands for both Digital Signal Processing and Digital Signal Processor, the latter being a special type of microprocessor chip that is designed specifically to carry out the former.
DSP calculations are used in practically every virtual instrument and effect, and usually they run on your computer's CPU, which is perfectly capable of performing such processing, although it is not designed purely for it.
A DSP system is a piece of add-on computer hardware that contains DSP chips. Once connected to your computer, you can use its plug-ins inside your music programs as though they were normal VSTs (or AU, RTAS, or whatever formats are catered for), but with the audio calculations being carried out using the DSP system's chips, rather than your computer's own CPU.
This means that those plug-ins will place practically no load on your computer, although there is a limit to how many plug-ins the DSP system can run at once. There are a number of different DSP platforms, the most popular being TC Electronic's PowerCore, Universal Audio's UAD and SSL's Duende, with a number of different pieces of hardware available for each (eg, some allow you to run more plug-ins than others).
Each of these companies is highly respected in the field of audio processing, and so one advantage of DSP systems is that the plug-ins available are of a consistently high quality, and indeed, the vast majority do not come in a non-DSP format, so you need the relevant DSP hardware to run them. You can't run PowerCore plug-ins on a Duende unit, for instance, although you can connect and use several different DSP systems at once.