Could be caused by: the digitiser/dongle, the VCR (heads or lack of stabilisation) or a bad tape. Unfortunately, it's a process of elimination here.
Before starting, check you have the "Video Stndard" set correctly ie NTSC or PAL (try PAL_B and then try other PAL settings).
First, hook up your VCR direct to your TV (use the AV input) and see how the tape plays.
If that's not OK, try another tape or another VCR if you can get one.
If it's OK on your TV, it's probably a weak-ish signal, enough for the TV but not for the digitiser. You probably need a video stabiliser such as a DVD recorder (or a proper Timebase Corrector) to pass the video signal through and stabilise it. Here are some examples: