Welcome to the forum.
Is the PC-8801 main unit set to 15kHz or 24kHz output? IIRC the PC-8801 needs to output in 15kHz mode to get rid of the line doubling.
Also keep in mind that PC-88/98 screens are naturally 16:10 aspect ratio, even on original 4:3 monitors. You can adjust the CRT beam to fill the monitor horizontally, but there should still be black bars on the top and bottom. If you adjust the picture to fill the entire monitor screen, the graphics will be distorted. If you run a circle command in BASIC, like CIRCLE(320,100),175, the output should look... circular.