That will depend on your tape player basically there's no one setting fits all solution, you must adjust the volume to be high enough and yet making sure it won't cause any distortion on Audacity. This will vary from tape to tape. Check this guide from Gaming Alexandria in case you don't know it yet:
https://www.youtube.com/watch?v=OWpTHZC53YEBut you are having trouble after several tries it might be a problem with the adjustment of the playback head in your tape deck and the record head in the machine on which the tape was recorded, if the adjustments between both were too far ahead from each other it will lead to phase cancellation and your recordings won't be usable. If the difference isn't too high it might still be possible to manually fix the generated WAV file, but it will require someone with expertise to do so and I'm not such person unfortunately.
The best way to fix this is to either have access to a real machine to test the tape and try to load the game on it with some other player and if it works then use that same player to record it in audacity or try to calibrate your tape deck's azimuth. There's usually a tiny screw that allows you to do that, if you own/owned any 8-bit cassette based PC you have probably done this when your games fail to load. You can buy a setup calibration cassette tape to help on this process. There's a decent tutorial here:
http://www.endino.com/archive/cassettes.htmlIf you consider playing with this I'd suggest buying a cheap player to test it to make sure you won't completely screw up the azimuth of your current one in case it has been creating proper dumps for most tapes.