A setting of 6 on the volume control corresponds to a clockface settting of 1 O'Clock on the LH500. I am not familiar with this amp, but many amps are well beyond their clean power limits when the volume is turned up this high.
I could be wrong, but I suspect the rig is just at it limits when turned up to 4. I'll try to explain: The level of the volume control only tells you how much of the amp's potential gain you are using, not how much headroom you have left. The control on your amp goes from 0-10, but it's a mistake to assume that setting the control to 5 will give you half of the amp's potential volume. The level of the input signal must be considered as well. Turn down your bass's output and you can turn the amp up more before it distorts. Basically this entails reducing gain in your bass's preamp and making up for the difference in the amp, but the totally gain remains the same. Unfortunately this will not allow you to play louder and once the amp hit's it power limits it will begin to distort, regardless of where the volume control is set.
So I suspect your probably need either another speaker and/or a bigger amp. If you have an 8 ohm 410XL, the LH500 only makes 350W. The good news is you can add a second XL410 and the amp will make more power (500W at 4 ohms) and you will benefit from mutual coupling between the speakers. The mutual coupling will give you bump in efficiency in the low end and fill out the sound.
Good luck