Is the P your first long scale bass? It'll take awhile to teach your left hand how to do what you do on a longer scale. That flying fickle finger of fate* is a sign you're hand isn't ready to do what you're asking of it. As suggested above, switch to upright fingering (1-2-4) in the lower frets at least for a while. Take your time.
*And a tip of my hat to Laugh-In.
*And a tip of my hat to Laugh-In.