If this is something you run into often, it’s likely still only for a limited number of servers? ssh and scp both respect .ssh/config, and I suspect (but haven’t tested) that sftp does too. If you add something like this to that file:

Host host1 host2
  Port 8080

then SSH connections to hosts named in that first line will use port 8080 by default and you can leave off the -p/-P when contacting those hosts. You can add multiple such sections if you have other hosts that require different ports, of course.