I was replying to another thread, but then I had an idea (see end of post), so I posted it here to prevent it getting lost.
I created a file called a-default.conf in /etc/httpd/conf.d/ and added a <VirtualHost> block for each IP on my server.
(The filename isn’t important, but Apache loads all the .conf files in that directory in alphabetical order… you just need to name it something that Apache will load before the file for the first domain.)
The reason it’s not inherently built into the system is because, as Fr3d pointed out, the solution is to simply make a primary default config for each IP that gets loaded first alphabetically. I am more then willing to add fr3d’s suggestion as a feature request though, but I have to confer with the developers to see if its something they think should be added.