I have a similar issue, but instead I'm using mod_alias to have selfoss in a directory outside my main docroot. I think it's something to do with all the mod_rewrite stuff that's going on in .htaccess

I have the same issue. Yeah, it looks related to mod_rewrite.

After some investigations,
I found that :

1/ www.mydomain.com/selfoss (where my domain.com is going to /home/user1/www): OK
--
2/ mydomain2.com/~user1/selfoss where mydomain2 is going to server_ip : NOK
3/ my_ip/~user1/selfoss (going to the same path /home/user1/www): NOK

2/ and 3/ are almost the same, and I get :
"The requested URL /home/~user1/www/selfoss/index.php was not found on this server"

I confirm index.php is here with the good permissions (and working in the 1st exemple.)

Apache logs say : "File does not exist: /var/www/home" (which is my %DOCUMENT _ROOT%/home)

It really sounds like a .htaccess issue, .htaccess looking into the wrong directory.
I think it could be an important issue since userdir_mod is used in most schools, universities, cool business places for employees, etc. I'm trying to fix it on my side but I'll probably need others eyes since I'm not very familiar with .htaccess directives.