That’s obviously not gonna work.
Even if OF was on tha same version as Retail (which is not) you’d have to have two different clients installed, one for OF and one for Retail, since pservers need to patch certain files if they want to function as pservers at all.
That’s why if you launch OF, it’ll patch those files rendering Retail unplayable, and vica versa. However, as OF is way behind the current version (Retail is v599, OF is v491), it’s not only patches the files required for it to launch at all, but also re-downloads about half the game, for there’s a lot of difference in data between the two.
So as to directly answer your question, yeah it’s completely normal behavior, and you’re doing it wrong. You need two (2) copies of the game, one for Retail BDO and another one for OgreFest BDO if you really want to play on both servers simultaneously.
Also, for OF it is suggested to be downloaded through our launcher forasmuch some people reported issues with a downgraded/patched Retail client.