The procedure here: SolarWinds Knowledge Base :: How to Upgrade SolarWinds Orion when Orion Failover Engine is Installed is VERY subtle around steps 1.d, 5-10, and 11a
One has to turn off the FOE on the primary server at step 1, and then you must never have it running on both servers at the same time until step 13 (after all of the upgrades)
The steps are there to ensure that only one server has a working NIC at any time the applications are running.
I think they have the secondary upgrade first to ensure the FOE is working before you start the upgrade procedure
If the failover engine replicates any files between the servers after the first server upgrade then the installer fails subtly on the second server upgrade (because it thinks it's already done bits of the upgrade on that server), and then possibly replicates a bad install config files back to the first server you did :/
I'd really like a non-stop in-service upgrade process for the FOE!!!!!