Troubleshooting the Upgrade Process

There are two main reason why an upgrade fails:

  • Migration failures
    Lighthouse will return to the pre-upgrade state. An example of such a failure is malformed data in the database, which does not conform to more stringent schema checks in the new version.

  • System level failures
    This would include environmental issues such as power loss during the upgrade. If such an issue prevented Lighthouse from rolling back to the previous state, Lighthouse may be left in an unusable state.

Recovering from a failed upgrade

If Lighthouse is in an unusable state (either as a result of upgrade problems, or any other catastrophic failure), you can

  • Promote a Secondary Lighthouse

    This requires multiple instances of Lighthouse. If the primary Lighthouse instance is unreachable, one of the existing secondary instances can be promoted to become the new primary Lighthouse instance.
    The old primary Lighthouse instance should be replaced with a fresh installation of Lighthouse. This can then be enrolled as a secondary to the newly promoted primary Lighthouse.

  • Restore a Configuration Backup
    Deploy a Lighthouse instance with the version you are upgrading from. Restore the configuration backup taken before the failed upgrade.

  • Manually retry via the CLI if a secondary upgrade fails.
    Determine the reason for the upgrade failure and resolve the issue before retrying the upgrade. Assistance from Opengear technical support may be required.

     

    Troubleshooting a failed Secondary Lighthouse Upgrade


    To retry the secondary upgrade, first list the secondaries and identify the IDs of the secondary lighthouses that need to be retried by running the retry_secondary_upgrades list command.

    retry_secondary_upgrades list command will list all the secondary lighthouses and their current status,as in the example below:

    root@lighthouse:~# retry_secondary_upgrades list
    ID UUID Hostname Firmware Version Fw Status Lighthouse Status
    2 lighthouse_configurations-2 lighthouse-2 22.11.2 not updated UpgradeFailed
    3 lighthouse_configurations-3 lighthouse-3 23.10.0 updated Enrolled
    4 lighthouse_configurations-4 lighthouse-4 22.11.2 not updated UpgradeFailed

    In the example above, the Lighthouses with IDs 2 and 4 have failed to upgrade (note that the “Enrolled” status is the desired status). To retrigger an upgrade for those lighthouses run the following command. Ensure that the lighthouse IDs are comma separated:

    retry_secondary_upgrades trigger -l 2,4

    The method above is the recommended way but there is a shortcut to retrigger all failed secondary upgrades by running the command below:

    retry_secondary_upgrades trigger --failed

    Note: Only one secondary will upgrade at a time.

    You should be able to view the secondary upgrade progress in the primary Lighthouse UI as you would for a normal upgrade.

     

Contact Support

In the unlikely event that Lighthouse does not automatically recover, or your troubleshooting efforts fail, contact Opengear support for advice. Provide the following:

  • If your instance is usable, download a Technical Support Report (from the Help menu) and share it with our team.

  • Share the configuration backup taken before the upgrade process .