Troubleshooting the Upgrade Process

There are two main reason why an upgrade fails:

  • Migration failures
    Lighthouse will return to the pre-upgrade state. An example of such a failure is malformed data in the database, which does not conform to more stringent schema checks in the new version.

  • System level failures
    This would include environmental issues such as power loss during the upgrade. If such an issue prevented Lighthouse from rolling back to the previous state, Lighthouse may be left in an unusable state.

Recovering from a failed upgrade

If Lighthouse is in an unusable state (either as a result of upgrade problems, or any other catastrophic failure), you can

  • Promote a dependent Lighthouse

    This requires multiple instances of Lighthouse. If the primary Lighthouse instance is unreachable, one of the existing dependent instances can be promoted to become the new primary Lighthouse instance.
    The old primary Lighthouse instance should be replaced with a fresh installation of Lighthouse. This can then be enrolled as a dependent to the newly promoted primary Lighthouse.

  • Restore a Configuration Backup
    Deploy a Lighthouse instance with the version you are upgrading from. Restore the configuration backup taken before the failed upgrade.

  • Manually retry via the CLI if a dependent upgrade fails.
    Determine the reason for the upgrade failure and resolve the issue before retrying the upgrade. Assistance from Opengear technical support may be required.

     

    Troubleshooting a failed dependent Lighthouse Upgrade


    To retry the dependent upgrade, first list the dependent and identify the IDs of the secondary lighthouses that need to be retried by running the retry_dependent_upgrades list command.

    retry_dependent_upgrades list command will list all the dependent lighthouses and their current status, as in the example below:

    root@lighthouse:~# retry_dependent_upgrades list
    ID UUID Hostname Firmware Version Fw Status Lighthouse Status
    2 lighthouse_configurations-2 lighthouse-2 22.11.2 not updated UpgradeFailed
    3 lighthouse_configurations-3 lighthouse-3 24.02.0 updated Enrolled
    4 lighthouse_configurations-4 lighthouse-4 22.11.2 not updated UpgradeFailed

    In the example above, the Lighthouses with IDs 2 and 4 have failed to upgrade (note that the “Enrolled” status is the desired status). To retrigger an upgrade for those lighthouses run the following command. Ensure that the lighthouse IDs are comma separated:

    retry_dependent_upgrades trigger -l 2,4

    The method above is the recommended way but there is a shortcut to retrigger all failed dependent upgrades by running the command below:

    retry_dependent_upgrades trigger --failed

    Note: Only one dependent will upgrade at a time.

    You should be able to view the dependent upgrade progress in the primary Lighthouse UI as you would for a normal upgrade.

     

Contact Support

In the unlikely event that Lighthouse does not automatically recover, or your troubleshooting efforts fail, contact Opengear support for advice. Provide the following:

  • If your instance is usable, download a Technical Support Report (from the Help menu) and share it with our team.

  • Share the configuration backup taken before the upgrade process.