[PATCH] uefirtmisc: add checking the resources for testing

[Sunny Wang]

Thanks for clarifying, Ivan.

The problem is that we may NOT know the details about the firmware implementation of runtime services. Any runtime service test can be related to SetVariable, so the returned error status may not always be EFI_OUT_OF_RESOURCES.
For the edk2 implementation, this issue (using up UEFI variable storage space) caused different error statuses from the following test cases. For proprietary implementation, that will be more and more possibilities.
   - For uefirtvariable, it gets EFI_BAD_BUFFER_SIZE.
   - For uefirttime, it gets EFI_DEVICE_ERROR because the implementation tries to emulate the time device by using UEFI variable storage.
  -  For uefirtmisc, it gets EFI_OUT_OF_RESOURCES. Actually, the EFI_OUT_OF_RESOURCES is NOT one of the returned statuses in UEFI spec GetNextHighMonotonicCount() section, so the current solution may not make completely sense.

In other words, the current solution doesn't solve the difficult situation that we currently face with. When the problem occurs (after running the stress test), we will still run into the issues below:
   - System will still no longer be able to run any test or run any applications/commands that are related to SetVariable (even no longer be able to install an OS or change setup option as the SetVariable won’t work). In the worst case, it would look like the FWTS test bricks the system. For recovering the system, we have to clear UEFI variable storage, which would require a reboot for using a tool or an option in the setup menu or re-flashing the firmware. For some ARM systems, we even need to use HW flasher/JTAG programmer to recover the system.
   - We would still get the false positive with some tests after uefirtmisc. Again, we don't know whether the firmware implementation is based on the UEFI variable or not and don't know what error status will be returned. This would also cause difficulty in debugging the failures.

Therefore, I would still like to know why we need the stress tests. If we don't have a reason to do this, can we remove them? If the stress test is not for any requirement in UEFI, ACPI, or other specs, can we enable/skip that by using a flag/option? If we have a concern about skipping the stress test, can we reduce the number/count of runs for the stress test?

Best Regards,
Sunny
-----Original Message-----
From: ivanhu <ivan.hu at canonical.com>
Sent: 20 January 2022 01:53
To: Sunny Wang <Sunny.Wang at arm.com>; fwts-devel at lists.ubuntu.com
Subject: Re: [PATCH] uefirtmisc: add checking the resources for testing

Hi Sunny,

On 1/19/22 5:23 PM, Sunny Wang wrote:
> Hi Ivan,
>
> Thanks for working on this.
>
> However, if my understanding is correct, this solution won't work. The UEFI variable storage space is still used up by uefirtmisc test. Then, any SetVariable related test following uefirtmisc would still fail due to no remaining space.

They are supposed to "SKIP" as well when running out of resources. If
not, file another bug for it and attach the result.log.

> For example, when we run " fwts -r stdout -q --sbbr", uefirtvariable uefirttime will be run after uefirtmisc. Then, both tests would still fail.
> It looks like you would like to skip the stress test for the system that doesn't have large UEFI variable storage space. If so, I think you can call QueryVariableInfo() and check the RemainingVariableStorageSize. It will be good to skip all the stress test if the RemainingVariableStorageSize is smaller than 64k.

It is not my intention to SKIP the tests. It is due to run out of
resources, we cannot get the conclusion of the result, PASS or FAIL, so
SKIP instead. Besides, it is not a good idea to test a runtime service
which depends on another runtime service, we cannot make sure all
runtime services work fine, it might get more false alarms. I prefer
keeping it simple.

Cheers,
Ivan

>
> Best Regards,
> Sunny
>
> -----Original Message-----
> From: fwts-devel <fwts-devel-bounces at lists.ubuntu.com> On Behalf Of Ivan Hu
> Sent: 19 January 2022 08:34
> To: fwts-devel at lists.ubuntu.com
> Subject: [PATCH] uefirtmisc: add checking the resources for testing
>
> BugLink: https://bugs.launchpad.net/fwts/+bug/1958206
>
> Signed-off-by: Ivan Hu <ivan.hu at canonical.com>
> ---
>  src/uefi/uefirtmisc/uefirtmisc.c | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/src/uefi/uefirtmisc/uefirtmisc.c b/src/uefi/uefirtmisc/uefirtmisc.c
> index f6038f5f..db33ad66 100644
> --- a/src/uefi/uefirtmisc/uefirtmisc.c
> +++ b/src/uefi/uefirtmisc/uefirtmisc.c
> @@ -91,6 +91,11 @@ static int getnexthighmonotoniccount_test(fwts_framework *fw, uint32_t multitest
>                                         "service is not supported on this platform.");
>                                 return FWTS_SKIP;
>                         }
> +                       if (status == EFI_OUT_OF_RESOURCES) {
> +                               fwts_skipped(fw, "Skipping test, run out of resources for "
> +                                       "GetNextHighMonotonicCount runtime service test.");
> +                               return FWTS_SKIP;
> +                       }
>                         fwts_failed(fw, LOG_LEVEL_HIGH, "UEFIRuntimeGetNextHighMonotonicCount",
>                                 "Failed to get high monotonic count with UEFI runtime service.");
>                         fwts_uefi_print_status_info(fw, status);
> --
> 2.25.1
>
>
> --
> fwts-devel mailing list
> fwts-devel at lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/fwts-devel
> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
>
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.