[FWTS-Live] fwts MTRR tests fail on AMD platform if MMIO is above 4G
Tung, Larry
Larry.Tung at amd.com
Fri Oct 23 08:09:36 UTC 2020
[AMD Official Use Only - Internal Distribution Only]
Hi Alex,
I've submitted a bug below,
https://bugs.launchpad.net/fwts/+bug/1901146
=====================
The effective memory type can be override by PAT mechanism and MTRRs.
The PAT(MSR0000_0277) register contains 8 page-attribute fields from PA0 to PA7. PA fields are selected by using three bits from the page-table entries {PAT, PCD, PWT}, and each fields have corresponding memory type in the PAT register.
If the memory range is not specified using the fixed-range and variable-range MTRRs, the default memory type in MTRRdefType(MSR0000_02FF) would be used.
The combination of MTRR and PAT memtype are described in AMD64 Architecture Programmer's Manual Volume 2 (#24593)
https://www.amd.com/system/files/TechDocs/24593.pdf
[cid:image001.jpg at 01D6A956.E40A7320]
Best Regards,
Larry
From: Alex Hung <alex.hung at canonical.com>
Sent: Thursday, October 22, 2020 3:52 PM
To: Tung, Larry <Larry.Tung at amd.com>
Cc: Lien, Eliot <Eliot.Lien at amd.com>
Subject: Re: [FWTS-Live] fwts MTRR tests fail on AMD platform if MMIO is above 4G
[CAUTION: External Email]
On Thu, Oct 22, 2020 at 1:33 AM Tung, Larry <Larry.Tung at amd.com<mailto:Larry.Tung at amd.com>> wrote:
[AMD Official Use Only - Internal Distribution Only]
Hi Alex,
Thanks for the information.
After clarifying, the TOM2-WB (Sys_Cfg[22]=1) would be override with PAT (MSR0000_0277) memtype. The page-attribute used to access the PCIe video controller MMIO should be WC.
So, I wonder if it's possible to add a "page-attribute" checking into the test tool ?
Certainly. We really appreciate your feedback and technical details. I was able to find a public AMD datasheet "54945_PPR_Family_17h_Models_00h-0Fh.pdf", but I don't quite understand how it overrides TOM2-WB.
Will you be able to share some details?
FYI, you can also send emails to our mailing list "fwts-devel at lists.ubuntu.com<mailto:fwts-devel at lists.ubuntu.com>" where more people can discuss if all information is publicly available.
Thanks,
Best Regards,
Larry
From: Alex Hung <alex.hung at canonical.com<mailto:alex.hung at canonical.com>>
Sent: Thursday, October 22, 2020 3:06 AM
To: Tung, Larry <Larry.Tung at amd.com<mailto:Larry.Tung at amd.com>>
Cc: Lien, Eliot <Eliot.Lien at amd.com<mailto:Eliot.Lien at amd.com>>
Subject: Re: [FWTS-Live] fwts MTRR tests fail on AMD platform if MMIO is above 4G
[CAUTION: External Email]
Hi,
That MSR is used to check memory types such as below:
/* Get the default memory type of memory between 4GB and second top of
* memory (TOM2) - i.e. is it write back (WB)
*/
if (strstr(fwts_cpuinfo->vendor_id, "AMD") || strstr(fwts_cpuinfo->vendor_id, "Hygon")) {
if (fwts_cpu_readmsr(fw, 0, AMD_SYS_CFG_MSR, &amd_sys_conf) == FWTS_OK)
if (amd_sys_conf & 0x200000)
amd_Tom2ForceMemTypeWB = true;
}
....
/* On AMD platforms, Tom2ForceMemTypeWB overwrites other memory types */
if (amd_Tom2ForceMemTypeWB && start >= 0x100000000) {
type = WRITE_BACK;
return type;
}
However, memory-mapped IO (like this one) should not be write-back:
"Write back is a storage method in which data is written into the cache every time a change occurs, but is written into the corresponding location in main memory only at specified intervals or under certain conditions.
PCI memory-mapped I/O registers aren't RAM that stays the same until the CPU changes them - and MMIO can change by itself to reflect hardware changes unlike memory. As a result, it is unsuitable for the CPU to write to MMIO "only at specific intervals or under certain conditions".
The scenario should be: you write a command to PCI MMIO, expect the PCI hardware to react as soon as writes are issued, and want to read an ack from it; however the PCI hardware will not do anything until many milliseconds later, long after your reads....
There are similar discussion @ https://community.intel.com/t5/Intel-ISA-Extensions/TSX-and-PCI-consistent-memory/td-p/1107558<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommunity.intel.com%2Ft5%2FIntel-ISA-Extensions%2FTSX-and-PCI-consistent-memory%2Ftd-p%2F1107558&data=04%7C01%7CLarry.Tung%40amd.com%7C337e705aab604c47282708d8765f7261%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637389499601755643%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MYGzTQbZ9cAwTJ40oIoVDe90oZ1oRIZU73tyz7BiANk%3D&reserved=0>, and I quote
=======================
FYI: The WB type will not work with memory-mapped IO. You can program the bits to set up the mapping as WB, but the system will crash as soon as it gets a transaction that it does not know how to handle. It is theoretically possible to use WP or WT to get cached reads from MMIO, but coherence has to be handled in software.
=======================
On Wed, Oct 21, 2020 at 3:04 AM Tung, Larry <Larry.Tung at amd.com<mailto:Larry.Tung at amd.com>> wrote:
[AMD Official Use Only - Internal Distribution Only]
Hi Alex,
On AMD system, there is a MSR to force the 4GB ~ TOM2 as WB memory type. However, we encounter MTRR test failed on PCIe GFx resource with incorrect WB memory type, if BAR is assigned to above 4G memory range. I looked into /mtrr.c form git hub, code logic seems would check if PCIe BAR is prefetchable, then it will return the "mustnot" mask as WB which may cause this failure.
May I have your comment on this please? Thanks
1.
/* if it's PCI mmio -> uncached typically except for video */
if (strstr(string, "0000:")) {
bool pref;
if (check_prefetchable(fw, string, address, &pref) != FWTS_OK)
return FWTS_ERROR;
if (pref) {
*must = 0;
*mustnot = WRITE_BACK;
} else {
*must = UNCACHED;
*mustnot = (~UNCACHED) & (~DEFAULT);
}
}
1.
if ((type & type_mustnot) != 0) {
failed++;
fwts_failed(fw, LOG_LEVEL_CRITICAL,
"MTRRIncorrectAttr",
"Memory range 0x%" PRIx64 " to 0x%" PRIx64 " (%s) "
"has incorrect attribute%s.",
start, end,
c2, cache_to_string(type & type_mustnot));
if (type_must == UNCACHED)
skiperror = true;
}
[cid:image003.jpg at 01D6A956.E40A7320]
[cid:image005.jpg at 01D6A956.E40A7320]
Error log with FWTS Version V20.07.00
mtrr: MTRR tests.
--------------------------------------------------------------------------------
MTRR overview
-------------
Reg 0: 0x0000000000000000 - 0x000000007fffffff ( 2048 MB) Write-Back
Reg 1: 0x0000000080000000 - 0x00000000bfffffff ( 1024 MB) Write-Back
Test 1 of 3: Validate the kernel MTRR IOMEM setup.
FAILED [CRITICAL] MTRRIncorrectAttr: Test 1, Memory range 0xfce0000000 to
0xfcefffffff (0000:08:00.0) has incorrect attribute Write-Back.
FAILED [CRITICAL] MTRRIncorrectAttr: Test 1, Memory range 0xfcf0000000 to
0xfcf01fffff (0000:08:00.0) has incorrect attribute Write-Back.
Test 2 of 3: Validate the MTRR setup across all processors.
PASSED: Test 2, All processors have the a consistent MTRR setup.
Test 3 of 3: Test for AMD MtrrFixDramModEn being cleared by the BIOS.
PASSED: Test 3, No MtrrFixDramModEn error detected.
Best Regards,
Larry
--
Cheers,
Alex Hung
--
Cheers,
Alex Hung
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/fwts-devel/attachments/20201023/d2152d0f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 88184 bytes
Desc: image001.jpg
URL: <https://lists.ubuntu.com/archives/fwts-devel/attachments/20201023/d2152d0f/attachment-0003.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.jpg
Type: image/jpeg
Size: 12314 bytes
Desc: image003.jpg
URL: <https://lists.ubuntu.com/archives/fwts-devel/attachments/20201023/d2152d0f/attachment-0004.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.jpg
Type: image/jpeg
Size: 38936 bytes
Desc: image005.jpg
URL: <https://lists.ubuntu.com/archives/fwts-devel/attachments/20201023/d2152d0f/attachment-0005.jpg>
More information about the fwts-devel
mailing list