mm/huge_memory: fix huge_zero_pfn race

Overview

In the Linux kernel, the following vulnerability has been resolved:

mm/huge_memory: fix huge_zero_pfn race

Patch series "mm/huge_memory: fix huge_zero_pfn race", v2.

There is a subtle race in the reference-counted huge_zero_folio implementation.

The fast path atomic logic fails to account for the fact that the shrinker (which drops the final huge_zero_refcount pin) can overwrite huge_zero_pfn with the ~0UL sentinel value in shrink_huge_zero_folio_scan() after a racing get_huge_zero_folio() installed a valid value there.

This results in huge_zero_folio being correctly set but huge_zero_pfn being set incorrectly and thus is_huge_zero_pfn() and consequently is_huge_zero_pmd() will misidentify the huge zero folio as being an ordinary THP folio.

This can result in the huge zero folio being split and otherwise treated incorrectly.

The solution to this is very subtle as there is an atomic fast path, and thus ordering in weakly ordered architectures has to be treated very carefully.

The first commit fixes the issue by introducing a spinlock around huge_zero_[pfn, folio, refcount] write, with careful consideration paid to load/store ordering in the fast path. It is placed first and kept as small as possible so that it can be backported on its own.

The second commit is a pure cleanup which reworks the CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic to better separate the persistent logic from the dynamically allocated one.

This patch (of 2):

If !CONFIG_PERSISTENT_HUGE_ZERO_FOLIO, the huge_zero_folio is refcounted by huge_zero_refcount and returned by mm_get_huge_zero_folio().

When the caller is done with the huge zero page, its reference count is decremented. Only a shrinker can set the reference count to zero.

A race can unfortunately occur between a shrinker decrementing the reference count to zero and a concurrent page fault.

This is because shrink_huge_zero_folio_scan() might, if very unlucky, be preempted between setting huge_zero_refcount to zero and writing an invalid value.

During this time get_huge_zero_folio() could write to huge_zero_pfn before shrink_huge_zero_folio_scan() resumes.

In this event the huge zero folio will be persistently misidentified causing the THP code path to be entered inappropriately for the huge zero folio:

            CPU 0                                   CPU 1

=======================================|================================= shrink_huge_zero_folio_scan() | atomic_cmpxchg() sets refcount to 0 | xchg() sets huge_zero_folio to NULL | get_huge_zero_folio() | | atomic_inc_not_zero() -> zero preempted for a long time | Allocate new huge zero folio | | Write valid huge_zero_folio v | Write valid huge_zero_pfn Overwrite huge_zero_pfn with ~0UL <--- Invalid overwrite!

This results in is_huge_zero_pfn() and is_huge_zero_pmd() incorrectly returning false for a huge zero page which could result in issues like the huge zero folio being incorrectly split.

Note that the issue is with huge_zero_pfn not huge_zero_folio, as get_huge_zero_folio() uses cmpxchg() gated on huge_zero_folio being NULL with a retry loop and shrink_huge_zero_folio_scan() uses xchg() to set huge_zero_folio.

Fix the issue by introducing a spinlock, huge_zero_lock, to prevent concurrent write of huge_zero_folio, huge_zero_pfn and huge_zero_refcount.

There needs to be significant care taken here to ensure correctness:

The fast path in get_huge_zero_folio() uses atomic_inc_not_zero(), which is outside of the critical section, and means huge zero allocation is gated on zero huge_zero_refcount.

The fast path doesn't use huge_zero_lock, so the critical section is irrelevant to it.

So invariants are required - huge_zero_refcount MUST:

• Only be set in the huge_zero_lock critical section to ensure serialisation of huge_zero_pfn, huge_zero_folio and ---truncated---

Affected

• Linux >= 6527d8ef68c3ca3c455e38ae2a37cd7810caec73 < 9c0fd1802ce06d7709f0bae4edeb085288f28764
• Linux >= 3b77e8c8cde581dadab9a0f1543a347e24315f11 < b7041ba61c5da4e0b56f9be58cfb87d7689724e4
• Linux >= 3b77e8c8cde581dadab9a0f1543a347e24315f11 < 9332b080ad57650d1dc582e54517f9fc78ef89cc
• Linux >= 3b77e8c8cde581dadab9a0f1543a347e24315f11 < f3a874a903053c53fb53ba287ea9eacda69c68e8
• Linux >= 3b77e8c8cde581dadab9a0f1543a347e24315f11 < 6024f6d0d9b9ca5138bfc4ac6f6e4bdf616e42b3
• Linux >= 3b77e8c8cde581dadab9a0f1543a347e24315f11 < 105d04edbec83010df5728f74d17fd9c108e7553
• Linux >= 3b77e8c8cde581dadab9a0f1543a347e24315f11 < ab7e4b407c7f58d1a003134eff3841f303d5ccc2
• Linux >= 3b77e8c8cde581dadab9a0f1543a347e24315f11 < 33192a26cddea7a7e4ca66e5c3eebd36fa8be2bb
• Linux fc1fbc5b017b6f5ef24a4a93f33cd022225e01c8
• Linux bd092a0f19423d7e9e81182314a96ecd6a14f3b7
• Linux b1daf8f862136894a4595770a44e4508808fb806
• Linux >= 5.10.47 < 5.10.267
• Linux >= 4.19.197 < 4.20
• Linux >= 5.4.129 < 5.5
• Linux >= 5.12.14 < 5.13
• Linux 5.13

Remediation

Refer to the linked advisories for vendor-supplied fixes and affected version ranges.