test_itertools: non-ASCII temp dir name corrupted under Windows mbcs codepage and not removed, causing FileNotFoundError in subsequent tests

[Aasyaco]

Windows CI failure: test_itertools corrupts Unicode temp directory name under mbcs encoding, breaks subsequent tests

Summary

On Windows (official Azure CI), the CPython 3.15 test suite produces cascading failures originating from test_itertools. The test creates a temporary directory whose name contains a non-ASCII character. Because the Windows default filesystem encoding is mbcs, the Unicode byte sequence is misinterpreted and the directory name becomes corrupted on disk. The directory is never cleaned up, which causes all subsequent tests that create temporary directories to fail with FileNotFoundError.

---

CPython Version

Python 3.15.0a7 (main branch)

Operating System / Platform

Windows (Azure Pipelines Microsoft-hosted runner)

Filesystem Encoding

>>> import sys
>>> sys.getfilesystemencoding()
'mbcs'

---

CI Pipeline Context

Build step:

PCbuild\build.bat -e $(buildOpt)

Test step:

PCbuild\rt.bat -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0 \
  --junit-xml="$(Build.BinariesDirectory)\test-results.xml" \
  --tempdir="$(Build.BinariesDirectory)\test"

Azure DevOps build log:
https://dev.azure.com/Python/cpython/_build/results?buildId=168864&view=logs&j=c8a71634-e5ec-54a0-3958-760f4148b765&t=ddcdae4e-111a-5c2a-2289-6b784c553924

---

Steps to Reproduce

1. Check out the CPython main branch.
2. Build on Windows:

   PCbuild\build.bat -e <buildOpt>
   

3. Run the test suite:

   PCbuild\rt.bat -q -uall -u-cpu -rwW --slowest --timeout=1200 ^
       -j0 --tempdir="%TEMP%"
   

4. Observe a directory with a corrupted name (e.g. packageæ/) left behind in the temp directory.
5. Observe that subsequent tests fail with FileNotFoundError.

---

Expected Behavior

• All tests pass cleanly on Windows during CI builds.
• No test leaves behind artifacts that interfere with other tests.
• Test isolation is preserved — temporary directories are created and removed correctly.
• Unicode filenames are handled consistently regardless of platform filesystem encoding.

---

Actual Behavior

• test_itertools creates a directory whose name contains a non-ASCII character (e.g. æ).
• Under the Windows default codepage (mbcs), the Unicode name is misinterpreted: æ (U+00E6) is stored as æ.
• The corrupted directory is not removed after test_itertools completes.
• Subsequent tests (e.g. test_class, test_pdb) fail when attempting to create temporary directories because the corrupted entry causes os_helper.temp_dir to raise FileNotFoundError.
• The runner reports a warning for test_itertools and marks unrelated tests as failed, significantly increasing triage cost.

---

Relevant Log Excerpt

From Azure build 168864:

Warning -- files was modified by test_itertools
After: ... 'packageæ/'

test_class failed (env changed)
FileNotFoundError: [WinError 3] unable to create temporary directory:
'D:\a\1\s\build\test_python_9140æ'

---

Root Cause (Analysis)

Two independent problems combine to produce this failure:

1. Encoding mismatch in test_itertools

The test constructs or receives a directory name containing a non-ASCII character. On Windows, sys.getfilesystemencoding() returns 'mbcs' rather than 'utf-8'. The UTF-8 byte sequence for æ (0xC3 0xA6) is reinterpreted under the ANSI code page as two separate characters, producing the visible corruption æ. This is a violation of cross-platform Unicode path handling requirements.

2. Missing cleanup in test_itertools

The test creates the directory but does not register it for cleanup (e.g. via addCleanup, self.addCleanup, or a context manager such as tempfile.TemporaryDirectory). The corrupted directory persists in the shared --tempdir location for the duration of the entire test run, causing every subsequent call to os_helper.temp_dir to fail with WinError 3.

---

Impact

Area	Description	Severity
CI pipeline stability	All Windows builds fail after test_itertools runs	High
Test isolation	Subsequent tests inherit corrupted filesystem state	High
Unicode correctness	Non-ASCII filenames silently corrupted on Windows	Medium
Triage cost	Unrelated failures obscure the root cause	Medium
Local Windows dev	Developers running the full suite locally may be affected	Low

---

Files Involved

• Lib/test/test_itertools.py — primary location of the encoding issue and missing cleanup
• Lib/test/support/os_helper.py — secondary impact point (temp_dir context manager)

---

Linked Resources

• Azure DevOps build 168864: https://dev.azure.com/Python/cpython/_build/results?buildId=168864
• PEP 529 — Change Windows filesystem encoding to UTF-8: https://peps.python.org/pep-0529/
• sys.getfilesystemencoding() docs: https://docs.python.org/3/library/sys.html#sys.getfilesystemencoding
• os.fsencode() docs: https://docs.python.org/3/library/os.html#os.fsencode

[vstinner]

Azure DevOps build log:
https://dev.azure.com/Python/cpython/_build/results?buildId=168864&view=logs&j=c8a71634-e5ec-54a0-3958-760f4148b765&t=ddcdae4e-111a-5c2a-2289-6b784c553924

Extract of logs:

PCbuild\rt.bat -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0 --junit-xml="D:\a\1\b\test-results.xml" --tempdir="D:\a\1\b\test"

D:\a\1\s>"D:\a\1\s\PCbuild\amd64\python.exe"  -m test --fast-ci -uall -u-cpu -rwW --slowest --timeout 1200 -j0 --junit-xml D:\a\1\b\test-results.xml --tempdir D:\a\1\b\test 
+ 'D:\a\1\s\PCbuild\amd64\python.exe' -u -W error -bb -E -m test --fast-ci -uall -u-cpu -rwW --slowest --timeout 1200 -j0 --junit-xml 'D:\a\1\b\test-results.xml' --tempdir 'D:\a\1\b\test' --dont-add-python-opts
== CPython 3.15.0a7+ (remotes/origin/52c01864c4778a351e5aa3584e86ed6fd212a5a4:52c01864c, Mar 19 2026, ) [MSC v.1944 64 bit (AMD64)]
== Windows-2022Server-10.0.20348-SP0 little-endian
== Python build: release with_assert
== cwd: D:\a\1\b\test\test_python_worker_2896æ
== CPU count: 2
== encodings: locale=cp1252 FS=utf-8
== resources: all,-cpu

Using random seed: 1027629067
0:00:00 Run 501 tests in parallel using 4 worker processes (timeout: 20 min, worker timeout: 25 min)
0:00:00 [  1/501] test.test_io.test_file passed
0:00:00 [  2/501] test_select passed
0:00:01 [  3/501] test_type_comments passed
0:00:02 [  4/501/1] test_range failed (env changed)
tests may fail, unable to create temporary directory 'D:\\a\\1\\s\\build\\test_python_4940æ': [WinError 3] The system cannot find the path specified: 'D:\\a\\1\\s\\build\\test_python_4940æ'
Traceback (most recent call last):
  File "D:\a\1\s\Lib\test\support\os_helper.py", line 524, in temp_dir
    os.mkdir(path)
    ~~~~~~~~^^^^^^
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'D:\\a\\1\\s\\build\\test_python_4940æ'
Stack (most recent call last):
  File "<frozen runpy>", line 196, in _run_module_as_main
  File "<frozen runpy>", line 87, in _run_code
  File "D:\a\1\s\Lib\test\libregrtest\worker.py", line 138, in <module>
    main()
  File "D:\a\1\s\Lib\test\libregrtest\worker.py", line 133, in main
    with os_helper.temp_cwd(work_dir, quiet=True):
  File "D:\a\1\s\Lib\contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "D:\a\1\s\Lib\test\support\os_helper.py", line 597, in temp_cwd
    with temp_dir(path=name, quiet=quiet) as temp_path:
tests may fail, unable to change the current working directory to 'D:\\a\\1\\s\\build\\test_python_4940æ': [WinError 3] The system cannot find the path specified: 'D:\\a\\1\\s\\build\\test_python_4940æ'
Traceback (most recent call last):
  File "D:\a\1\s\Lib\test\support\os_helper.py", line 563, in change_cwd
    os.chdir(os.path.realpath(path))
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'D:\\a\\1\\s\\build\\test_python_4940æ'
Stack (most recent call last):
  File "<frozen runpy>", line 196, in _run_module_as_main
  File "<frozen runpy>", line 87, in _run_code
  File "D:\a\1\s\Lib\test\libregrtest\worker.py", line 138, in <module>
    main()
  File "D:\a\1\s\Lib\test\libregrtest\worker.py", line 133, in main
    with os_helper.temp_cwd(work_dir, quiet=True):
  File "D:\a\1\s\Lib\contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "D:\a\1\s\Lib\test\support\os_helper.py", line 598, in temp_cwd
    with change_cwd(temp_path, quiet=quiet) as cwd_dir:
test_attributes (test.test_range.RangeTest.test_attributes) ... ok
test_comparison (test.test_range.RangeTest.test_comparison) ... ok
test_contains (test.test_range.RangeTest.test_contains) ... ok
test_count (test.test_range.RangeTest.test_count) ... ok
test_empty (test.test_range.RangeTest.test_empty) ... ok
test_exhausted_iterator_pickling (test.test_range.RangeTest.test_exhausted_iterator_pickling) ... ok
test_index (test.test_range.RangeTest.test_index) ... ok
test_invalid_invocation (test.test_range.RangeTest.test_invalid_invocation) ... ok
test_issue11845 (test.test_range.RangeTest.test_issue11845) ... ok
test_iterator_invalid_setstate (test.test_range.RangeTest.test_iterator_invalid_setstate) ... ok
test_iterator_pickling (test.test_range.RangeTest.test_iterator_pickling) ... ok
test_iterator_pickling_overflowing_index (test.test_range.RangeTest.test_iterator_pickling_overflowing_index) ... ok
test_iterator_setstate (test.test_range.RangeTest.test_iterator_setstate) ... ok
test_iterator_unpickle_compat (test.test_range.RangeTest.test_iterator_unpickle_compat) ... ok
test_large_exhausted_iterator_pickling (test.test_range.RangeTest.test_large_exhausted_iterator_pickling) ... ok
test_large_operands (test.test_range.RangeTest.test_large_operands) ... ok
test_large_range (test.test_range.RangeTest.test_large_range) ... ok
test_odd_bug (test.test_range.RangeTest.test_odd_bug) ... ok
test_pickling (test.test_range.RangeTest.test_pickling) ... ok
test_range (test.test_range.RangeTest.test_range) ... ok
test_range_constructor_error_messages (test.test_range.RangeTest.test_range_constructor_error_messages) ... ok
test_range_iterators (test.test_range.RangeTest.test_range_iterators) ... ok
test_range_iterators_invocation (test.test_range.RangeTest.test_range_iterators_invocation) ... ok
test_repr (test.test_range.RangeTest.test_repr) ... ok
test_reverse_iteration (test.test_range.RangeTest.test_reverse_iteration) ... ok
test_slice (test.test_range.RangeTest.test_slice) ... ok
test_strided_limits (test.test_range.RangeTest.test_strided_limits) ... ok
test_types (test.test_range.RangeTest.test_types) ... ok
test_user_index_method (test.test_range.RangeTest.test_user_index_method) ... ok

----------------------------------------------------------------------
Ran 29 tests in 1.076s

OK
Warning -- files was modified by test_range
Warning --   Before: ['.azure-pipelines/', '.coveragerc', '.devcontainer/', '.editorconfig', '.git/', '.gitattributes', '.github/', '.gitignore', '.mailmap', '.pre-commit-config.yaml', '.readthedocs.yml', '.ruff.toml', '.well-known/', 'Android/', 'Apple/', 'Doc/', 'Grammar/', 'Include/', 'InternalDocs/', 'LICENSE', 'Lib/', 'Mac/', 'Makefile.pre.in', 'Misc/', 'Modules/', 'Objects/', 'PC/', 'PCbuild/', 'Parser/', 'Platforms/', 'Programs/', 'Python/', 'README.rst', 'Tools/', 'aclocal.m4', 'config.guess', 'config.sub', 'configure', 'configure.ac', 'iOS/', 'install-sh', 'pyconfig.h.in', 'python.bat']
Warning --   After:  ['.azure-pipelines/', '.coveragerc', '.devcontainer/', '.editorconfig', '.git/', '.gitattributes', '.github/', '.gitignore', '.mailmap', '.pre-commit-config.yaml', '.readthedocs.yml', '.ruff.toml', '.well-known/', 'Android/', 'Apple/', 'Doc/', 'Grammar/', 'Include/', 'InternalDocs/', 'LICENSE', 'Lib/', 'Mac/', 'Makefile.pre.in', 'Misc/', 'Modules/', 'Objects/', 'PC/', 'PCbuild/', 'Parser/', 'Platforms/', 'Programs/', 'Python/', 'README.rst', 'Tools/', 'aclocal.m4', 'config.guess', 'config.sub', 'configure', 'configure.ac', 'iOS/', 'install-sh', 'junk95142.zip', 'pyconfig.h.in', 'python.bat', 'ziptestmodule']

[vstinner]

On Windows, sys.getfilesystemencoding() returns 'mbcs' rather than 'utf-8'.

That's wrong: sys.getfilesystemencoding() returns utf-8 on Windows. You can see FS=utf-8 in logs above (line == encodings: locale=cp1252 FS=utf-8).

You can also see sys.filesystem_encoding: utf-8/surrogatepass in the previous build step, in "Display build info".

Steps to Reproduce:
(...)
PCbuild\rt.bat -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0 --tempdir="%TEMP%"

I built Python in debug mode and I ran:

PCbuild\rt.bat -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j1 test_range --tempdir=%TEMP%

I'm unable to reproduce the error.

[Aasyaco]

I am sorry. The real error message is

tests may fail, unable to create temporary directory 'D:\\a\\1\\s\\build\\test_python_9140æ': [WinError 3] The system cannot find the path specified: 'D:\\a\\1\\s\\build\\test_python_9140æ'
Traceback (most recent call last):
  File "D:\a\1\s\Lib\test\support\os_helper.py", line 524, in temp_dir
    os.mkdir(path)
    ~~~~~~~~^^^^^^
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'D:\\a\\1\\s\\build\\test_python_9140æ'
Stack (most recent call last):
  File "<frozen runpy>", line 196, in _run_module_as_main
  File "<frozen runpy>", line 87, in _run_code
  File "D:\a\1\s\Lib\test\libregrtest\worker.py", line 138, in <module>
    main()
  File "D:\a\1\s\Lib\test\libregrtest\worker.py", line 133, in main
    with os_helper.temp_cwd(work_dir, quiet=True):
  File "D:\a\1\s\Lib\contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "D:\a\1\s\Lib\test\support\os_helper.py", line 597, in temp_cwd
    with temp_dir(path=name, quiet=quiet) as temp_path:
tests may fail, unable to change the current working directory to 'D:\\a\\1\\s\\build\\test_python_9140æ': [WinError 3] The system cannot find the path specified: 'D:\\a\\1\\s\\build\\test_python_9140æ'
Traceback (most recent call last):
  File "D:\a\1\s\Lib\test\support\os_helper.py", line 563, in change_cwd
    os.chdir(os.path.realpath(path))
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'D:\\a\\1\\s\\build\\test_python_9140æ'
Stack (most recent call last):
  File "<frozen runpy>", line 196, in _run_module_as_main
  File "<frozen runpy>", line 87, in _run_code
  File "D:\a\1\s\Lib\test\libregrtest\worker.py", line 138, in <module>
    main()
  File "D:\a\1\s\Lib\test\libregrtest\worker.py", line 133, in main
    with os_helper.temp_cwd(work_dir, quiet=True):
  File "D:\a\1\s\Lib\contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "D:\a\1\s\Lib\test\support\os_helper.py", line 598, in temp_cwd
    with change_cwd(temp_path, quiet=quiet) as cwd_dir:

This is only a example.

[serhiy-storchaka]

It seems that some of functions still use non-Unicode Windows API.

We use non-ASCII temporary directory name specially to catch such bugs.

[vstinner]

It seems that some of functions still use non-Unicode Windows API.

At least, os.chdir() is implemented with SetCurrentDirectoryW() which takes a wide string (LPCWSTR path).

This is only a example.

@Aasyaco: Can you reproduce the error outside the Azure Pipeline?

[Aasyaco]

Thank you for the patience and correction.

The traceback shows the corrupted path test_python_9140æ is
already invalid before reaching os_helper.temp_cwd() — the
corruption happens upstream in libregrtest/worker.py line 133,
where work_dir is constructed. The os_helper code is just the crash site, not the source.

I also want to correct my earlier analysis: the root cause is not
test_itertools or mbcs filesystem encoding. I apologise for the inaccurate attribution.