Update the python tests to also check QAPI and the QAPI Sphinx
extensions. The docs/sphinx/qapidoc_legacy.py file is not included in
these checks, as it is destined for removal soon. mypy is also not
called on the QAPI Sphinx extensions, owing to difficulties supporting
Sphinx 3.x - 8.x while maintaining static type checking support. mypy
*is* called on all of the QAPI tools themselves, though.
flake8, isort and mypy use the tool configuration from the existing
python directory (in setup.cfg). pylint continues to use the special
configuration located in scripts/qapi/ - that configuration is more
permissive. If we wish to unify the two configurations, that's a
separate series and a discussion for a later date.
The list of pylint ignores is also updated, owing again to the wide
window of pylint version support: newer versions require pragmas to
occasionally silence the "too many positional arguments" warning, but
older versions do not have such a warning category and will instead yelp
about an unrecognized option. Silence that warning, too.
As a result of this patch, one would be able to run any of the following
tests locally from the qemu.git/python directory and have it cover the
QAPI tooling as well. All of the following options run the python tests,
static analysis tests, and linter checks; but with different
combinations of dependencies and interpreters.
- "make check-minreqs" Run tests specifically under our oldest supported
Python and our oldest supported dependencies. This is the test that
runs on GitLab as "check-python-minreqs". This helps ensure we do not
regress support on older platforms accidentally.
- "make check-tox" Runs the tests under the newest supported
dependencies, but under each supported version of Python in turn. At
time of writing, this is Python 3.8 to 3.13 inclusive. This test helps
catch bleeding-edge problems before they become problems for developer
workstations. This is the GitLab test "check-python-tox" and is an
optionally run, may-fail test due to the unpredictable nature of new
dependencies being released into the ecosystem that may cause
regressions.
- "make check-dev" Runs the tests under the newest supported
dependencies using whatever version of Python the user happens to have
installed. This is a quick convenience check that does not map to any
particular GitLab test.
(Note! check-dev may be busted on Fedora 41 and bleeding edge versions
of setuptools. That's unrelated to this patch and I'll address it
separately and soon. Thank you for your patience, --mgmt)
Finally, finally, finally: this means that QAPI tooling will be linted
and type-checked from the GitLab pipelines.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-id: 20250604200354.459501-5-jsnow@redhat.com
[Edited license choice per review --js]
Signed-off-by: John Snow <jsnow@redhat.com>
We pin all dependencies for the "check-minreqs" test because pip lacks a
dependency resolver that installs "the oldest possible package that
meets dependency criteria". So, in order to test our stated minimum
requirements, we pin all of our dependencies (and their dependencies,
transitively) at the oldest possible versions that still work and pass
tests; proving that our minimum requirements are correct.
(It also ensures no new features accidentally sneak in from developers
on newer platforms.)
A few transitive dependencies were omitted from the pinned dependency
file by accident; as a result, pip's dependency solver can pull in newer
dependencies, which we don't want. This patch corrects the previous
oversight and pins the missing dependencies.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-id: 20250604200354.459501-4-jsnow@redhat.com
Python 3.8 went "end of life" in October 2024 and Fedora 42 dropped
this version already, so the "python" CI job is currently failing.
Thus it's time to drop support for this Python version in QEMU, too.
While we're at it, also look for "python3.13" in the configure script.
Message-ID: <20250425120710.879518-1-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Debian 10 is not anymore a supported distro, since Debian 12 was
released on June 10, 2023. Our supported build platforms as of today
all support at least 3.8 (and all of them except for Ubuntu 20.04
support 3.9):
openSUSE Leap 15.5: 3.6.15 (3.11.2)
CentOS Stream 8: 3.6.8 (3.8.13, 3.9.16, 3.11.4)
CentOS Stream 9: 3.9.17 (3.11.4)
Fedora 37: 3.11.4
Fedora 38: 3.11.4
Debian 11: 3.9.2
Debian 12: 3.11.2
Alpine 3.14, 3.15: 3.9.16
Alpine 3.16, 3.17: 3.10.10
Ubuntu 20.04 LTS: 3.8.10
Ubuntu 22.04 LTS: 3.10.12
NetBSD 9.3: 3.9.13*
FreeBSD 12.4: 3.9.16
FreeBSD 13.1: 3.9.18
OpenBSD 7.2: 3.9.17
Note: NetBSD does not appear to have a default meta-package, but offers
several options, the lowest of which is 3.7.15. However, "python39"
appears to be a pre-requisite to one of the other packages we request
in tests/vm/netbsd.
Since it is safe under our supported platform policy, bump our
minimum supported version of Python to 3.8. The two most interesting
features to have by default include:
- the importlib.metadata module, whose lack is responsible for over 100
lines of code in mkvenv.py
- improvements to asyncio, for example asyncio.CancelledError
inherits from BaseException rather than Exception
In addition, code can now use the assignment operator ':='
Because mypy now learns about importlib.metadata, a small change to
mkvenv.py is needed to pass type checking.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
There are many Python 3.12 issues right now, but a particularly
problematic one when debugging them is that one cannot even use
minreqs.txt in a Python 3.12 virtual environment to test with
locked package versions.
Bump the mypy and wrapt versions to fix this, while remaining
within the realm of versions compatible with Python 3.7.
This requires a workaround for a mypy false positive
qemu/qmp/qmp_tui.py:350: error: Non-overlapping equality check (left operand type: "Literal[Runstate.DISCONNECTING]", right operand type: "Literal[Runstate.IDLE]") [comparison-overlap]
where mypy does not realize that self.disconnect() could change
the value of self.runstate.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
The version of pyflakes that is listed in python/tests/minreqs.txt
breaks on Python 3.8 with the following message:
AttributeError: 'FlakesChecker' object has no attribute 'CONSTANT'
Now that we do not support EOL'd Python versions anymore, we can
update to newer, fixed versions. It is a good time to do so, before
Python packages start dropping support for Python 3.7 as well!
The new mypy is also a bit smarter about which packages are actually
being used, so remove the now-unnecessary sections from setup.cfg.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-27-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Python 3.6 was EOL 2021-12-31. Newer versions of upstream libraries have
begun dropping support for this version and it is becoming more
cumbersome to support. Avocado-framework and qemu.qmp each have their
own reasons for wanting to drop Python 3.6, but won't until QEMU does.
Versions of Python available in our supported build platforms as of today,
with optional versions available in parentheses:
openSUSE Leap 15.4: 3.6.15 (3.9.10, 3.10.2)
CentOS Stream 8: 3.6.8 (3.8.13, 3.9.16)
CentOS Stream 9: 3.9.13
Fedora 36: 3.10
Fedora 37: 3.11
Debian 11: 3.9.2
Alpine 3.14, 3.15: 3.9.16
Alpine 3.16, 3.17: 3.10.10
Ubuntu 20.04 LTS: 3.8.10
Ubuntu 22.04 LTS: 3.10.4
NetBSD 9.3: 3.9.13*
FreeBSD 12.4: 3.9.16
FreeBSD 13.1: 3.9.16
OpenBSD 7.2: 3.9.16
Note: Our VM tests install 3.9 explicitly for FreeBSD and 3.10 for
NetBSD; the default for "python" or "python3" in FreeBSD is
3.9.16. NetBSD does not appear to have a default meta-package, but
offers several options, the lowest of which is 3.7.15. "python39"
appears to be a pre-requisite to one of the other packages we request in
tests/vm/netbsd. pip, ensurepip and other Python essentials are
currently only available for Python 3.10 for NetBSD.
CentOS and OpenSUSE support parallel installation of multiple Python
interpreters, and binaries in /usr/bin will always use Python 3.6. However,
the newly introduced support for virtual environments ensures that all build
steps that execute QEMU Python code use a single interpreter.
Since it is safe to under our supported platform policy, bump our
minimum supported version of Python to 3.7.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20230511035435.734312-24-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
This command is to be used to add various packages (or ensure they're
already present) into the configure-provided venv in a modular fashion.
Examples:
mkvenv ensure --online --dir "${source_dir}/python/wheels/" "meson>=0.61.5"
mkvenv ensure --online "sphinx>=1.6.0"
mkvenv ensure "qemu.qmp==0.0.2"
It's designed to look for packages in three places, in order:
(1) In system packages, if the version installed is already good
enough. This way your distribution-provided meson, sphinx, etc are
always used as first preference.
(2) In a vendored packages directory. Here I am suggesting
qemu.git/python/wheels/ as that directory. This is intended to serve as
a replacement for vendoring the meson source for QEMU tarballs. It is
also highly likely to be extremely useful for packaging the "qemu.qmp"
package in source distributions for platforms that do not yet package
qemu.qmp separately.
(3) Online, via PyPI, ***only when "--online" is passed***. This is only
ever used as a fallback if the first two sources do not have an
appropriate package that meets the requirement. The ability to build
QEMU and run tests *completely offline* is not impinged.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-7-jsnow@redhat.com>
[Use distlib to lookup distributions. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
The pipenv tool was nice in theory, but in practice it's just too hard
to update selectively, and it makes using it a pain. The qemu.qmp repo
dropped pipenv support a while back and it's been functioning just fine,
so I'm backporting that change here to qemu.git.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20230210003147.1309376-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
