`packages` is empty when using a non-pypi.org simple index (e.g. Sonatype Nexus)

[iamdbychkov]

Currently the metadata fetch requires the pypi.org-specific JSON API. (See also #260 )

When PYINSP_INDEX_URL points at a non-pypi.org simple index (Sonatype Nexus PyPI in my case), python-inspector resolves dependencies but the packages array in the output is empty - package metadata is never fetched.

For example: PYINSP_INDEX_URL=https://my-nexus-pypi.com/repository/pypi/simple:

root@2ae255f06ad8:/project# python-inspector -p3.13 -rrequirements.txt -olinux --json-pdt /tmp/test.json --verbose
...
retrieve package data from pypi:
  retrieved package 'pkg:pypi/cx-oracle@7.3.0'
  retrieved package 'pkg:pypi/redis@2.10.6'
done!

root@2ae255f06ad8:/project# jq '.packages[].name' /tmp/test.json
# (no output - packages is [])

Note the verbose log claims retrieved package ..., yet packages is empty - the retrieval silently failed.

Without the index (default pypi.org) it works:

root@2ae255f06ad8:/project# env -u PYINSP_INDEX_URL python-inspector -p3.13 -rrequirements.txt -olinux --json-pdt /tmp/test.json --verbose
...
root@2ae255f06ad8:/project# jq '.packages[].name' /tmp/test.json
"cx-oracle"
"redis"

Root cause is the get_pypi_data_from_purl builds the metadata URL from the index base and queries the pypi.org-specific JSON API, then returns None on any non-200:

python-inspector/src/python_inspector/package_data.py

Lines 65 to 77 in a841c7c

	base_path = (
	index_url.removesuffix("/simple") + "/pypi" if index_url else "https://pypi.org/pypi"
	)
	api_url = f"{base_path}/{name}/{version}/json"
	from python_inspector.utils import get_response_async
	response = await get_response_async(api_url)
	if not response:
	return None
	info = response.get("info") or {}

With a Nexus index this resolves to https://my-nexus-pypi.com/repository/pypi/pypi/redis/2.10.6/json, which 404s - Nexus's PyPI format implements the simple index (PEP 503), not the /pypi/<pkg>/<ver>/json API - so the function bails out and no PackageData is produced.

I initially assumed --use-pypi-json-api was the toggle for this. It doesn't help here: in its default (off) position get_pypi_data_from_purl still unconditionally queries the JSON API, and in the on position the help states --index-url is ignored (i.e. pypi.org), so it can't reach a private index either way.

I'm not sure what the correct approach would be, I feel like --use-pypi-json-api option should be fixed that way we download the selected sdist/wheel and parse its PKG-INFO for License / Home-page / Author / Requires-Dist. The simple index already provides the file list + digests, so the JSON API's urls are largely redundant; its unique value is the rich info block, which PKG-INFO also carries.

While searching for solution I've also stumbled upon PEP 691 https://packaging.python.org/en/latest/specifications/simple-repository-api/#json-based-simple-api-for-python-package-indexes
Which might be better than parsing of PKG-INFO (pkginfo package exists though, so it would be easy)

Either way, would like to hear the maintainers thoughts on that. If it something that should be fixed - I would try.

[iamdbychkov]

Seem's like there is PR addressing the issue, not sure about current state though
#258

But it doesn't look like it fixes --use-pypi-json-api usage, which I believe is the right approach.