Why does `pip` throw `ModuleNotFoundError` even though `setuptools` appears to be installed?

Question

On an EC2 instance, I am attempting to install a Python package using pip, but receive an exception that states ModuleNotFoundError: No module named 'setuptools'.

For context, the command I run is

pip install --user mlflow

I attempt to resolve the issue my installing setuptools with the following command: sudo yum install python36-setuptools. Yet, the message returned indicates that setuptools is already installed:

Package python36-setuptools-36.2.7-1.33.amzn1.noarch already installed and latest version
Nothing to do

Please advise.

UPDATE: After switching the EC2 instance from Amazon Linux AMI 2018.03.0 to Amazon Linux 2 AMI I was able to successfully install mlflow using pip.

The question still remains - why was I able to successfully install using Amazon Linux 2 AMI but not Amazon Linux AMI 2018.03.0?

@Panki It appears that the `pip` I'm using is for `python3.x`. The command `pip --version` has output `pip 20.1.1 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)`. — Jubbles, Jul 28 '20 at 02:26
Does `python -m pip install...` in place of simple `pip install...` give different results? (Use whatever is appropriate to you for 'python', maybe 'python3') — Rusi, Jul 29 '20 at 04:03
Your `pip` is from `/usr/local/lib`, but `yum` is definitely going to install to `/usr/lib`. Did you install a separate python, or otherwise do something with `sudo pip`? — muru, Jul 29 '20 at 04:04
@Rusi - Yes, on the original EC instance (`Amazon Linux AMI 2018.03.0`), using `python -m pip install --user mlflow` works, but it installs `mlflow` for `python2.x`. When I try to install for `python3.x` - `python3 -m pip install --user mlflow` - the installation fails and throws exception `ModuleNotFoundError: No module named 'setuptools'`. — Jubbles, Jul 29 '20 at 04:16
@muru - I installed Python3 via `sudo yum install -y python36`. To install `pip`, I ran the following two commands: (1) `curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py` and (2) `sudo python3 get-pip.py`. — Jubbles, Jul 29 '20 at 04:20
I guess your problem starts with the (2) sudo. How/where thats to be cleaned up, I'm not sure. Since 3.5 pip [comes with python](https://stackoverflow.com/a/50410256/3700414) — Rusi, Jul 29 '20 at 04:52

score 1 · Answer 1 · answered Aug 04 '20 at 08:47

Without access to the machine, it will be very hard to verify this, because you may have done something in the past that caused this issue.
The most probable explanation is the following:

It did not work with the old AMI because pip points tho python2's pip but the setuptools installed is for python 3.6. You would probably have needed to install python-setuptools-36.2.7.
It probably works on the other AMI because the correct setuptools is already installed.

This would also answer the second part of the question, because python 2 and python 3 are incompatible in a way that results in python-2-libraries not being usable with python 3.

Another possibility is that for some reason there was a permission problem that caused pip running as a non-root user not being able to access setuptools.
Or your PYTHONPATH was in some way faulty so that the package could not be found.

See this regarding the default python version and support for python 2 in your AMI.

score 0 · Answer 2 · answered Oct 14 '20 at 21:23

I just ran into this in the very same EC2 AMI when trying to install a package from source. Like you, I had also updated my pip from the default that comes with the python36 package, although only the per-user module, not with sudo.

I fixed this by simply reverting to the default version.

python3 -m pip install pip==9.0.3

I was installing alembic==1.4.2 (which is a pinned dependency for airflow 1.10.12) when I got this inexplicable setuptools error - even though I made sure to explicitly target the python36 pip and not the system pip.

The only reason I had even updated pip was because of this pip bug which was blocking the IPython package. That requires python 3.7 starting with v7.17+ - another pinned airflow dependency.

[...snip...]
Successfully installed setuptools-50.3.0 wheel-0.35.1
  Removed build tracker: '/tmp/pip-req-tracker-c93y9xdk'
  Installing build dependencies ... done
  Running command /usr/bin/python3 /home/airflow/.local/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpqnn43_v0
  Getting requirements to build wheel ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
    status = self.run(options, args)
  File "/home/airflow/.local/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/home/airflow/.local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 324, in run
    reqs, check_supported_wheels=not options.target_dir
  File "/home/airflow/.local/lib/python3.6/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/home/airflow/.local/lib/python3.6/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/airflow/.local/lib/python3.6/site-packages/pip/_internal/resolution/legacy/resolver.py", line 340, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/home/airflow/.local/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 483, in prepare_linked_requirement
    req, self.req_tracker, self.finder, self.build_isolation,
  File "/home/airflow/.local/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 91, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/home/airflow/.local/lib/python3.6/site-packages/pip/_internal/distributions/sdist.py", line 38, in prepare_distribution_metadata
    self._setup_isolation(finder)
  File "/home/airflow/.local/lib/python3.6/site-packages/pip/_internal/distributions/sdist.py", line 96, in _setup_isolation
    reqs = backend.get_requires_for_build_wheel()
  File "/home/airflow/.local/lib/python3.6/site-packages/pip/_vendor/pep517/wrappers.py", line 161, in get_requires_for_build_wheel
    'config_settings': config_settings
  File "/home/airflow/.local/lib/python3.6/site-packages/pip/_vendor/pep517/wrappers.py", line 265, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 86, in _build_backend
    obj = import_module(mod_path)
  File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'setuptools'

Why does `pip` throw `ModuleNotFoundError` even though `setuptools` appears to be installed?

2 Answers2