11/30/2021

Python Pip

Find out how to use pip on a Raspberry Pi to easily install useful Python software. What you will learn. By following this guide, you will learn: What is pip? How to use pip to install Python packages on Raspbian; What you will need. Spotted a mistake? Enjoying the project? Any opinions on the website? Installing packages using pip and virtual environments¶. This guide discusses how to install packages using pip and a virtual environment manager: either venv for Python 3 or virtualenv for Python 2. These are the lowest-level tools for managing Python packages and are recommended if higher-level tools do not suit your needs. In Python, pip is a standard package management system that is used to install and manage other software modules. In general, pip is a package manager for many Python packages or modules to install it to use in the code when required.

This tutorial walks you through how to package a simple Python project. It willshow you how to add the necessary files and structure to create the package, howto build the package, and how to upload it to the Python Package Index.

A simple project¶

This tutorial uses a simple project named example_pkg. If you are unfamiliarwith Python’s modules and import packages, take a fewminutes to read over the Python documentation for packages and modules. Evenif you already have a project that you want to package up, we recommendfollowing this tutorial as-is using this example package and then trying withyour own package.

To create this project locally, create the following file structure:

Once you create this structure, you’ll want to run all of the commands in thistutorial within the top-level folder - so be sure to cdpackaging_tutorial.

src/example_pkg/__init__.py is required to import the directory as a package,and can simply be an empty file.

Creating the package files¶

You will now create a handful of files to package up this project and prepare itfor distribution. Create the new files listed below and place them in theproject’s root directory - you will add content to them in the following steps.

Creating a test folder¶

tests/ is a placeholder for unit test files. Leave it empty for now.

Creating pyproject.toml¶

pyproject.toml is the file that tells build tools (like pip 10+ andbuild) what system you are using and what is required for building. Thedefault if this file is missing is to assume a classic setuptools build system,but it is better to be explicit; if you have a pyproject.toml file, youwill be able to rely on wheel and other packages being present.

This file should be ideal for most setuptools projects:

build-system.requires gives a list of packages that are needed to build yourpackage. Listing something here will only make it available during the build,not after it is installed.

build-system.build-backend is technically optional, but you will getsetuptools.build_meta:__legacy__ instead if you forget to include it, soalways include it. If you were to use a different build system, such asflit or poetry, those would go here, and the configuration detailswould be completely different than the setuptools configuration describedbelow. See PEP 517 and PEP 518 for background and details.

Configuring metadata¶

There are two types of metadata: static and dynamic.

  • Static metadata (setup.cfg): guaranteed to be the same every time. This issimpler, easier to read, and avoids many common errors, like encoding errors.

  • Dynamic metadata (setup.py): possibly non-deterministic. Any items that aredynamic or determined at install-time, as well as extension modules orextensions to setuptools, need to go into setup.py.

Static metadata should be preferred and dynamic metadata should be used only asan escape hatch when absolutely necessary.

setup.cfg is the configuration file for setuptools. It tellssetuptools about your package (such as the name and version) as well as whichcode files to include. Eventually much of this configuration may be able to moveto pyproject.toml.

Open setup.cfg and enter the following content. Update the package nameto include your username (for example, example-pkg-theacodes), this ensuresthat you have a unique package name and that your package doesn’t conflict withpackages uploaded by other people following this tutorial.

There are a variety of metadata and optionssupported here. This is in configparser format; do not place quotes around values.This example package uses a relatively minimal set of metadata:

  • name is the distribution name of your package. This can be any name aslong as only contains letters, numbers, _ , and -. It also must notalready be taken on pypi.org. Be sure to update this with your username,as this ensures you won’t try to upload a package with the same name as onewhich already exists when you upload the package.

  • version is the package version. See PEP 440 for more details onversions. You can use file: or attr: directives to read from a file orpackage attribute (simple attributes do not require import).

  • author and author_email are used to identify the author of thepackage.

  • description is a short, one-sentence summary of the package.

  • long_description is a detailed description of the package. This isshown on the package detail page on the Python Package Index. Inthis case, the long description is loaded from README.md which isa common pattern, using the file: directive.

  • long_description_content_type tells the index what type of markup isused for the long description. In this case, it’s Markdown.

  • url is the URL for the homepage of the project. For many projects, thiswill just be a link to GitHub, GitLab, Bitbucket, or similar code hostingservice.

  • project_urls lets you list any number of extra links to show on PyPI.Generally this could be to documentation, issue trackers, etc.

  • classifiers gives the index and pip some additional metadataabout your package. In this case, the package is only compatible with Python3, is licensed under the MIT license, and is OS-independent. You shouldalways include at least which version(s) of Python your package works on,which license your package is available under, and which operating systemsyour package will work on. For a complete list of classifiers, seehttps://pypi.org/classifiers/.

In the options category, we have controls for setuptools itself:

  • package_dir is a mapping of package names and directories.An empty package name represents the “root package” — the directory inthe project that contains all Python source files for the package — soin this case the src directory is designated the root package.

  • packages is a list of all Python import packages that should be included in the Distribution Package.Instead of listing each package manually, we can use the find: directiveto automatically discover all packages and subpackages andoptions.packages.find to specify the package_dir to use. In thiscase, the list of packages will be example_pkg as that’s the onlypackage present.

  • python_requires gives the versions of Python supported by yourproject. Installers like pip will look back though older versions ofpackages until it finds one that has a matching Python version.

There are many more than the ones mentioned here. SeePackaging and distributing projects for more details.

If you create a setup.py file, this will enable direct interactionwith setup.py (which generally should be avoided), and editableinstalls. This file used to be required, but can be omitted in modernsetuptools.

Warning

If you include the file, you must have a call to setup() in it,even if there are no arguments:

Anything you set in setup.cfg can instead be set via keyword argument tosetup(); this enables computed values to be used. You will also needsetup() for setting up extension modules for compilation. Currently,having this file also is required if you want to use editable installs with pip.

Python Pipe

setup.py is the build script for setuptools. It tells setuptoolsabout your package (such as the name and version) as well as which code filesto include.

Open setup.py and enter the following content. Update the package nameto include your username (for example, example-pkg-theacodes), this ensuresthat you have a unique package name and that your package doesn’t conflict withpackages uploaded by other people following this tutorial.

setup() takes several arguments. This example package uses a relativelyminimal set:

  • name is the distribution name of your package. This can be any name aslong as only contains letters, numbers, _ , and -. It also must notalready be taken on pypi.org. Be sure to update this with your username,as this ensures you won’t try to upload a package with the same name as onewhich already exists when you upload the package.

  • version is the package version see PEP 440 for more details onversions.

  • author and author_email are used to identify the author of thepackage.

  • description is a short, one-sentence summary of the package.

  • long_description is a detailed description of the package. This isshown on the package detail page on the Python Package Index. Inthis case, the long description is loaded from README.md which isa common pattern.

  • long_description_content_type tells the index what type of markup isused for the long description. In this case, it’s Markdown.

  • url is the URL for the homepage of the project. For many projects, thiswill just be a link to GitHub, GitLab, Bitbucket, or similar code hostingservice.

  • project_urls lets you list any number of extra links to show on PyPI.Generally this could be to documentation, issue trackers, etc.

  • classifiers gives the index and pip some additional metadataabout your package. In this case, the package is only compatible with Python3, is licensed under the MIT license, and is OS-independent. You shouldalways include at least which version(s) of Python your package works on,which license your package is available under, and which operating systemsyour package will work on. For a complete list of classifiers, seehttps://pypi.org/classifiers/.

  • package_dir is a dictionary with package names for keys and directoriesfor values. An empty package name represents the “root package” — thedirectory in the project that contains all Python source files for thepackage — so in this case the src directory is designated the rootpackage.

  • packages is a list of all Python import packages that should be included in the Distribution Package.Instead of listing each package manually, we can use find_packages()to automatically discover all packages and subpackages under package_dir.In this case, the list of packages will be example_pkg as that’s theonly package present.

  • python_requires gives the versions of Python supported by yourproject. Installers like pip will look back though older versions ofpackages until it finds one that has a matching Python version.

There are many more than the ones mentioned here. SeePackaging and distributing projects for more details.

Creating README.md¶

Open README.md and enter the following content. You can customize thisif you’d like.

Because our build script loads README.md to provide a long_descriptionfor setup(), the README.md must be included along with your codewhen you generate a source distribution.setuptools 36.4.0 or above will automatically include README.mdif it exists.

Creating a LICENSE¶

It’s important for every package uploaded to the Python Package Index to includea license. This tells users who install your package the terms under which theycan use your package. For help picking a license, seehttps://choosealicense.com/. Once you have chosen a license, openLICENSE and enter the license text. For example, if you had chosen theMIT license:

Including other files¶

The files listed above will be included automatically in yoursource distribution. If you want to control what goes in thisexplicitly, see Including files in source distributions with MANIFEST.in.

The final built distribution willhave the Python files in thediscovered or listed Python packages. If you want to control what goes here,such as to add data files, see Including Data Files from the setuptools docs.

Python Get Pip

Generating distribution archives¶

The next step is to generate distribution packages for the package. These are archives that are uploaded to the PackageIndex and can be installed by pip.

Make sure you have the latest versions of PyPA’s build installed:

Tip

If you have trouble installing these, see theInstalling Packages tutorial.

Now run this command from the same directory where pyproject.toml is located:

This command should output a lot of text and once completed should generate twofiles in the dist directory:

Note

If you run into trouble here, please copy the output and file an issueover on packaging problems and we’ll do our best to help you!

The tar.gz file is a Source Archive whereas the .whl file is aBuilt Distribution. Newer pip versions preferentially installbuilt distributions, but will fall back to source archives if needed. Youshould always upload a source archive and provide built archives for theplatforms your project is compatible with. In this case, our example package iscompatible with Python on any platform so only one built distribution is needed.

Uploading the distribution archives¶

Finally, it’s time to upload your package to the Python Package Index!

The first thing you’ll need to do is register an account on TestPyPI. TestPyPI is a separate instance of the package index intended for testing andexperimentation. It’s great for things like this tutorial where we don’tnecessarily want to upload to the real index. To register an account, go tohttps://test.pypi.org/account/register/ and complete the steps on that page.You will also need to verify your email address before you’re able to uploadany packages. For more details on Test PyPI, seeUsing TestPyPI.

Now you’ll create a PyPI API token so you will be able to securely uploadyour project.

Install Pip

Go to https://test.pypi.org/manage/account/#api-tokens and create a newAPI token; don’t limit its scope to a particular project, since youare creating a new project.

Don’t close the page until you have copied and saved the token — youwon’t see that token again.

Now that you are registered, you can use twine to upload thedistribution packages. You’ll need to install Twine:

Once installed, run Twine to upload all of the archives under dist:

You will be prompted for a username and password. For the username,use __token__. For the password, use the token value, includingthe pypi- prefix.

After the command completes, you should see output similar to this:

Once uploaded your package should be viewable on TestPyPI, for example,https://test.pypi.org/project/example-pkg-YOUR-USERNAME-HERE

Installing your newly uploaded package¶

You can use pip to install your package and verify that it works.Create a new virtualenv (see Installing Packages fordetailed instructions) and install your package from TestPyPI:

Python pip upgrade

Make sure to specify your username in the package name!

pip should install the package from Test PyPI and the output should looksomething like this:

Note

This example uses --index-url flag to specify TestPyPI instead oflive PyPI. Additionally, it specifies --no-deps. Since TestPyPI doesn’thave the same packages as the live PyPI, it’s possible that attempting toinstall dependencies may fail or install something unexpected. While ourexample package doesn’t have any dependencies, it’s a good practice to avoidinstalling dependencies when using TestPyPI.

You can test that it was installed correctly by importing the package.Run the Python interpreter (make sure you’re still in your virtualenv):

and from the interpreter shell import the package:

Note that the Import Package is example_pkg regardless of whatname you gave your Distribution Packagein setup.py (in this case, example-pkg-YOUR-USERNAME-HERE).

Next steps¶

Congratulations, you’ve packaged and distributed a Python project!✨ 🍰 ✨

Python

Keep in mind that this tutorial showed you how to upload your package to TestPyPI, which isn’t a permanent storage. The Test system occasionally deletespackages and accounts. It is best to use Test PyPI for testing and experimentslike this tutorial.

When you are ready to upload a real package to the Python Package Index you cando much the same as you did in this tutorial, but with these importantdifferences:

  • Choose a memorable and unique name for your package. You don’t have to appendyour username as you did in the tutorial.

  • Register an account on https://pypi.org - note that these are two separateservers and the login details from the test server are not shared with themain server.

  • Use twineuploaddist/* to upload your package and enter your credentialsfor the account you registered on the real PyPI. Now that you’re uploadingthe package in production, you don’t need to specify --repository; thepackage will upload to https://pypi.org/ by default.

  • Install your package from the real PyPI using python3-mpipinstall[your-package].

Python Pip

At this point if you want to read more on packaging Python libraries here aresome things you can do:

  • Read more about using setuptools to package libraries inPackaging and distributing projects.

  • Read about Packaging binary extensions.

  • Consider alternatives to setuptools such as flit, hatch,and poetry.