<jupyter_start><jupyter_text>Building Python packagesA carefully crafted directory structure makes the extension to building packages from the code relatively easy. One has to be careful since applying 'build' and 'setuptools' requires giving those tools some TLC; after all this time, they are still a little clunky. This notebook will show an example.  Folder structureSuch a structure is in the 'buildable' directory, following the structure from https://packaging.python.org/en/latest/tutorials/packaging-projects/<jupyter_code>!tree -I __pycache__ ../buildable
!cd ../buildable; python -m pytest<jupyter_output>[1m============================= test session starts ==============================[0m
platform linux -- Python 3.10.4, pytest-7.1.1, pluggy-1.0.0
rootdir: /home/jsnagi/proj/github/nagi49000/tutorial-memory-refs/python/buildable
collected 3 items                                                              [0m

number_returns/number_returns_tests/ints/test_gimmes.py [32m.[0m[32m.[0m[32m               [ 66%][0m
number_returns/number_returns_tests/strs/test_gimme_strs.py [32m.[0m[32m            [100%][0m

[32m============================== [32m[1m3 passed[0m[32m in 0.02s[0m[32m ===============================[0m<jupyter_text>The package tree structures are labelled with \_\_init\_\_.py files in the directories of the hierarchy, and _only_ in the directories of the hierarchy. This encapsulates the folder inclusions to the "tests" and 2nd "number_returns" directories, and allows any self-import structure within number_returns to work straight from the code, or from the installed package in exactly the same way. Indeed, the code under the second "number_returns" is exactly what will be installed into the site-packages directory on a pip install.The only hack that needs to be put in place is a sys.path mod in tests/\_\_init\_\_.py (which is at least entirely scoped to that set of tests), which bridges the two trees described by the \_\_init\_\_.py files between "tests" and "number_returns". The trees for the installable code and tests are deliberately separate since only one of the trees will be packaged into a pip install-able module. The tests module is given its own name (rather than just being called tests) so that "buildable" can have multiple modules, each with their own test directory, and pytest will not get confused by multiple modules called "tests" (this confusion can be removed by treating the multiple test modules as namespace packages, but this then runs the risks of test name clashes in the multiple tests folders).<jupyter_code>!cat ../buildable/number_returns/number_returns_tests/__init__.py<jupyter_output>import os
import sys

sys.path.append(os.path.join(os.path.dirname(__file__), "..", "src"))<jupyter_text>Whilst strictly speaking not necessary, the \_\_init\_\_.py files enforce that the package builds will be "regular packages" as opposed to "namespace packages". The default action for developers should be making a "regular package", as this enforces tighter scoping.https://stackoverflow.com/questions/37139786/is-init-py-not-required-for-packages-in-python-3-3/https://packaging.python.org/en/latest/guides/packaging-namespace-packages/  File structureThe packaging process is covered by two main files:- setup.cfg - this specifies the install environment and general setup for the package; it defines what gets packaged, and what dependencies should be checked for/installed at the 'pip install' stage. This used to be a setup.py, but a py file is run and checked dynamically. Having a static file means the entire setup can be checked upfront in the build process. There are still a few things that only work with a setup.py and not a setup.cfg; the default option should be to use a setup.cfg- pyproject.toml - this specifies the build environment for the packagewith other supporting files that are brought along for the ride:- LICENSE - contains licensing info, in this case a copy of the MIT license- README.md - readme for the package  Build processThe build process is straightforward to run, and generates two objects that can be used to install; a wheel, and a tar.gz of the installables (which then still needs to go through the final parts of the build process on pip install).On build, it is always worth checking the output to see that all the files are packaged. Unzipping the tar.gz is also a good way of checking that all the relevant files have been packaged.<jupyter_code>!cd ../buildable/number_returns; python -m build
!tree -I __pycache__ ../buildable<jupyter_output>[01;34m../buildable[00m
└── [01;34mnumber_returns[00m
    ├── [01;34mdist[00m
    │   ├── number_returns-0.0.1-py3-none-any.whl
    │   └── [01;31mnumber_returns-0.0.1.tar.gz[00m
    ├── LICENSE
    ├── [01;34mnumber_returns_tests[00m
    │   ├── __init__.py
    │   ├── [01;34mints[00m
    │   │   ├── __init__.py
    │   │   └── test_gimmes.py
    │   └── [01;34mstrs[00m
    │       ├── __init__.py
    │       └── test_gimme_strs.py
    ├── pyproject.toml
    ├── README.md
    ├── setup.cfg
    └── [01;34msrc[00m
        ├── [01;34mnumber_returns[00m
        │   ├── __init__.py
        │   ├── [01;34mints[00m
        │   │   ├── gimmes.py
        │   │   └── __init__.py
        │   └── [01;34mstrs[00m
        │       ├── gimme_strs.py
        │       └── __init__.py
        └── [01;34mnumber_returns.egg-info[00m
            ├── dependency_links.txt
            ├── PKG-INFO
            ├── SOURCES.txt
            └── top_level.txt

[...]<jupyter_text>Either the tar.gz or wheel can be used to install the package. Note that doing a pip install in the notebook is usually a bad idea; installs should be taken care of by proper environment management. It is here for pedagogy.<jupyter_code>!python -m pip install ../buildable/number_returns/dist/number_returns-0.0.1-py3-none-any.whl<jupyter_output>Processing /home/jsnagi/proj/github/nagi49000/tutorial-memory-refs/python/buildable/number_returns/dist/number_returns-0.0.1-py3-none-any.whl
number-returns is already installed with the same version as the provided wheel. Use --force-reinstall to force an installation of the wheel.<jupyter_text>The package can then be used as usual<jupyter_code>import number_returns.strs.gimme_strs
number_returns.strs.gimme_strs.gimme5str()
number_returns.__file__  # this yields a location in the install area
# clean up after oursleves
!python -m pip uninstall -y number_returns<jupyter_output>Found existing installation: number-returns 0.0.1
Uninstalling number-returns-0.0.1:
  Successfully uninstalled number-returns-0.0.1
StarcoderdataJupyterStructuredCleanDedup