Awkward Array is a library for nested, variable-sized data, including arbitrary-length lists, records, mixed types, and missing data, using NumPy-like idioms.
Arrays are dynamically typed, but operations on them are compiled and fast. Their behavior coincides with NumPy when array dimensions are regular and generalizes when they're not.
Given an array of objects with x, y fields and variable-length nested lists like
array = ak.Array([
[{"x": 1.1, "y": [1]}, {"x": 2.2, "y": [1, 2]}, {"x": 3.3, "y": [1, 2, 3]}],
[],
[{"x": 4.4, "y": [1, 2, 3, 4]}, {"x": 5.5, "y": [1, 2, 3, 4, 5]}]
])the following slices out the y values, drops the first element from each inner list, and runs NumPy's np.square function on everything that is left:
output = np.square(array["y", ..., 1:])The result is
[
[[], [4], [4, 9]],
[],
[[4, 9, 16], [4, 9, 16, 25]]
]The equivalent using only Python is
output = []
for sublist in array:
tmp1 = []
for record in sublist:
tmp2 = []
for number in record["y"][1:]:
tmp2.append(np.square(number))
tmp1.append(tmp2)
output.append(tmp1)Not only is the expression using Awkward Arrays more concise, using idioms familiar from NumPy, but it's much faster and uses less memory.
For a similar problem 10 million times larger than the one above (on a single-threaded 2.2 GHz processor),
- the Awkward Array one-liner takes 4.6 seconds to run and uses 2.1 GB of memory,
- the equivalent using Python lists and dicts takes 138 seconds to run and uses 22 GB of memory.
Speed and memory factors in the double digits are common because we're replacing Python's dynamically typed, pointer-chasing virtual machine with type-specialized, precompiled routines on contiguous data. (In other words, for the same reasons as NumPy.) Even higher speedups are possible when Awkward Array is paired with Numba.
Our presentation at SciPy 2020 provides a good introduction, showing how to use these arrays in a real analysis.
Awkward Array can be installed from PyPI using pip:
pip install awkward1You will likely get a precompiled binary (wheel), depending on your operating system and Python version. If not, pip attempts to compile from source (which requires a C++ compiler, make, and CMake).
Awkward Array is also available using conda, which always installs a binary:
conda install -c conda-forge awkward1If you have already added conda-forge as a channel, the -c conda-forge is unnecessary. Adding the channel is recommended because it ensures that all of your packages use compatible versions:
conda config --add channels conda-forge
conda update --all
|
|
- Report bugs, request features, and ask for additional documentation on GitHub Issues.
- If you have a "How do I...?" question, ask about it on StackOverflow with the [awkward-array] tag. Be sure to include tags for any other libraries that you use, such as Pandas or PyTorch.
- To ask questions in real time, try the Gitter Scikit-HEP/awkward-array chat room.
Be sure to clone this repository recursively to get the header-only C++ dependencies.
git clone --recursive https://github.com/scikit-hep/awkward-1.0.gitYou can install it on your system with pip, which uses exactly the same procedure as deployment. This is recommended if you do not expect to change the code.
pip install .[test,dev]Or you can build it locally for incremental development. The following reuses a local directory so that you only recompile what you've changed. This is recommended if you do expect to change the code.
python localbuild.py --pytest testsThe --pytest tests runs the integration tests from the tests directory (drop it to build only).
For more fine-grained testing, we also have tests of the low-level kernels, which can be invoked with
python dev/generate-tests.py
python -m pytest -vv -rs tests-spec
python -m pytest -vv -rs tests-cpu-kernelsFurthermore, if you have an Nvidia GPU, you can build and locally install the experimental CUDA plug-in with
pip uninstall -y awkward1-cuda-kernels
python dev/generate-cuda.py
./cuda-build.sh --installThe --install does a local pip install on your system, which is the only way to use it. You can run its tests with
python dev/generate-tests.py
python -m pytest -vv -rs tests-cuda-kernels
python -m pytest -vv -rs tests-cuda- Continuous integration and continuous deployment are hosted by Azure Pipelines.
- Release history (changelog) is hosted by ReadTheDocs.
- awkward-array.org is hosted by Netlify.
- CONTRIBUTING.md for technical information on how to contribute.
- Code of conduct for how we work together.
- The LICENSE is BSD-3.
Python projects can simply import awkward1.
C++ projects can link against the shared libraries libawkward-cpu-kernels.so and libawkward.so or their static library equivalents. These libraries are shipped, along with the include files, as part of pip's installation.
- See the dependent-project directory for examples.
- Original motivations document from July 2019, now out-of-date.
- StrangeLoop talk on September 14, 2019.
- PyHEP talk on October 17, 2019.
- CHEP talk on November 7, 2019.
- CHEP 2019 proceedings (to be published in EPJ Web of Conferences).
- Demo for Coffea developers on December 20, 2019.
- Demo for Numba developers on January 22, 2020.
- Summary poster on February 27, 2020.
- Demo for Electron Ion Collider users (video) on April 8, 2020.
- Presentation at SciPy 2020 (video) on July 5, 2020.
- Tutorial at PyHEP 2020 (video with interactive notebook on Binder) on July 13, 2020.
Support for this work was provided by NSF cooperative agreement OAC-1836650 (IRIS-HEP), grant OAC-1450377 (DIANA/HEP) and PHY-1520942 (US-CMS LHC Ops).
Thanks especially to the gracious help of awkward-array contributors (including the original repository).
💻: code, 📖: documentation, 🚇: infrastructure, 🚧: maintainance, ⚠: tests and feedback, 🤔: foundational ideas.