The issue could, under specific conditions, unintentionally delete files from your system during the uninstallation process. Anaconda has published more details in the related blogpost about the security fix for the miniconda and Anaconda Distribution Windows installers as well.

conda-forge is committed to fix the miniforge and mambaforge installers equally to reduce the possible impact on conda-forge users and has worked with Anaconda to mitigate the issue.

• As such, we are strongly recommending all users of miniforge and mambaforge to update immediately to the latest versions of miniforge and mambaforge. Please download them from the miniforge repository's main page or the release specific page.
• For older versions, we are providing a security patch for already installed miniforge and mambaforge installations. You can download these from release specific page as well, under the names Miniforge3-uninstaller-patch-Windows-x86_64.exe and Mambaforge-uninstaller-patch-Windows-x86_64.exe.

In order for this flaw to be triggered, a specific combination of factors must align, including uninstallation permissions, system access, usage of Windows, and an existing installation of miniforge or mambaforge.

We have produced a list of all possibly compromised artifacts.

If you use conda-forge in very sensitive environments (which we do not recommend!), please remove these artifacts from your system.

To date, we know of no compromised artifacts in conda-forge.

API tokens for the main conda-forge channel were never exposed and remain secure to our knowledge.

Our Response​

We took the following steps to respond to this incident.

• We immediately started a token rotation of all of our feedstock tokens and our staging area upload tokens as precautionary measures. This token rotation hit a few bugs, but was completed as of January 13, 2023.
• We produced a census of all packages uploaded between December 19, 2022 and January 13, 2023. This data is available for download as a JSON file.
• We examined all the artifacts built during this time period for the malicious files listed by CicleCI. We did not find any of those files in our artifacts.
• As detailed below, we have begun retooling our system for feedstock tokens to be more robust and enable greater flexibility in our response to incidents like this.
• We have begun systematically invalidating old tokens, decommissioning old bots, and minimizing permissions of our current tokens in order to further enhance conda-forge's security.

Rotating all of our tokens was taken as a precautionary measure. Unfortunately, during this token rotation, one of our bots encountered a bug which resulted in us losing the tokens for a very large fraction of feedstocks. This situation resulted in an extended outage that lasted about five days and was resolved on January 13, 2023, when the full token rotation was completed.

What did we learn?​

We learned a few things about our system for feedstock tokens and general maintenance of our CI service integrations. We probably should have known them already, but here we are.

• We used the same feedstock token across multiple CI services. This limited our ability to immediately invalidate tokens associated with a single CI service and exposed all services if any single service had an incident.
• Our token system only allowed one valid token per feedstock. This limitation means that we cannot recover from partially failed token resets/rotations and are subject to race conditions during the reset/rotation process that can cause failed package uploads.
• We need to be more proactive about cleaning up deprecated/removed CI services. The use of CircleCI in conda-forge has been deprecated for quite a while. Had we taken the time, and had the foresight, to remove all of our secrets from CircleCI when it was deprecated, we could have avoided the security incident all together.

We have begun retooling our system for feedstock tokens in order to fix the issues identified above and allow us to have more flexibility in responding to security incidents. We have also started the process of decommissioning several of our old CI services. These changes will take time to implement. You can follow the progress on our various public issue trackers.

Closing Thoughts & What can you do?​

We, the conda-forge core dev team, want to thank everyone for their patience and support as we have responded to the various security incidents and bugs detailed above. It goes without saying that the public nature of conda-forge's infrastructure carries risks. On the other hand, by being public, anyone can look and verify our artifact builds. Security for conda-forge is about reducing risk, and we will continue to do our best.

As a reminder, we do not recommend that you use conda-forge in environments with sensitive information. conda-forge's software is built by our users and the core dev team cannot verify or guarantee that this software is not malicious or has not been tampered with.

Our best defense against security incidents in conda-forge is you! Our feedstock maintainers are in the best position to notice incidents and issues. Please responsibly report anything you find to us at condaforge+security@gmail.com.

The first issue I started working on when the internship began was Better anchoring of announcements(#1611). The goal of this issue was to fix the anchor for each year and also specific announcements in the Announcements section so as to provide better navigation of the Announcements page. This was also the time when I was feeling quite overwhelmed and anxious since I was just starting and was unsure if I would be able to give my best. But thanks to my awesome mentors @Katherine and @Matt, who have always been so helpful, I was able to have a good start. We solved this issue in two parts. The first part was to add anchors to each year which is solved with Improve anchors for each year in the Announcements section. (#1766), and the second part was adding anchors to each announcement and fixing the RSS feed.

The part of the documentation I focused on after completing the first issue was Maintainers' Documentation. Many open issues needed to be taken care of to make the Maintainers' documentation more useful and accessible for new maintainers. The open tickets that we have worked on are:

1. Document extras feedstock-name (#1769) and Explain how to become a maintainer (#1331). Closed with add extra section-recipe maintainer and feedstock-name (#1772).

   As we started with improving the Maintainer documentation, these were the issues we picked first to work on. The first issue was documenting how maintainers can use the "feedstock-name" directive for naming feedstocks differently than their package names in staged recipes. The second issue was documenting how one should become a package maintainer.

2. Add more steps in Improve the documentation section (#1651). Closed with Update "Improve the documentation" section with more steps (#1776).

   In this issue we added some additional steps for people who would like to start contributing to conda-forge, especially to documentation.

3. Add more information about Grayskull in the documentation itself (#1655). Closed with #1777.

   The documentation on Grayskull in docs lacked the answers to questions like what exactly Grayskull is and how one should use Grayskull to generate a recipe. With this issue, we added more documentation on Grayskull for users.

4. Clarify feedstock LICENSE.txt (#803). Closed with Add Feedstock repository structure section (#1786).

   The docs for contributing and maintaining conda recipes discuss when and how to distribute the license for a particular package. The auto-generated feedstock repositories also include a license in the root, which is different from the related package license. With this issue, we added documentation on the differences between those two licenses and briefly explained the feedstock repository structure.

5. DOC: New maintainer (#1117). Closed with Add "How regro-cf-autotick-bot create version update PR?" section. (#1788).

   With this issue we improved docs for the new maintainers and the working of the bot. A "How does regro-cf-autotick-bot create automatic version updates?" section was added to the docs, which explains the whole process of creating an automated version update PRs by bot.

6. Add Perl package hints to documentation (#1536). Working on this #1790.

   With this issue we added ​​packaging instructions for Perl packages with different build systems in the documentation.

7. DOC: Update documentation about tokens (#1532). Closed with #1793.

   Feedstocks have stopped storing encrypted tokens to upload packages, but outdated information on tokens was still present in the documentation. With this issue we removed the outdated information and also added a new section "How to update your feedstock token?" for maintainers.

8. Improve the documentation on arch_rebuild.txt (#1668). Closed with #1794.

   With this issue we improved the documentation on arch_rebuild.txt and how maintainers can add a feedstock to arch-rebuild.txt if it requires rebuilding with different architectures/platforms (such as ppc64le or aarch64).

9. Document migrators (#1355), Update migration docs (#862), and document migrators (#737) . Closed with Documenting Migrators and Migrations. (#1801).

   With these, we added more documentation on migrations and migrators, which would help maintainers find answers to questions like - What is a migrator/migration, and what does it do? When can (and why would) they should reject a migration PR? And so on.

10. Add a section in docs on security aspects of conda-forge (#1808). Closed with #1812.

    Currently, information regarding the security considerations of conda-forge builds is scattered throughout the documentation, and therefore it is hard to find and read. With this issue, we will put all the information together in one place, which will help maintainers and users to know more about how conda-forge secures its packages and infrastructure.

I met some wonderful people during the internship who helped me with all my questions and doubts that I had. The experience during the internship also helped me get better opportunities after completing the Outreachy internship. I have learned so many things during the internship that it would make a long list if I were to write all of those. But the most important things I learned are:

• The importance of documentation and how to write good documentation.
• The best practices to follow while writing documentation.
• More about conda-forge and packaging tools.

And above all, Outreachy helped me feel more confident about my skills and overcome the imposter syndrome I had before. Thanks again to my awesome mentors and the kind people of the conda-forge community! :)

Recently we've been able to add GPU-enabled TensorFlow builds to conda-forge! This was quite a journey, with multiple contributors trying different ways to convince the Bazel-based build system of TensorFlow to build CUDA-enabled packages. But we managed, and the pull request got merged.

We now have a configuration in place that creates CUDA-enabled TensorFlow builds for all conda-forge supported configurations (CUDA 10.2, 11.0, 11.1, and 11.2+). Building out the CUDA packages requires beefy machines -- on a 32 core machine it still takes around 3 hours to build a single package. Our build matrix now includes 12 CUDA-enabled packages & 3 CPU packages (because we need separate packages per Python version). As one can imagine, this isn't easily possible on an average "home computer".

For this purpose, we have written an Ansible playbook that lets us boot up cloud machines which then build the feedstock (using the build-locally.py script). Thanks to the generous support of OVH we were able to boot multiple 32-core virtual machines simultaneously to build the different TensorFlow variants.

We have open-sourced the Ansible playbook in GitHub and we're working towards making it (more) generally useful for other long-running builds!

With the TensorFlow builds in place, conda-forge now has CUDA-enabled builds for PyTorch and Tensorflow, the two most popular deep learning libraries.

We are still missing Windows builds for TensorFlow (CPU & CUDA, unfortunately) and would love the community to help us out with that. There is an open PR, but it probably needs some poking in Bazel to get it to pass: conda-forge/tensorflow-feedstock#111.

We hope that these new GPU builds will enable many more packages to be added to the conda-forge channel! We are already looking forward to the 2.6.2 and 2.7 releases of TensorFlow and to adding Windows support in the future. We hope you enjoy this work.

Installation​

You can now select between GPU enabled (default) and CPU packages using the tensorflow-gpu and tensorflow-cpu packages. Just run

mamba install tensorflow-gpu -c conda-forge
# OR
conda install tensorflow-gpu -c conda-forge

When installing the tensorflow package, the package resolution will now default to the GPU-enabled builds of tensorflow if the local machine has a GPU (these builds can be identified by "cuda" at the beginning of the version number). Note that GPU-enabled packages can also work on CPU-only machines, but one would need to override the enviornment variable CONDA_OVERRIDE_CUDA like below. This could be handy if you are in a situation where your current node (e.g. login node) on an HPC does not have GPUs, but the compute nodes with GPUs do not have internet access.

CONDA_OVERRIDE_CUDA="11.2" conda install tensorflow cudatoolkit>=11.2 -c conda-forge
# OR
CONDA_OVERRIDE_CUDA="11.2" mamba install tensorflow cudatoolkit>=11.2 -c conda-forge

Note that you should select the cudatoolkit version most appropraite for your GPU; currently, we have "10.2", "11.0", "11.1", and "11.2" builds available where the the "11.2" builds are compatible with all cudatoolkits>=11.2. You could also force a specific version of cudatoolkit by specifying it like above. Moreover, you could ensure you get a sepcific build of tensorflow by appending the package name like tensorflow==2.7.0=cuda* or tensorflow==2.7.0=cuda112*. If you want the slimmer "cpu-only" package, then you can install tensorflow-cpu directly or equivalently tensorflow==2.7.0=cpu*. At the time of writing (February 2022), on a machine without a GPU, one would always get the -cpu variant unless overriden like above. This decision has been made to allow greater accessibility for users with limited bandwidth and resources.

Thanks to​

• Mark Harfouche (@hmaarrfk) & Ista Zahn (@izahn) for their initial work on the TensorFlow GPU builds, and all other TensorFlow maintainers. Uwe Korn (@xhochy) for his work on the Bazel scripts & TensorFlow -- and all the other maintainers of the TensorFlow feedstock!
• NVIDIA for pushing cudatoolkit and cudnn on conda-forge that makes this possible
• OVH for their generous sponsoring of large build machines that we could use to build the recipes
• Bloomberg for their sponsorship of QuantStack's involvement with conda-forge
• Andreas Trawoger (@atrawog) for the Ansible scripts that this is based on
• Thorsten Beier (@derthorsten) and Adrien Delsalle (@adriendelsalle) for their contributions to the recipe

Our Response​

We took the following steps to respond to this incident.

1. We immediately turned off all builds on Travis CI by suspending the Travis CI GitHub App.
2. We immediately disclosed the bug to Travis CI through our contacts there.
3. Once Travis CI indicated to us that they were ready, we rotated all feedstock tokens and later our anaconda.org token for our staging channel. The anaconda.org token for the main conda-forge channel was never disclosed in this incident. Further, only ~70 feedstocks had their tokens exposed in this incident.
4. We examined our artifacts and marked as broken any artifacts that were uploaded from PRs. We think we found everything, but we are not completely sure. Our criterion for marking things broken was more generous than it needed to be.
5. We issued PRs to rebuild any broken artifacts via our bots.
6. We put in changes to conda-smithy to help prevent inadvertent uploads of artifacts from PRs in the future.

Closing Thoughts & What can you do?​

I (MRB) want to recognize the quick work of our core dev team in handling this incident. It goes without saying that the public nature of conda-forge's infrastructure carries risks. On the other hand, by being public, anyone can look and verify our artifact builds. Security for conda-forge is about reducing risk and we will continue to do our best.

Our best defense against security incidents in conda-forge is you! Our feedstock maintainers are in the best position to notice incidents and issues. Please responsibly report anything you find to us at condaforge+security@gmail.com.

As the introduction for Grayskull reads; "The main goal of this project is to generate concise recipes for conda-forge." In this tutorial we learn how to contribute a Python package to the conda-forge channel using Grayskull to generate the recipe.

Let us get started.

1. Install grayskull using conda through the conda-forge channel:

   $ conda install -c conda-forge grayskull
   

2. Fork and clone the conda-forge staged-recipes repository from GitHub.

3. Checkout a new branch from the master branch.

4. Through CLI, cd inside the 'staged-recipes/recipes' directory.

5. Call grayskull and pass the pypi repository, followed by the name of the package you want to contribute to conda-forge. For example: grayskull pypi abc

   Or you could use grayskull pypi abc --strict-conda-forge to remove some selectors which are not necessary for conda-forge and adapt recipes to fit better in the conda-forge ecosystem.

   Grayskull will create a folder with the same name as the package (in this case: 'abc') in the 'recipes' folder of the 'staged-recipes' directory. This folder will contain the meta.yaml file and also the license file if the package includes a license in the PyPI distribution.

6. Go through the generated meta.yaml file. For simpler packages, the generated recipes are nearly perfect, but for some packages you might need to make certain tweaks.

7. Commit and push the changes. git add recipe/abc/meta.yaml git commit -m "add a commit message" git push

8. Create a PR.

9. Once the CI is passing, post a comment saying: This is ready for review @conda-forge-admin, please ping team

Once the PR gets merged, your package will be available on the conda-forge channel. Tada! It's that easy.

Participant Application Process:​

First, please review the Outreachy Eligibility and Application Information page to learn more about eligibility for Outreachy.

Steps for applicants to conda-forge:​

1. Confirm your eligibility on the Outreachy site
2. Look at the Conda-forge projects available on the Outreachy site, consider your options, and if you have questions, communicate with the project mentors.
3. Begin by contributing to the project by looking at our issues page. As you make contributions, record them on the Outreachy site.
4. Once you have made a few contributions, begin to write your application. Ask the mentors to review the application before you submit it.

Participant Expectations​

You will be working full-time on your project for three months. You will meet with your mentor(s) frequently and participate in the open-source development process -- writing code, reviewing code, testing, and so on. You will be expected to write a blog entry each week.

Project Contribution Information​

As part of the application process, all applicants must make at least one contribution to be accepted as an intern for this project. Only applicants who make a contribution will be eligible to be accepted as interns.

While we don't have one we highly recommend the first-time contributor to be a conda user and/or submit a package to conda-forge via staged-recipes. That will ensure the contributor understands the value of what we do and means that they are willing to participate in our community.

Applicants can contribute to this project through the project repository or contribution page. The project uses an issue tracker to keep information about bugs to fix, project features to implement, documentation to write, and more. Applicants can look for newcomer-friendly issues to use for their first contributions by looking for the following issue tags in the project issue tracker: Docs, Good first issue

We here at conda-forge have a large number of potential Outreachy endeavors around documentation, maintenance, and development. These tasks are high-impact, affecting the entire conda-forge ecosystem. They also cover multiple systems including databases, conda's CDN provider, continuous integration providers, and user interactions on GitHub.

How do I work with the conda-forge community?​

Outreachy applicants can get help and feedback from both mentors and community members. Community members discuss their contributions in a public chat. Outreachy applicants can often learn from those discussions.

Please introduce yourself on the public project chat:

• Gitter - Follow this link to join this project's public chat.
• Outreachy mentors will often be in the community public chat. The project mentor's usernames are: @viniciusdc.

Here are some ready-to-go ways you can get started contributing on your own.

• Find an open issue to tackle or report a bug to the issue tracker;
• Don't be afraid to communicate: Ask if you can help write a new feature or help Automate project setups;
• Improving current tooling and testing features is always welcome.

As this project main goal is enhancing our current documentation, here are some preliminary tasks that you can inspect to get ideas:

• Write and improve the project's documentation;
• Link to duplicate issues, and suggest new issue labels, to keep things organized;
• Go through open issues and suggest closing old ones;
• Ask clarifying questions on recently opened issues to move the discussion forward;
• We also have issues regarding the main functionalities of our bot, in particular the autotick bot. You could find some new information or ideas for your contributing proposals.

Good starter tasks:​

Small Starter Tasks​

As with most organizations, there are lots of small issues that need addressing usually related to problems such as bad recipes, old documentation and others. These will make good first issues to resolve or "update". This will also be an opportunity to familiarise yourself with the conda-forge environment.

Larger tasks​

There are a few potential larger tasks that can come after a few smaller task contributions. These are included into our three main bases:

• Users: In this case, some good starter tasks are mainly checking the actual contents of conda-forge users documentations, and ideas to better express its contents.
• Maintainers: There are a bunch of missed topics in this area, some information have to be updated or rewritten for better understanding. Writing a complete guide containing the actual steps and standard model for a package recipe, building process (just a simple discussion) and how conda-forge bot recognize defective licenses, recipes and packages in general is highly welcomed. For further understanding of the general system check this link.
  • It can be funny to say, but lots of helpful ideas and bug solutions appear on our gitter channel, so if you have time to write guides about them... it's also an incredible task.
• And organization Our environment is changing everyday, because of that a lot of information is lost in this process or even worse, not documented at all! which leads to some difficulties inserting new members to develop and further enhance the current process.
  • The related work on this matter is highly welcomed and for a better grasp of the situation you can start with this guideline and read some of our posts in our blog
  • Revitalizing ideas/projects for the conda-forge blog are definitely welcomed;
  • Currently we have some interesting projects going on inside our ecosystem, which in return will need good documentation... Some of the projects conda-forge is affiliated include the auto-tick bot, symbol-exporter and a new service we are eager to start developing is the distributed-bot. All of them have a great coverage of subjects and lots of people to help and give advice about the service structure and functionalities.

Improving the documentation​

You can help improve the documentation as it is version-controlled in the conda-forge.github.io repository on GitHub. The source text is stored there in the src/subdirectory and is formatted using Python's reStructuredText system.

You can propose quick edits directly through the GitHub website, if you have an account there --- for instance, this link will take you directly to a web-based editor for this section page in our docs. In general, the file corresponding to each page in the GitHub browser has a little pencil icon in its top-right that lets you open it up for editing.

The more manual process is as follows:

• Fork the conda-forge.github.io repository to your own GitHub user account.
• Clone that fork onto your computer.
• Check out a new branch deriving from master to do your work.
• Make and commit your changes.
• Submit a pull request to the main repository proposing your changes.

Happy editing!

Strong Growth​

The conda-forge community has grown immensely this year. Here are some numbers to help give you an idea of the scale of our growth.

• The community has added 3,751 new, unique conda packages this year, along with a corresponding number of new feedstocks.
• For the majority of 2020, the conda-forge channel on anaconda.org exceeded 100 million downloads per month.
• In July of 2020, the conda-forge channel passed 2 billion total, all-time downloads.
• We've grown our core developer community, adding seven new members to the conda-forge Core team and at least two members to the staged-recipes team.
• We now have over 2,500 recipe maintainers in the conda-forge GitHub organization.

Big New Features​

We've also shipped a ton of big updates to our core infrastructure this year. These updates include

• PyPy support: We added support for PyPy 3.6 and now supply one of the biggest stacks of PyPy-enabled packages in the PyPy ecosystem.
• automerge: We now support the automatic merging of PRs on feedstocks using the automerge label or through an opt-in setting in the conda-forge.yml.
• R 4.0 migration: This migration was the first one to use our automerge infrastructure at scale. With it, we completed a complete rebuild/upgrade of the R ecosystem in about a week.
• Python updates: We deprecated Python 2.7, completed the Python 3.8 migration, and got about 75% of the way through the Python 3.9 migration.
• compiler upgrades: We upgraded our compiler infrastructure to GCC 9 and clang 11.
• CentOS 7 and CentOS 6 EOL: We shipped an option to enable our compilers to use the CentOS 7 sysroot in preparation for the CentOS 6 EOL. We hope to complete the move to CentOS 7 early next year.
• miniforge: We built our own standalone, miniconda-like installers. These support a broad range of platforms, including osx-arm64 and linux-aarch64.
• standalone Windows stack: We fully decoupled our Windows recipes from the defaults channel by rebuilding the msys2 recipes.
• Apple silicon support: We added support for Apple silicon with our osx-arm64 platform. This platform is our first one to use a fully cross-compiled infrastructure.
• CUDA support: We added support for building CUDA packages on windows and added CUDA 11.0 support.

We know that this year has been extremely difficult for so many of our community members and that the fantastic success of conda-forge would not have been possible without the active participation and support of our community. Thank you everyone so much for the work you put into conda-forge this year, making it the wonderful, community-led resource that it is.

We wish everyone a happy, healthy, and peaceful new year!

What makes it non-surprising [that no other free conda package distribution services have appeared] is that, at the moment, any third party channel like conda-forge is free. The TOS change does not apply to conda-forge, nor to other channels hosted on anaconda.org; the TOS change in question applies only to the "defaults" channel and other software hosted on repo.anaconda.com.

While having alternative hosting is in our plans, we cannot afford the costs. We are just a community of volunteers. We have experimented with uploading the conda-forge artifacts to GitHub and continue to do so (see regro/releases). We also have put those artifacts behind an experimental repodata server. One of our core devs (@wolfv) is working to setup quetz with conda-forge artifacts as well.

It is very important to recognize that Anaconda Inc kindly donates use of their hosting and employee time to us, absorbing all of these costs. They host about 1.8 TB of our data and serve over 100 million downloads of artifacts from that data each month. This is a highly significant donation and conda-forge would not exist without it. Anaconda Inc employees also provide help with maintaining some of the most complex package recipes in conda-forge.

Note that Anaconda Inc has also said that part of the revenue from the TOS change will be donated to OSS projects. (see this blog post). You should be aware that conda-forge is a part of NumFOCUS, and so it stands to benefit from the change in TOS, as do many other OSS projects.

We absolutely welcome help from the community to move our efforts on building out more hosting infrastructure for conda-forge forward. This could be anything from spending time developing quetz to providing hosting/mirroring for our data. Please do get in contact with us if you'd like to help out!

This will install a conda environment with python and conda in it. Installed conda will be able to install packages like numpy, scipy. Currently there are about 100 packages out of 10000 packages pre-built for this platform.

All these packages are built on conda-forge's current macOS x86_64 infrastructure. In order to do so, we have made lots of changes to the infrastructure including, conda, conda-build, conda-smithy, constructor, conda-forge-ci-setup to facilitate cross-compiling which is the process of compiling a package that will run on a host platform (osx-arm64 in our case), with the compilation done on a build platform (osx-64 or linux-64 in our case).

osx-arm64 is the first conda platform that is completely bootstrapped using conda-build's cross-compiling facility. Previously, when adding a new platform, conda-build was built with an existing python and pip environment on the new platform. With cross-compiling, when the compilers and a sysroot is set up on a different platform, an existing conda-build installation (on osx-64 and linux-64 in this case) will be able to start building packages right away.

Cross-compiling builds for osx-arm64​

In order to cross compile packages for osx-arm64 we need compilers. So, we first built clang=11.0.0.rc1 which has support for targetting osx-arm64. We also built compiler-rt=11.0.0.rc1 as a universal build support both osx-64 and osx-arm64.

Linker, archiver, otool, install_name_tool was built using the cctools-port project by Thomas Pöchtrager.

One issue we ran into was that the macOS 11.0 Big Sur Beta 7 required that all executables and shared libraries be ad-hoc signed which is signing without a verified signature. On suggestion of cctools-port developer we added support to cctools-port to sign these executables using ldid which can be used on Linux as well as macOS to sign.

Using these, the first cross compiled package we built was libcxx to facilitate C++ builds. For the osx-arm64 sysroot we used the MacOSX11.0.sdk already installed on Azure pipelines and Travis-CI. Due to licensing issues, we cannot distribute this, but it can be downloaded from the Apple developer website even on Linux.

With clang we have a C/C++ compiler, but lack a Fortran compiler. We used the GCC fork for darwin-arm64. First, a cross compiler (build == host != target) was built. Using that compiler, we built a cross-native compiler (build != host == target) which gave use the shared libraries like libgfortran.dylib.

We also added support for cross compiling rust programs to the rust packages in conda and installing rust_osx-64 on Linux will give you a compiler that will build packages for osx-64.

As we haven't done cross-compilation before, many packages needed to be updated. Most were trivial changes that we automated later on. These included getting a newer config.sub to identify the new autotools platform arm64-apple-darwin20.0.0, adding options to CMake with the environment variable CMAKE_ARGS to correctly set up the toolchain and recipes were update to use cmake ${CMAKE_ARGS} ... Running tests when building were also disabled by guarding commands like make check, make test, ctest with the env variable CONDA_BUILD_CROSS_COMPILATION.

Cross-compiling python extensions is quite tricky as distutils is not really setup to do this. Thanks to the project crossenv this is unofficially supported with a few quirks. With crossenv, we can run a python on the build machine (osx-64 or linux-64 in this case) that acts like it is on osx-arm64. crossenv monkey-patches a few functions like os.uname and sets up values like _PYTHON_SYSCONFIG_DATA to make python running on osx-64 or linux-64 behave like osx-arm64. One issue is that, monkey-patching sys.platform doesn't work and therefore if a python package in it's setup.py uses sys.platform to differentiate OSes this will lead to unintended consequences if you are cross-compiling from linux-64. Therefore, we have to use osx-64 as our build system when cross-compiling for osx-arm64. Note that packages using sysconfig.get_platform() will get the correct platform.

For creating an installer for conda, we needed a standalone conda executable to bootstrap the conda environment. For other platforms we relied on conda-standalone which is a standalone conda executable created using pyinstaller. Since pyinstaller does not support cross-compile, we decided to use micromamba as the bootstrapper and added features to micromamba so that it can function as the bootstrapper.

How to add a osx-arm64 build to a feedstock​

All the below changes will be done by a bot and the packages the bot will send PRs to is determined by the list of packages at conda-forge-pinning and their dependences. If you would like to add support, please send a PR adding the feedstock name to the above list. After that PR is merged, you can monitor the status at conda-forge status-page and if a particular PR is stalled you can send a PR to the feedstock to fix it.

Following instructions are for when you want to add support manually.

Add the following to conda-forge.yml (on Linux or OSX),

build_platform:
  osx_arm64: osx_64
test: native_and_emulated

You can rerender using,

conda smithy rerender

For python packages, add one or more of the following to recipe/meta.yaml as needed, noting that you must only add numpy, cython, and/or pybind11 if they are used in host: as well,

requirements:
  build:
    - python                                 # [build_platform != target_platform]
    - cross-python_{{ target_platform }}     # [build_platform != target_platform]
    - cython                                 # [build_platform != target_platform]
    - numpy                                  # [build_platform != target_platform]
    - pybind11                               # [build_platform != target_platform]

For autotools package, add the following to recipe/meta.yaml,

requirements:
  build:
    - gnuconfig   # [unix]

and to recipe/build.sh,

# Get an updated config.sub and config.guess
cp $BUILD_PREFIX/share/gnuconfig/config.* .

For cmake packages, add the following to recipe/build.sh,

cmake ${CMAKE_ARGS} ..

For meson packages, add the following to recipe/build.sh,

meson ${MESON_ARGS} builddir/

For rust packages, add the following to recipe/meta.yaml,

requirements:
  build:
    - {{ compiler('rust') }}

If there's a line like make check in recipe/build.sh that cannot be run when cross-compiling, do the following,

if [[ "$CONDA_BUILD_CROSS_COMPILATION" != "1" ]]; then
  make check
fi

After these changes, another rerendering might be required.

Some useful jinja variables,

1. build_platform - conda subdir for BUILD_PREFIX. eg: linux-64
2. target_platform - conda subdir for PREFIX. eg: osx-arm64

Some useful environment variables,

1. build_platform
2. target_platform
3. CONDA_BUILD_CROSS_COMPILATION - 1 if cross compiling
4. CMAKE_ARGS - arguments to pass to cmake
5. CC_FOR_BUILD - C compiler for build platform
6. CXX_FOR_BUILD - C++ compiler for build platform
7. HOST - a triplet for host passed to autoconf. eg: arm64-apple-darwin20.0.0
8. BUILD - a triplet for build passed to autoconf. eg: x86_64-conda-linux-gnu

Some useful configure options in conda-forge.yml

1. build_platform - a dictionary mapping build subdir to host subdir. eg:

   build_platform:
     osx_arm64: osx_64
     linux_ppc64le: linux_64
     linux_aarch64: linux_64
   

2. test_on_native_only - a boolean to turn off testing on cross compiling. If the tests don't require emulation (for eg: check that a file exists), then test_on_native_only: false will run the tests even when cross compiling.

Building locally​

For building locally add the following in $HOME/conda_build_config.yaml.

SDKROOT:
  - /path/to/MacOSX11.0.sdk

After that, look for the config you want to run in .ci_support folder in the root of the feedstock For eg: .ci_support/osx_arm64_.yaml. Then run,

conda build recipe -m .ci_support/osx_arm64_.yaml -c conda-forge -c conda-forge/label/rust_dev

This should start a new build for osx-arm64.

Testing packages​

In order to test packages intended to run on future Apple Silicon hardware, Apple provides a machine called Developer Transition Kit (DTK). Jonathan Helmus and Eli Rykoff has helped with testing these packages on DTKs. Thanks to Eli Rykoff, we are now running tests for these packages as a daily cron job which has led to finding several bugs in our cross compiling infrastructure and also bugs in our recipes.

To test cross compiled recipes, transfer the built conda package to the host and run,

conda build --test /path/to/package -c conda-forge

This work would not have been possible without the help of many people including the upstream maintainers of compiler infrastructure (which includes conda, conda-build, cctools, tapi, cctools-port, ldid, llvm, clang, compiler-rt, openmp, libcxx, crossenv, rust, gcc-darwin-arm64), conda-forge/help-osx-arm64 team including Matt Becker, Eli Rykoff and Uwe Korn who sent PRs to fix recipes, conda-forge/bot team and also all the conda-forge maintainers of the 100 feedstocks who reviewed and fixed PRs.

Isuru Fernando

What Constitutes a Good Version Number​

Version numbers should constitute a set that has the following properties

1. The set must be unbounded
2. The set must be orderable (maybe)

Of course sets that meet these requirements might not convey a lot of information about the software they represent other than if two things are equivalent and their comparative ages. Note that the requirement to be orderable may not be needed, but is generally useful when considering the idea of an "upgrade" since it provides a clear delineation between older and newer packages. In many cases, the structure of the version number provides additional information. For some projects the version number includes the date of the release, often using cal ver. Many projects use semantic versioning, which attempts to encode information about the underlying source code's API in the version number.

Version Numbers and API Pinning​

One of the most important places where version numbers are specified is during the pinning of APIs. Source code often requires specific APIs from the libraries it uses. This requires a pin specifying which versions of the underlying libraries can be used. The package manager then uses these pins to make certain a compatible environment is created.

However, these pins (or even the lack of pins) produce problems. Firstly, the pins are a one-time, local statement about the current and future, global ecosystem of packages. For instance a pin of scipy to the current major version number may not hold up over time, newer versions of scipy may break the API while not changing the major version number. Similarly the lack of pin for scipy could be false as the API breaks. Even pins that establish firm upper and lower bounds may be false as new versions of the pinned library restore the missing API. These issues are particularly problematic for dependency systems that tie the pins to a particular version of the source code, requiring a new version to be created to update the pins. Conda-Forge is able to avoid some of these issues via repodata patching, dynamically updating a package's stated requirements. Overall this process is fraught, as each package depends on different portions of a library's API, a version bump that breaks one package may leave others unscathed.

A Potential Path Forward​

All of the above issues are caused by the confusion of the map for the territory. The map, in this case the version number of a library, can not accurately represent the territory, the API itself. To fix this issue we need a more accurate description of the territory. Achieving this will not be easy, but I think there is an approach that gets close enough to limit the number of errors.

We need a programmatic way to check if a particular library, for a particular version, provides the required API. I think this can be achieved iteratively, with each step providing additional clarity and difficulty of implementation. Note that in the steps below, I'm using python packaging as an example, but I imagine that these steps are general enough to apply to other languages and ecosystems.

1. Determine which libraries are requirements of the code, this is provided by tools like depfinder and are starting to be integrated into the Conda-Forge bot systems (although they are still highly experimental and being worked on).
2. Determine if the a version of the library provides the needed modules. This could be accomplished by using depfinder to find the imports and use the mapping provided by libcfgraph between the import names and the versions of packages that ship those imports.
3. Determine if an imported module provides the symbols being imported. This would require a listing of all the symbols in a given python module, including top level scoped variables, function names, class names, methods, etc.
4. For callables determine if the used call signature matches the method or function definition.

The depfinder project has made significant advances along this path, providing an easy to use tool to extract accurate import and package requirement data from source code. Depfinder even has cases to handle imports that are within code blocks that might make the requirement optional or use the python standard library. Future work on depfinder, including using more accurate maps between imports and package names and providing metadata on package requirements that are collectively exhaustive (for instance imports of pyqt4 vs. pyqt5 in a try: except: block), will provide even more accurate information on requirements.

At each one of the above stages we can provide significant value to users, maintainers and source code authors by helping them to keep their requirements consistent and warning when there are conflicts. Conda-Forge can update its repodata as new versions of imported libraries are created, to properly represent if that version is API compatible with it's downstream consumers. Additionally the tables that list all the symbols and call signatures can be provided to 3rd party consumers that may want to patch their own metadata or check if a piece of source code is self consistent in its requirements. This will also help with the loosening of pins, creating more solvable environments for Conda-Forge and other packaging ecosystems. Furthermore, as this tooling matures and becomes more accurate it can be incorporated into the Conda-Forge bot systems to automatically update dependencies during version bumps and repodata patches, helping reduce maintenance burden.

Tools built from the symbol table can also have impacts far beyond Conda-Forge. For instance, the symbol tables could allow source code authors to have a line by line inspection of their code, revealing which lines force the use of older or newer versions of dependencies. This could enable large scale migrations of source code with surgical precision, enabling developers to extract and re-write the few lines of code preventing the use of a new version of a library.

Caveats​

There are some important caveats to this approach that need to be kept in mind.

1. All of this work is aimed at understanding the API of a given library, this approach can not provide insight into the code inside of the API, or if changes there impact downstream consumers. For instance, version updates that fix bugs and security flaws in library code may not change the API at all. From this tooling's perspective there is no reason to upgrade since the API is not different. Of course there is a strong reason to upgrade in this case, since buggy or vulnerable libraries could be a huge headache and liability for downstream code and should be removed as quickly as possible.
2. Some features may depend on broader adoption by the community. For instance, this approach would benefit greatly from python type hints, since the API could be constrained down to the expected types. Such type constraints would provide much more accuracy to the API version range as any changes could be detected. However, type hints may not be adopted in the python community at a high enough rate to truly be useful for this application.
3. Source code is fundamentally flexible. There may be knots of code that even this approach could not cut through, especially as multiple languages and runtime module loading come into the picture. My personal hope would be that the code recognizes when these situations occur, provides its best guess of what is going on, and provides sufficient metadata to users so that they understand the decreased accuracy of the results. Fundamentally the tooling can only provide very educated guesses and context to users, who then need to go figure out what is actually going on inside the code.

Conclusion​

Version number based pins are imprecise representations of API compatibility. More accurate representations based on source code inspection would make the Conda-Forge ecosystem more robust and flexible while reducing maintenance burden. Some of the path to achieving this is built, and near future steps can be achieved with current tooling and databases.

The R 4.0 migration rebuilt every package in conda-forge that had r-base as a requirement, including more than 2200 feedstocks. A migration of this size in conda-forge faces several hurdles. First, since every feedstock is a separate GitHub repository, one needs to merge more 2200 pull requests (PRs). Second, conda-forge's packages on anaconda.org are behind a CDN (content delivery network). This service reduces web hosting costs for Anaconda Inc. but introduces an approximately 30 minute delay from when a package is uploaded to anaconda.org and when it will appear as available using conda from the command line. Thus, even if the dependencies of a package have been built, we have to wait until they appear on the CDN before we can successfully issue the next PR and have it build correctly. Finally, the existing bot and conda infrastructure limited the throughput of the migrations, due in part to the speed of the conda solver.

Given the size of the R 4.0 migration, we took this opportunity to try out a bunch of new technology to speed up large-scale migrations. The main enhancements were using GitHub Actions to automerge PRs, using mamba to quickly check for solvability of package environments, and enabling long-running migration jobs for the autotick bot. All told, the bulk of the feedstocks for R 4.0 were rebuilt in less than a week, with many PRs being merged in 30 minutes or less from when they were issued. These enhancements to the autotick bot and conda-forge infrastructure can be used to enhance future migrations (e.g., Python 3.9) and reduce maintenance burdens for feedstocks.

Automerging conda-forge PRs​

In a typical migration on conda-forge, we issue a PR to a feedstock and then ask the feedstock maintainers to make sure it passes and merge it. In the case of the R 4.0 migration, the maintainers of R packages on conda-forge use a maintenance team (i.e., @conda-forge/r) on the vast majority of feedstocks. This team is small and so merging over 2000 PRs by hand is a big undertaking. Thus, with their permission, we added the conda-forge automerge functionality to all R feedstocks that they maintain. The automerge bot, which relies on GitHub Actions, is able to automatically merge any PR from the autotick bot that passes the recipe linter, the continuous integration services, and has the special [bot automerge] slug in the PR title. This feature removed the bottleneck of waiting for maintainers to merge PRs and reduced the maintenance burden on the R maintenance team.

Checking Solvability with mamba​

While being able to automatically merge PRs removed much of the work of performing the R 4.0 migration, it relied on the PR building correctly the first time it was issued. Due to the CDN delays and the build times of a package's dependencies, the dependencies of a package may not be immediately available after all of their migration PRs are merged. If the bot issued the packages migration PR before the dependents are available, the PR would fail with an unsolvable environment and have to be restarted manually. This failure would negate any of the benefits of using automerge in the first place.

To control for this edge case, we employed the mamba package to check for the solvability of a PR's environments before the PR was issued. mamba is a fast alternative to conda that produces solutions for environments orders of magnitude more quickly. Since, we have to perform our checks of PR environments many times, an extremely fast solver was essential for making the code efficient enough to run as part of the autotick bot. We ended up using mamba to try to install the dependencies for every variant produced by the feedstock to be migrated. With this check in place, the autotick bot was able to issue migration PRs that passed on the first try and were thus automatically merged, many within 30 minutes or less.

Improving the Autotick Bot's Efficiency​

Finally, we made several upgrades to the autotick bot infrastructure to increase the uptime of the bot and its efficiency. First, we moved from an hourly cron job to a set of chained CI jobs. This change eliminated downtime between the runs of the bot. Second, we started to refactor the autotick bot from one monolithic piece of code into a distributed set of microservices which perform various independent tasks in parallel. These independent tasks, used for things like checking the statuses of previously issued PRs, are run separately allowing the bot to spend more time issuing PRs. Finally, we optimized the internal prioritization of the PRs to make sure the bot was spending more time on larger migrations where there is more work to do. More work on the autotick bot infrastructure, including work done by Vinicius Cerutti as part of the Google Summer of Code program, will further streamline the bot's operation.

Despite some initial hiccups with the bot infrastructure, the migration ran quite smoothly for an endeavor of its size. The vast majority of migration PRs were completed within a week from when we started, which is a first for a migration of this size on conda-forge. The largest issue was solved recently, with the fixing of the openjdk recipe and the removal of aarch64 and ppc64le builds from r-java, enabling the last large piece of the R ecosystem to be updated.

Looking forward, the improvements we made for the R 4.0 migration seem broadly applicable to other migration tasks, including the yearly python minor version bump. These kinds of large-scale migrations are particularly suitable, since they usually involve few changes to the feedstock itself and usually fail on CI when a broken package would be produced. Faster migrations will help to provide the latest features to downstream users and keep transition times to a minimum, helping to foster greater stability of the ecosystem and the seamless experience users have come to expect from conda-forge.

Have some thoughts about conda-forge and how it can be expanded in a way that is sustainable? Join us in this virtual Birds of a Feather discussion where we'll discuss maintenance, pain points, opportunities within conda-forge. Any and all are welcome, and we especially are seeking new viewpoints and opinions!

BOF questions:​

State of Packaging​

• What is the state of conda-forge today?
• What improvements are in the works?

New features/ideas​

• What is something that's not in the works that you'd like to see updated/improved?
• What are ways that folks who are interested but new to the community can get involved?

Maintenance​

• What is the maintenance burden and what pain points does it create?
• What are strategies to mitigate this burden?

Questions to the audience​

• What are your pain points when engaging with conda forge?
• What is something new you'd like to see happen?
• What feedback do audience members have?

A brief conda-forge primer​

Conda-Forge is an ecosystem and community that grew around building packages for the conda package manager. Conda-Forge uses continuous integration services to build packages from GitHub repos called feedstocks. This structure enables teams of contributors to maintain packages via a pull request based workflow. At time of writing Conda-Forge has over 10000 feedstocks and ships more than 120 million packages a month.

Self-Service Structure​

Conda-Forge is built around a self-service structure for each stage in a feedstock's lifecyle. The creation of new feedstocks relies on would be maintainers to submit PRs to staged-recipes. Although language specific help teams and staged-recipes reviewers provide some assistance and oversight, the PR submitter plays the most important role in proposing the package and shepherding it to acceptance. Once the feedstock is accepted the maintenance is federated with most upkeep being performed by the maintainers, who have extensive permissions and control over the feedstock. If fixes or updates are needed for a package, maintainers and users are encouraged to open their own pull requests.

This structure can present a few challenges for minimizing op. risk. The most important challenge is the disconnect between feedstock maintainers and users. While most maintainers are package users, most of our users are not maintainers, and are unlikely to become maintainers. The disparity between maintainers and users can come from a few sources, some under our control and others not. For instance we can write better documentation, lowering the barrier to entry, but we don't have control over how our user's incentive structures value Conda-Forge contributions. This produces a gap in representation in the Conda-Forge organizational structure, where non-maintainer users' issues and desires are not communicated to maintainers and Core.

For instance, are we servicing the needs of developers using our binaries as dependencies to code they are compiling locally. As another example, are there support gaps for developers and scientists using Conda-Forge in academic and government laboratories, who might not have the skills or capacity to fix feedstocks. Our reliance on the public GitHub platform may prevent some users without access from raising their concerns. Since these users may be under-represented we don't even know if we are meeting their needs and how best to help.

Top-Down Control​

While the majority of Conda-Forge's permissions structure is federated, certain important parts are centralized, with the Core developers making key decisions. Often these decisions are focused on stability of the ecosystem, for instance what versions of languages to support. Additionally, maintenance and enhancements to the Conda-Forge infrastructure are mostly performed by Core developers.

However, the Core developers are usually experienced feedstock maintainers, expert conda users, and have bought into the Conda-Forge ecosystem and mission. This means that decisions can be made without the perspective of new users or maintainers, or from potential users that are skeptical of the Conda-Forge approach.

For instance, decisions about application binary interface pins are usually made by core, although these changes have impacts on downstream maintainers. It is possible that most maintainers don't know about what these pins are, how they are changed and how that affects their feedstocks.

Automation​

Automation has been used to great effect to make Conda-Forge possible. The various bots and web services enable Conda-Forge's current scale, providing help and support from running builds, bumping versions, and checking feedstock quality. However, this automation presents its own operational risks and magnifies existing operational risks.

Automation has a tendency to fail when we least expect it and often we lack the ability to fix it. The January 2018 Travis-CI outage is a great example of this, where the CI service we were using for macOS builds experienced reduced capacity and then a complete outage, causing builds to queue for days. Recently there was a sudden decrease in the number of parallel builds on Azure causing a similar queue of builds. Automation can cause issues by enabling users to make decisions without all the needed information. While many feedstocks have effective smoke tests for their packages the autotick bot doesn't currently check for new dependencies, potentially leading to missing or incorrect package metadata.

Conclusion​

Overall Conda-Forge has managed its operational risk well. Most importantly Conda-Forge's transparent open source nature allows us to address these issues head on by engaging with the community.

Supported platforms are,

• Linux-x86_64 (glibc 2.12 or newer)
• OSX-x86_64 (OSX 10.9 or newer)
• Linux-aarch64 (glibc 2.17 or newer)
• Linux-ppc64le (glibc 2.17 or newer)

How to use PyPy​

To use the PyPy builds you can do the following,

conda config --set channel_priority strict
conda create -n pypy pypy
conda activate pypy

If you don't have a conda installation already, you can use miniforge-pypy3 which gives you a conda installation powered by pypy itself.

However as of the writing of this post, not many conda packages can be installed into this environment, but noarch packages which do not depend on the python version nor the interpreter can be installed. For eg, mpmath is a noarch package without any dependencies.

conda install mpmath    # succeeds
conda install numpy     # fails as of March 10, 2020

UPDATE: numpy and scipy builds are working as of April 10, 2020.

All python C extensions needs to be rebuilt for the PyPy ABI. This is currently on the way and can be tracked at the status page.

python_abi Package​

As part of adding support for PyPy and to keep the older python builds working, a python_abi package was added. This defines the abi for the python package and any non-noarch python packages will have a dependency on this package. Older python downstream packages like numpy had their metadata patched to add a CPython ABI. You can ask for a specific python ABI.

conda install "python_abi=*=*_cp27mu"

If you are using python packages from packages other than defaults, you will be able to install python extensions built with CPython ABI into PyPy builds as their metadata have not been patched. Solution in this case is to hotfix the metadata which is available to only high volume conda channel or to rebuild those packages with the new python packages and mark the older ones as broken.

Using the newer packages will rerender the following requirement to add a dependency on python_abi. For eg,

requirements:
  host:
    - python 3.6
  run:
    - python

is rendered as,

requirements:
  host:
    - python 3.6.10 h9d8adfe_1009_cpython
  run:
    - python >=3.6,<3.7.0a0
    - python_abi 3.6 *_cp36m

whereas,

requirements:
  host:
    - python 3.6 *_73_pypy
  run:
    - python

is rendered as,

requirements:
  host:
    - python 3.6.9 0_73_pypy
  run:
    - python >=3.6,<3.7.0a0
    - python_abi 3.6 *_pypy36_pp73

Note that the PyPy ABI tag has pp73 at the end which indicates that the ABI is stable only for PyPy3.6 7.3.x series.

This opens up the possibility of adding debug builds of python and building extension packages using the Python Debug ABI.

Discussion on the PyPy builds can be found in the issue conda-forge/conda-forge.github.io#867.

Introduction​

All jokes aside, the new project grayskull was created with the intention of generating better Conda recipes that would allow to package properly projects available in different channels such as PyPI, CRAN, Conan, GitHub register, GitHub repositories and so on. On top of that, Grayskull is also being developed to help conda-forge to update recipes.

Current status​

Currently, Grayskull (version 0.2.1) is able to generate recipes just looking for packages on PyPI, and it is available on PyPI and conda-forge. The GitHub repository for this package is: marcelotrevisani/grayskull.

Before Grayskull, we just had conda-build skeleton to generate recipes for Python packages on PyPI. In all other aspects, the difference of quality of the generated recipes, and also the time spent to generate them have a big discrepancy when compared to conda-build skeleton and grayskull. Grayskull generates recipes taking in consideration the platform, Python version available, selectors, compilers (Fortran, C and C++), packages constrains, license type, license file, and so forth. It uses metadata available from multiple sources to try to create the best recipe possible.

Installation​

You can install grayskull using pip or conda. Grayskull does not rely on conda to run and can generate recipes with minimum dependencies.

With conda​

Grayskull is available on the conda-forge channel.

conda install -c conda-forge grayskull

With pip​

pip install grayskull

Grayskull vs conda-build skeleton​

There are some differences of recipes generated by grayskull and conda skeleton. Taking as example the pytest recipe, which has selectors for platforms, Python version constrains, and has several package constrains as well.

Grayskull (0.2.1) - took 4 seconds to generate the recipe​

{% set name = "pytest" %}
{% set version = "5.3.5" %}

package:
  name: {{ name|lower }}
  version: {{ version }}

source:
  url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/{{ name }}-{{ version }}.tar.gz
  sha256: 0d5fe9189a148acc3c3eb2ac8e1ac0742cb7618c084f3d228baaec0c254b318d

build:
  number: 0
  skip: true   # [py2k]
  entry_points:
    - pytest=pytest:main
    - py.test=pytest:main
  script: {{ PYTHON }} -m pip install . -vv

requirements:
  host:
    - pip
    - python
    - setuptools >=40.0
    - setuptools_scm
  run:
    - atomicwrites >=1.0      # [win]
    - attrs >=17.4.0
    - colorama      # [win]
    - importlib-metadata >=0.12        # [py<38]
    - more-itertools >=4.0.0
    - packaging
    - pathlib2 >=2.2.0        # [py<36]
    - pluggy <1.0,>=0.12
    - py >=1.5.0
    - python
    - wcwidth

test:
  imports:
    - pytest
  commands:
    - pip check
    - pytest --help
    - py.test --help
  requires:
    - pip

about:
  home: https://pypi.org/project/pytest/
  summary: 'pytest: simple powerful testing with Python'
  dev_url: https://github.com/pytest-dev/pytest
  license: MIT
  license_file: LICENSE

extra:
  recipe-maintainers:
    - marcelotrevisani

Skeleton (3.18.11) - took 31 seconds to generate the recipe​

{% set name = "pytest" %}
{% set version = "5.3.5" %}

package:
  name: "{{ name|lower }}"
  version: "{{ version }}"

source:
  url: "https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/{{ name }}-{{ version }}.tar.gz"
  sha256: 0d5fe9189a148acc3c3eb2ac8e1ac0742cb7618c084f3d228baaec0c254b318d

build:
  number: 0
  script: "{{ PYTHON }} -m pip install . -vv"

requirements:
  host:
    - atomicwrites >=1.0
    - attrs >=17.4.0
    - colorama;sys_platform =="win32"
    - importlib-metadata >=0.12
    - more-itertools >=4.0.0
    - packaging
    - pathlib2 >=2.2.0
    - pip
    - pluggy >=0.12,<1.0
    - py >=1.5.0
    - python
    - wcwidth
  run:
    - atomicwrites >=1.0
    - attrs >=17.4.0
    - colorama;sys_platform =="win32"
    - importlib-metadata >=0.12
    - more-itertools >=4.0.0
    - packaging
    - pathlib2 >=2.2.0
    - pluggy >=0.12,<1.0
    - py >=1.5.0
    - python
    - wcwidth

about:
  home: The package home page
  license: MIT
  license_family: MIT
  license_file:
  summary: "pytest: simple powerful testing with Python"
  doc_url:
  dev_url:

extra:
  recipe-maintainers:
    - your-github-id-here

Original recipe on conda-forge for pytest 5.3.5​

{% set version = "5.3.5" %}

package:
  name: pytest
  version: {{ version }}

source:
  url: https://pypi.io/packages/source/p/pytest/pytest-{{ version }}.tar.gz
  sha256: 0d5fe9189a148acc3c3eb2ac8e1ac0742cb7618c084f3d228baaec0c254b318d

build:
  skip: True  # [py27]
  number: 1
  script: "{{ PYTHON }} setup.py install --single-version-externally-managed --record record.txt"
  entry_points:
    - py.test = py.test:main
    - pytest = py.test:main

requirements:
  host:
    - pip
    - python
    - setuptools >=40.0
    - setuptools_scm
  run:
    - atomicwrites >=1.0  # [win]
    - attrs >=17.4.0
    - colorama  # [win]
    - importlib_metadata >=0.12  # [py<38]
    - more-itertools >=4.0
    - packaging
    - pathlib2 >=2.2.0  # [py<36]
    - pluggy >=0.12,<1.0
    - py >=1.5.0
    - python
    - setuptools >=40.0
    - wcwidth
  run_constrained:
    # pytest-faulthandler 2 is a dummy package.
    # if an older version of fault-handler is installed, it will conflict with pytest >=5.
    - pytest-faulthandler >=2

test:
  commands:
    - pytest -h
  imports:
    - pytest

about:
  home: https://docs.pytest.org/en/latest/
  license: MIT
  license_file: LICENSE
  summary: 'Simple and powerful testing with Python.'
  description: |
    The pytest framework makes it easy to write small tests, yet scales to
    support complex functional testing for applications and libraries.
    doc_url: https://docs.pytest.org/en/latest/
    dev_url: https://github.com/pytest-dev/pytest/

extra:
  recipe-maintainers:
    - flub
    - goanpeca
    - nicoddemus
    - ocefpaf
    - mingwandroid

Major differences​

In the case of noarch: python, Grayskull is smart enough to detect when the recipe supports it, which is not done by Skeleton. It is important to highlight that Skeleton does not detect compilers as well. Nevertheless, Grayskull always try to detect it.

Usage Grayskull (0.2.1)​

Project options:

$ grayskull --help
usage: grayskull [-h] [--version] {pypi} ...

Grayskull - Conda recipe generator

positional arguments:
{pypi}         Options to generate PyPI recipes
pypi         Generate recipes based on PyPI

optional arguments:
-h, --help     show this help message and exit
--version, -v  Print Grayskull version and exit

$ grayskull pypi --help
usage: grayskull pypi [-h] [--maintainers MAINTAINERS [MAINTAINERS...]]
[--output OUTPUT]
pypi_packages [pypi_packages ...]

positional arguments:
pypi_packages         Specify the PyPI package name.

optional arguments:
-h, --help            show this help message and exit
--maintainers MAINTAINERS [MAINTAINERS ...], -m MAINTAINERS [MAINTAINERS...] List of maintainers which will be added to the recipe.
--output OUTPUT, -o OUTPUT Path to where the recipe will be created

To generate the recipe you can just call grayskull and pass the channel (as for now we are just supporting PyPI, it should be pypi) and the package name. You should also specify an output folder using the option --output or -o and it will create the package folder, and the recipe in there. It is important to note that the user can specify a list of maintainers which will be added to the recipe using the option --maintainers.

Example for pytest:

If you need to specify the package version you can do it just puting the equal sign after the package name and the version just right after that. Example:

grayskull pypi requests=2.21.0

or

grayskull pypi requests==2.21.0

If you want to generate multiple recipes just pass a list of packages, such as:

grayskul pypi pytest requests=2.21.0 colorama

Future plans​

• For the next major version (1.0.0) it is planned to add the functionality to be able to load the recipe and update just parts of it;
• Generate Conda recipes using CRAN (R) channel (2.0.0);
• Generate Conda recipes using Conan (C++) channel (3.0.0);

Issues​

Any problem, question, suggestions please feel free to open an issue on the repository.

Contributions are very welcome! :)

This work was possible thanks to the NumFOCUS Small Development Grant program.

We here at conda-forge have a large number of potential Google Sumer of Code projects around maintenance and development of the autotick bot infrastructure. These projects are high-impact, affecting the entire conda-forge ecosystem. They also cover multiple systems including databases, conda's CDN provider, continuous integration providers, and user interactions on GitHub.

Want to be a part of the team? Great! Take a look at the projects below and get in touch with us on GitHub! You can check the GSoC label for a detailed listing of the issues that need work.

1. Maintenance and Refactoring

   We have a large backlog of maintenance and refactoring issues that are great for people with a range of experience from beginners to true code ninjas.

   • Integration Testing for the Autotick Bot

     Run true integration testing on a copy of the graph to better test code changes and improve our CI process.

     Issue: regro/cf-scripts261

     Experience Level: advanced

   • Work on the "code hardening" Milestone

     Address any of the issues in the milestone above related to code refactoring and cleanup.

     Issues: regro/cf-scripts milestone #4

     Experience Level: beginner to advanced

2. Automated PRs for Globally Pinned Packages

   conda-forge maintains a list of globally pinned packages. These are typically dependencies whose version needs to be the same across all of conda-forge (e.g., the compiler versions or packages like HDF5). While we have infrastructure to run a migration of the downstream packages from a given pinned package, we do not have automated infrastructure to propose the migration of the pin itself. The project here is to add this functionality to our infrastructure.

   Issue: regro/cf-scripts#665

   Experience Level: advanced

3. Check the conda CDN for Updated Packages before Issuing PRs in a Migration

   conda relies on a CDN provider to serve the index of available packages. There is a moderate ~30 minute delay between when a package is uploaded to anaconda.org and when it will appear in the conda index. We currently do not take this delay into account when issuing PRs in a migration.

   Issue: regro/cf-scripts#595

   Experience Level: beginner

4. Finish Migrations with PR into the conda-forge Pinnings File

   Right now, when the migration of say package ABC to version X from version Y is done, we do not automatically merge the change in the globally pinned value of ABC into our listing of global pinnings. We should be issuing a PR to the pinnings file once we have determined that a suitable fraction of the packages effected by a migration have been properly rebuilt.

   Issue: regro/cf-scripts#595

   Experience Level: moderate

5. Fully Render conda Packages in order to Determine Migration Dependencies

   Determining the dependencies of a given package is actually a computation expensive task due to the way conda recipes are structured and parametrized through the use of Jinja2 and conda-build-config.yaml files. Currently, the autotick bot examines the static metadata in the meta.yaml file and not the fully rendered metadata. For this reason, we sometimes miss dependencies of a given package that need to be migrated first. Addressing this issue involves both calling the rendering process and also scaling that process to the entire set of conda-forge packages.

   Issue: regro/cf-scripts#664

   Experience Level: moderate

Conda-Forge handles this by having a pinning file that tracks all the currently supported ABIs. These pinned ABIs are then used to build the downstream packages, making certain that all are consistent. As new versions of pinned software are released the pins are updated, causing a migration of the pin, and the rebuilding of all packages which rely on the pinned package. In the past, this was handled by a change to the global pinnings and a subsequent migration via the auto-tick bot. While this worked, there were issues that this created. Firstly, this approach could cause unsatisfiable build dependencies for new packages, as some of the new package's dependencies had been compiled with the new pins, but not all. Secondly, migrations happened in series, if a second pin was moved while the first was being migrated then the migration could go wrong as packages which were being rebuilt for the first pin got the second pin before they were ready.

Conda-Forge Core has recently approved CFEP-9, a migration policy to fix these issues. With CFEP-9 pinnings are proposed as small yaml snippets which contain the new pins. The auto-tick bot then starts migrating the packages in order, applying the yaml snippet to each package in turn. If a second pinning change is issued then the bot starts the migration for that package too, enabling the two migrations to work independently. If a package needs a change in both pins then the maintainers can choose the order in which they apply the pins by merging one pin before the other.

This approach will yield much greater stability in migrations and will enable more maintainers to issue migrations. Migrations can be issued by putting a PR into conda-forge/conda-forge-pinning-feedstock, adding a file to the migrations folder, PRs into the auto-tick bot are not needed anymore.