2020-08-19 conda-forge core meeting

Zoom link What time is the meeting in my time zone last weeks meeting

Attendees

Agenda

Standing items

  • [x] intros for new folks on the call

  • [x] (CJ) budget

Your new agenda items

  • [ ] (ED) @sylvain: Any updates from OVH on the windows VM?

  • [ ] (AS) qgpu - GPU build agents.

    • Drone or Azure? Drone is a simple go executable and you can run it in docker. Azure build agent is heavy weight?

    • Pick one and go

  • [x] (CJ) Version bumps offering dependency analysis as hints

    • Hinting system for bot for what we think the dependencies should be based on the analysis of the source code

    • Currently using depfinder. Only works for Python.

    • Around 6000 packages have been analyzed

    • For 30% of the packages depfinder and CF metadata agree

    • https://github.com/regro/cf-scripts/pull/1126

    • https://github.com/regro/cf-graph-countyfair/tree/master/audits/depfinder

    • https://github.com/regro/cf-graph-countyfair/blob/master/audits/depfinder/_net_audit.json

    • Can we do this for C packages?

      • Post-build steps do something to the DSOs that are created. Can’t do it beforehand. You can check after the build.

      • (CJ) Can the C builds publish this information?

      • (JJ) Maybe?

      • (FF) C builds will fail if the dependency isn’t there at build-time. Python won’t.

    • Can we do this for R?

      • Get metadata from CRAN and update

      • Use skeleton to get the R dependencies. Grayskull doesn’t handle R recipes yet.

    • Number of packages: 6k is Python, 2k is R. Between these two we’ll have 80% of the ecosystem covered.

    • (MB) How should we handle information loss? Optional dependencies - maybe capture as a comment in the meta.yaml? Version bounds on the dependencies?

      • Optional dependencies - capture in extra section of info

    • (CJ, addendum) Mapping between conda-forge, pypi and imports: https://github.com/regro/cf-graph-countyfair/blob/master/mappings/pypi/name_mapping.yaml

      • Note that the mapping is imperfect, since it relies on the conda-forge test: imports: metadata.

      • It would be nice if we could get this directly from the package/source

  • [x] (KK) cudatoolkit package in conda-forge

    • Still only ship pieces that are redistributable per the EULA (shared libraries)

    • nvbug #: 3052604

      • Internal NVIDIA system used for tracking these types of approvals

      • Link (only works on NVIDIA intranet): http://nvbugs/3052604/

    • (KK) Approval to host the same version of cudatoolkit in conda-forge as is on defaults

      • Maybe don’t just copy the recipe from defaults. Or at least revisit

      • Would like to have a few Nvidia folks maintain the recipe. Over time migrate it to the CUDA team at Nvidia.

      • (JK) Maybe use variants to maintain all of the versions in one branch

    • (JJ) What about cudnn? Can we move that over to CF?

    • (KK) All of the cuda libs should be shippable from CF. As long as we can show the EULA with pre-link or post-link. Internal at NVIDIA is fine with just having this as a pre-link script messaging mechanism.

    • (IF) Have you considered splitting up the recipe where all the different libraries end up in different conda installable units?

      • (KK) Yes that’s the long term plan. Were trying to do the windows side. Their team is mostly focused on Linux.

      • (JJ) If we do end up doing Windows, then make sure we have all of the windows versions. Strict channel priority is harmful if you have only one or two packages available versus default having many.

  • [x] (CHL) FYI: conda 4.8.4 behavior change — virtual package constraints now enforced

    • Will definitely impact many CUDA packages (e.g., can no longer install CUDA 10-dependent packages on CUDA 9.x systems); e.g., https://github.com/conda/conda/issues/10152

    • Will be working on solver messaging because errors are usually opaque and irrelevant

    • There’s an env var that you can set to change conda’s view on the cuda version: CONDA_OVERRIDE_CUDA

Stuff from last week that we didn’t get to

Who is taking these action items?

  • [x] Dropping python 3.6

    • need an announcement cycle

    • should we follow NEP29? NEP29 + 6 months?

      • https://numpy.org/neps/nep-0029-deprecation_policy.html#drop-schedule

    • End of life for Python 3.x versions:

      • https://devguide.python.org/#status-of-python-branches

    • No pypy for 3.7

      • https://foss.heptapod.net/pypy/pypy/-/wikis/py3.7%20status

    • Action Item: Send to issue (get input from pypy team and others)

    • (CJ) py36 should stick around until pypy comes out (it’s in the near horizon). That’s going to be soon, so it’s not like we’re keeping

    • TODO: (ED) Python versions:

      • Keep 3 Python versions

      • Move off of old versions with the community moves (when scipy, matplotlib, numpy, etc.)

      • We can keep an old version around temporarily if we need to (e.g., pypy doesnt have py37 yet)

Active votes

Subteam updates

Bot

ARM

POWER

CUDA

Docs

staged-recipes

website

security+systems

CI infrastructure

Compiler upgrade

CFEP updates

Open PRs

  • cfep-04 X11 and CDT policy

    • INACTIVE - Merge in with some inactive-esque status?

    • Needs new champion. Thanks for your work on this pkgw! Has unaddressed comments from pkgw as from Jan 10, 2020

  • cfep-06 Staged-recipes review lifecycle

    • INACTIVE - Merge in with some inactive-esque status?

    • Lingering comment from @saraedum. @jakirkham, can you reply? Has unadressed comment from @saraedum from Jan 8, 2020

    • (MRB) The stalebot has solved the worst of the issues here. I think we could defer this one permanently.

  • cfep-10 Feedstock statuses, unmaintained

    • INACTIVE - Merge in with some inactive-esque status?

    • Needs another review. Has unaddressed updates from pkgw as of Jan 11, 2020

  • cfep-12 Removing packages that violate the terms of the source package

    • Stalled since May 26, 2020

    • Active debate about moving to “broken” vs deleting from conda-forge channel

    • Active vote, ends on 2020-03-11

    • What were the results of the vote?

    • Did we hear back from NumFOCUS?

  • cfep-17 Handling pin backports and dependency rebuilds

    • Stalled debate about implementation details between Isuru, CJ and Matt

    • UPDATE 2020-07-22: We in principle have agreement to render the extra pinnings needed directly in the feedstock on a temporary basis (i.e., until the migration has ended).

Discussion

Check in on previous action items

Copy previous action items from last meeting agenda.

This meeting

Last meeting

2 meetings ago

  • [ ] Figure out how to communicate breaking changes to users. Likely should open up an issue immediately for futher discussion. Ping @kkraus, plus capture notes from further up in these meeting notes

  • [ ] (Eric) TODO: Make strict an option in conda_forge.yaml and turn it on by default. Open issue in conda-smithy

3 meetings ago

  • [ ] Eric to add a new page to our docs around how to engage with conda-forge and affiliated in a commercial relationship.

  • [x] Eric will get the NVBug link from Keith and archive it in the conda-forge google drive.

  • [ ] John K. will update the cuda toolkit feedstock on the git repo to note the NVBug link to the internal NVIDIA issue tracker

  • [ ] Jonathan will update docs to note that some non-exhaustive list of packages (like cuda-toolkit, MKL, etc.)

  • [ ] Jonathan will review this PR

Move to Issue Tracker

  • [ ] (Kale) schedule conda working group

  • [ ] cfep-10 next steps: CJ to call a vote for feedback

  • [ ] cfep-06 next steps: Ask staged recipes team to champion this CFEP and move it forward

  • [ ] jakirkham & CJ-wright to sync on adding CUDA to the migration bot

  • [ ] (Eric) Scheduling Anaconda <-> conda-forge sync on anaconda.org requirements gathering

    • Will try and get this scheduled in the next month.

  • [ ] (Anthony) Reach out to NumFocus to figure out legal ramifications of not including licenses in files.

  • [ ] (Eric) check internally for funding levels for hotels & flying folks from the community in?

  • [ ] (Eric) Figure out finances of conda-forge to support themselves?

  • [ ] (jjhelmus) Open up CFEP for which python’s we’re going to support

  • [ ] (jakirkham) write a blog post on CUDA stuff we discussed today

  • [ ] (jakirkham) update docs on how to add CUDA support to feedstocks

  • [ ] (jakirkham) will open an issue on conda-smithy to investigate Drone issues. (ping the aarch team)

    • https://github.com/conda-forge/conda-forge.github.io/issues/954

  • [ ] (ED) Who we are page? Some combination of a FAQ and a who is everyone. FAQ things like:

    • who’s the POC for CF <> Anaconda, CF <> NumFocus, CF <> Azure

    • who’s the POC for the various subteams?

    • Informal information: roles, day jobs, bios, the whole nine yards, why you’re here, etc.

    • Public or internal? I don’t really care either way. Anyone feel strongly one way or the other?

    • opt-in to public bios

    • software carpentry has a large number of instructors and has https://carpentries.org/instructors

    • some concern about “yet another place to keep stuff up to date”

  • [ ] (CJ) Form finance subteam

  • [ ] (ED) document strategies for reproducible environments using conda-forge

  • [ ] (UK) Static libraries stuff

    • [ ] Add linting hints to builds to find them

    • [x] Recommend how to package them -> CFEP-18

    • [x] We should write docs saying we don’t provide support and this is a bad idea. -> CFEP-18