# 2016-09-09: General discussion Time: 14:00 UTC Hangout link: [](https://hangouts.google.com/call/v5olhwzpfzgzpoq5i3wthjpqpie)[https://hangouts.google.com/call/v5olhwzpfzgzpoq5i3wthjpqpie](https://hangouts.google.com/call/v5olhwzpfzgzpoq5i3wthjpqpie) **Attendees** * Jonathan Helmus, Filipe, Michael, Ray, [Eric Dill](https://conda-forge.hackpad.com/ep/profile/yJqDqpPqJyz), Björn Grüning, [Matt Craig](https://conda-forge.hackpad.com/ep/profile/yBvjHx0Ad3Y) (late) **Standing items** * How many repos? ~1100 * How many contributors? ~220 * New core devs? **Notes** Bioconda updates: * Rebuilding binaries for the conda-build 2.0 when the source tarballs that disappear. Bioconda is arching the sources. * Automate process to archive source tarball and test in a container (nice as a service to create a bundle-container to run packages). Core Devs * Eric Dill (invite) * Peter M. Landwehr (already invited) Split builds * conda-build issue (xref?) [conda/conda build#1338](https://github.com/conda/conda-build/issues/1338) * continuum compiler toolchain to use gcc (Linux), clang and gfortran (OS X) consistently. Pre-releases/RC * Needs a champion to write a proposal! * Eric Dill will take this on. Hopefully a CFEP will land within one week, 2016-09-16 * Eric suggests having both dev and -dev labels. The former is for "cutting edge people" but the latter is people who only want to be testing/using the new version of only one thing (plus any dependencies). * dev is a bad name. These packages are more for testing than for development. Testing? RC? * Filipe thinks we should not accept versions earlier than RC (Not really do not accept but encourage people to call their dev version a RC. The thinking is that conda-forge is a place to release binaries and nightly testing builds, for example, are beyond the scope IMO.) The feather-feedstock maintainers question: * They want to build Python 2.7 with a modern Visual Studio and conda-forge should suggest to them that this will create a different ecosystem that is compatible with conda-forge. conda-build 2.0 and conda-build-all. Mike asked if we are ready to use conda-build 2.0. conda-forge needs to check: * where are the pins to conda-build <2.0 * check conda-inspect * check the upload script * check conda-smithy Use pip in the build script. * On Windows need conda > 4.2 * Need to check if the entry_points must be declared or not in the recipe. **Agenda** * Next meeting: can we do 2016-09-16? * Update from the bioconda community. Tarball archiving and automatic Container (Docker, rkt) builds. * Archives: [bioconda/bioconda recipes#2194](https://github.com/bioconda/bioconda-recipes/issues/2194) * Container: [bioconda/bioconda recipes#2297](https://github.com/bioconda/bioconda-recipes/issues/2297) * Is conda-forge interested in a similar integration? * OSX - getting back to a usable, coherent, stack * libc++ (clang) vs libstdc++ (gcc/g++) * Apple's Blocks extension to C (these are like lambdas) isn't in recent (or non-Apple) GCC: [](https://gcc.gnu.org/ml/gcc/2009-09/msg00264.html)[https://gcc.gnu.org/ml/gcc/2009-09/msg00264.html](https://gcc.gnu.org/ml/gcc/2009-09/msg00264.html) * Can we link gfortran and LLVM system/c++ libraries together w/o violating GPL w/runtime exception (compiler_rt + libc++) - not if link is done statically to the best of my knowledge, and also can gfortran be built on top of compiler_rt? These are big unknowns. * Minimum OSX required for clang (10.8, I think?) * Actually clang is usable beginning in 10.7. So, this would be viable given your compatibility constraints. * Also, all the refs I have seen suggest that this will still have C++11 support. * Compatibility with defaults (built on 10.7, uses gcc) - where will people break? I think only if mixing packages - how do we assure that we have all the ones we need? * Metadata unification with Continuum - are we OK with adding some fields to about section to match Anaconda standard? * example at [](https://github.com/ContinuumIO/anaconda-recipes/blob/master/colander/meta.yaml)[https://github.com/ContinuumIO/anaconda-recipes/blob/master/colander/meta.yaml](https://github.com/ContinuumIO/anaconda-recipes/blob/master/colander/meta.yaml) * license_family * doc_url * dev_url * constrain summary to 80 chars (longer stuff use description) * Can we add this to linter, and add to recipes as we update them? * What support for unicode should we have? Any? Summary/description only? * CUDA/cuDNN update * Improving infrastructure * Better workflows with staged-recipes * Fast finish AppVeyor on merge ( [conda forge/staged recipes#1142](https://github.com/conda-forge/staged-recipes/pull/1142) ) * Drop Travis CI matrix ( [conda forge/staged recipes#1234](https://github.com/conda-forge/staged-recipes/pull/1234) ) * Use CircleCI for feedstock generation ( [conda forge/staged recipes#916](https://github.com/conda-forge/staged-recipes/issues/916) ) * Keeping recipes out of PRs ( [conda forge/staged recipes#942](https://github.com/conda-forge/staged-recipes/issues/942) ) * Bank work in partial conversion ( [conda forge/staged recipes#915](https://github.com/conda-forge/staged-recipes/issues/915) ) * Notifications (how do we stay on top of them) * MSYS2 * Available on defaults - was in conda 4.1.7, but that was pulled. Coming in 4.1.8. * Discussing Ray Donnelly's work on MSYS2 packages and how we want to use and integrate these into conda-forge. * Some use cases to consider OpenBLAS, FFTW, build tools, others? * Binary data * Do we include it in recipes? * What kinds do we allow if any (e.g. icons)? * How do we verify the licensing? * How do we verify that they are safe? * Dev releases: Where do they happen? * Do we do them at conda-forge? * Maybe add a label. * Do we let others do them with a feedstock on their own repo? * How do we enforce whatever we decide? * Channel mirroring * Can this point be a little bit explained? I thought about this as well and would like to contribute to this point. * Eric Dill has put together a script for copying a package from one channel to another here: [conda forge/conda forge.github.io#134](https://github.com/conda-forge/conda-forge.github.io/pull/134) * I have a really, really crude script that copies all of the packages in one channel to another that I just put at: [](https://gist.github.com/mwcraig/8473cf840f6d29236d6d8af699404555)[https://gist.github.com/mwcraig/8473cf840f6d29236d6d8af699404555](https://gist.github.com/mwcraig/8473cf840f6d29236d6d8af699404555) * conda-build-all can copy from one channel to another: `conda build-all --inspect-channels conda-forge --upload-channels astropy some_packge_recipe` will copy the `some_package` from the channel conda-forge to astropy if it can, or build it if it doesn't exist on conda-forge. Discussion about what the desired behavior should be has started at: [SciTools/conda build all#46](https://github.com/SciTools/conda-build-all/issues/46) * Feedstock history * Is it sacred? * Do we rebase/force push? * If so, under what conditions? * How do we avoid multiple people doing this simultaneously? * I don't think you can. * IMHO, if it's just one author in staged recipes, sure. If feedstock, no force push - only to PRs to feedstock. If people don't mind merge PRs, it sure is a lot simpler to not rebase. I have messed up rebasing a few times recently... =( * Drop numpy 1.10 and reduce our build matrix. (Numba now works with numpy 1.11.) * Feedstocks philosophy: Explicit vs implicit / reproducible vs redundant * Signing packages * Should be easy to do. ( [](http://conda.pydata.org/docs/signed-packages.html)[http://conda.pydata.org/docs/signed-packages.html](http://conda.pydata.org/docs/signed-packages.html) ) * There has been some interest previously. * HTTPError: 503 Server Error: Service Unavailable: Back-end server is at capacity for url... * Seems we are regularly running into this issue under normal usage conditions. * Had discussed previously caching packages on AppVeyor and trying to reuse those to start. * Maybe we need to consider caching on all CIs. * Building our own Miniconda-like self-extracting scripts with packages via [`constructor`](https://github.com/conda/constructor). * There have been improvements on Continuum's side that should help this. In short, repodata (the package index for a given channel) was being generated for each anaconda.org query. This was unnecessarily high cost, and some caching schemes have been implemented. * Handling removal of unpinned/improperly pinned packages. * Has been done manually thus far. * This doesn't scale well though. * Should we (semi) automate removal? * Should we hot-fix broken packages? ( [conda forge/conda forge.github.io#170](https://github.com/conda-forge/conda-forge.github.io/pull/170) ) * Should we label them as broken * Not currently buildable packages * In particular open source code that is out of scope for CIs. * Examples include Qt4, Qt5, possibly PyQt4, possibly PyQt5, gcc, VTK, etc. * How do we indicate they are built manually? * Are we ok with uploading non-built binaries? * When do we determine something is ok to be built manually? * What procedures should people follow for building manually? * Use a standard build docker image, VM, or vagrant file * Sign package? * Implement reproducible builds where feasible (linux) * [](https://reproducible-builds.org/)[https://reproducible-builds.org/](https://reproducible-builds.org/) * What changes do we need to make in conda-smithy elsewhere? * What other build infrastructure could we utilize? * Would be nice to provide some volunteer builder abstraction, so that we could have an elastic worker farm that would be somewhat resilient. * Standardizing build images is probably (relatively) easy - how to orchestrate, though? * [Staged Releases](https://conda-forge.hackpad.com/DZNKZdgiMbF) * Windows BLAS Solutions * Still don't have a BLAS for Windows yet need something. * Don't build a BLAS * NumPy has a small subset of BLAS functionality. * Not sure what to do with SciPy (unable to find Windows wheels for them either). * Build OpenBLAS with C support only. * Will be pretty slow. * Should work on all Pythons. * Build OpenBLAS with MinGW compilers. * Works with Python 2.7 and 3.4. * Won't work with Python 3.5? * Reuse something like R's BLAS. * Is there a package for something like this? * Will it have the same issues with Python 3.5? * ATLAS?