Python 3.9 Compatibility

[amirhessam88]

Can we get an update for Python version 3.9 as well?

[jli]

I'd also love a 3.9 wheel!

fwiw i was able to build my own wheel on 3.9 after apt-get install build-essential gfortran

[kalashnov]

I had no luck building wheels for 3.9 (also I do not have root privileges)/ I run in the following error:

  ERROR: Command errored out with exit status 1:
   command: /zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-j1g6z3bg/glmnet_2a4b2922fe5a4660b6e4a169242c965e/setup.py'"'"'; __file__='"'"'/tmp/pip-install-j1g6z3bg/glmnet_2a4b2922fe5a4660b6e4a169242c965e/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-djq55rki
       cwd: /tmp/pip-install-j1g6z3bg/glmnet_2a4b2922fe5a4660b6e4a169242c965e/
  Complete output (343 lines):
  /tmp/pip-install-j1g6z3bg/glmnet_2a4b2922fe5a4660b6e4a169242c965e/setup.py:14: DeprecationWarning:
  
    `numpy.distutils` is deprecated since NumPy 1.23.0, as a result
    of the deprecation of `distutils` itself. It will be removed for
    Python >= 3.12. For older Python versions it will remain present.
    It is recommended to use `setuptools < 60.0` for those Python versions.
    For more details, see:
      https://numpy.org/devdocs/reference/distutils_status_migration.html
  
  
    from numpy.distutils.core import Extension, setup

<...>


    INFO: compile options: '-I/zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/include -I/home/users/georgyk/miniconda3/include/python3.9 -c'
    extra options: '-msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512vl -mavx512bw -mavx512dq -Werror'
    WARN: CCompilerOpt.dist_test[630] : CCompilerOpt._dist_test_spawn[764] : Command (gcc -pthread -B /home/users/georgyk/miniconda3/compiler_compat -Wl,--sysroot=/ -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC -I/zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/include -I/home/users/georgyk/miniconda3/include/python3.9 -c /zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/lib/python3.9/site-packages/numpy/distutils/checks/cpu_avx512_clx.c -o /tmp/tmpfppa577g/zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/lib/python3.9/site-packages/numpy/distutils/checks/cpu_avx512_clx.o -MMD -MF /tmp/tmpfppa577g/zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/lib/python3.9/site-packages/numpy/distutils/checks/cpu_avx512_clx.o.d -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512vl -mavx512bw -mavx512dq -Werror) failed with exit status 1 output ->
    /zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/lib/python3.9/site-packages/numpy/distutils/checks/cpu_avx512_clx.c: In function ‘main’:
    /zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/lib/python3.9/site-packages/numpy/distutils/checks/cpu_avx512_clx.c:20:17: error: implicit declaration of function ‘_mm512_dpbusd_epi32’; did you mean ‘_mm512_4dpwssd_epi32’? [-Werror=implicit-function-declaration]
                 a = _mm512_dpbusd_epi32(a, _mm512_setzero_si512(), a);
                     ^~~~~~~~~~~~~~~~~~~
                     _mm512_4dpwssd_epi32
    /zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/lib/python3.9/site-packages/numpy/distutils/checks/cpu_avx512_clx.c:20:15: error: incompatible types when assigning to type ‘__m512i {aka __vector(8) long long int}’ from type ‘int’
                 a = _mm512_dpbusd_epi32(a, _mm512_setzero_si512(), a);
                   ^
    cc1: all warnings being treated as errors
    
    WARN: CCompilerOpt.feature_test[1563] : testing failed
    INFO: CCompilerOpt.feature_test[1547] : testing feature 'AVX512_SKX' with flags (-msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512vl -mavx512bw -mavx512dq)
    INFO: C compiler: gcc -pthread -B /home/users/georgyk/miniconda3/compiler_compat -Wl,--sysroot=/ -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC
    
    INFO: compile options: '-I/zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/include -I/home/users/georgyk/miniconda3/include/python3.9 -c'
    extra options: '-msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512vl -mavx512bw -mavx512dq -Werror'
    INFO: CCompilerOpt.cc_test_flags[1073] : testing flags (-mavx512er -mavx512pf)
    INFO: C compiler: gcc -pthread -B /home/users/georgyk/miniconda3/compiler_compat -Wl,--sysroot=/ -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC
    
    INFO: compile options: '-I/zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/include -I/home/users/georgyk/miniconda3/include/python3.9 -c'
    extra options: '-mavx512er -mavx512pf'
    INFO: CCompilerOpt.feature_test[1547] : testing feature 'AVX512_KNL' with flags (-msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512er -mavx512pf)
    INFO: C compiler: gcc -pthread -B /home/users/georgyk/miniconda3/compiler_compat -Wl,--sysroot=/ -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC
    
    INFO: compile options: '-I/zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/include -I/home/users/georgyk/miniconda3/include/python3.9 -c'
    extra options: '-msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512er -mavx512pf -Werror'
    INFO: CCompilerOpt.cc_test_flags[1073] : testing flags (-mavx512ifma -mavx512vbmi)
    INFO: C compiler: gcc -pthread -B /home/users/georgyk/miniconda3/compiler_compat -Wl,--sysroot=/ -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC
    
    INFO: compile options: '-I/zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/include -I/home/users/georgyk/miniconda3/include/python3.9 -c'
    extra options: '-mavx512ifma -mavx512vbmi'
    INFO: CCompilerOpt.feature_test[1547] : testing feature 'AVX512_CNL' with flags (-msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512vl -mavx512bw -mavx512dq -mavx512ifma -mavx512vbmi)
    INFO: C compiler: gcc -pthread -B /home/users/georgyk/miniconda3/compiler_compat -Wl,--sysroot=/ -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC
    
    INFO: compile options: '-I/zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/include -I/home/users/georgyk/miniconda3/include/python3.9 -c'
    extra options: '-msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512vl -mavx512bw -mavx512dq -mavx512ifma -mavx512vbmi -Werror'
    INFO: CCompilerOpt.cc_test_flags[1073] : testing flags (-mavx5124fmaps -mavx5124vnniw -mavx512vpopcntdq)
    INFO: C compiler: gcc -pthread -B /home/users/georgyk/miniconda3/compiler_compat -Wl,--sysroot=/ -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC
    
    INFO: compile options: '-I/zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/include -I/home/users/georgyk/miniconda3/include/python3.9 -c'
    extra options: '-mavx5124fmaps -mavx5124vnniw -mavx512vpopcntdq'
    INFO: CCompilerOpt.feature_test[1547] : testing feature 'AVX512_KNM' with flags (-msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512er -mavx512pf -mavx5124fmaps -mavx5124vnniw -mavx512vpopcntdq)
    INFO: C compiler: gcc -pthread -B /home/users/georgyk/miniconda3/compiler_compat -Wl,--sysroot=/ -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC
    
    INFO: compile options: '-I/zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/include -I/home/users/georgyk/miniconda3/include/python3.9 -c'
    extra options: '-msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512er -mavx512pf -mavx5124fmaps -mavx5124vnniw -mavx512vpopcntdq -Werror'
    INFO: CCompilerOpt.__init__[1803] : skip features (SSE SSE3 SSE2) since its part of baseline
    INFO: CCompilerOpt.__init__[1807] : initialize targets groups
    INFO: CCompilerOpt.__init__[1809] : parse target group simd_test
    INFO: CCompilerOpt._parse_target_tokens[2020] : skip targets (VXE XOP VSX3 VSX4 ASIMD NEON VXE2 VSX2 FMA4 VSX VX) not part of baseline or dispatch-able features
    INFO: CCompilerOpt._parse_policy_not_keepbase[2132] : skip baseline features (SSE2)
    INFO: CCompilerOpt.generate_dispatch_header[2353] : generate CPU dispatch header: (build/src.linux-x86_64-3.9/numpy/distutils/include/npy_cpu_dispatch_config.h)
    INFO: CCompilerOpt.feature_extra_checks[1627] : Testing extra checks for feature 'AVX512F' (AVX512F_REDUCE)
    INFO: C compiler: gcc -pthread -B /home/users/georgyk/miniconda3/compiler_compat -Wl,--sysroot=/ -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC
    
    INFO: compile options: '-I/zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/include -I/home/users/georgyk/miniconda3/include/python3.9 -c'
    extra options: '-msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -Werror'
    INFO: CCompilerOpt.feature_extra_checks[1627] : Testing extra checks for feature 'AVX512_SKX' (AVX512BW_MASK AVX512DQ_MASK)
    INFO: C compiler: gcc -pthread -B /home/users/georgyk/miniconda3/compiler_compat -Wl,--sysroot=/ -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC
    
    INFO: compile options: '-I/zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/include -I/home/users/georgyk/miniconda3/include/python3.9 -c'
    extra options: '-msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512vl -mavx512bw -mavx512dq -Werror'
    INFO: C compiler: gcc -pthread -B /home/users/georgyk/miniconda3/compiler_compat -Wl,--sysroot=/ -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC -O2 -isystem /home/users/georgyk/miniconda3/include -fPIC
    
    INFO: compile options: '-I/zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/include -I/home/users/georgyk/miniconda3/include/python3.9 -c'
    extra options: '-msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mno-mmx -mavx512cd -mavx512vl -mavx512bw -mavx512dq -Werror'
    INFO: get_default_fcompiler: matching types: '['arm', 'gnu95', 'intel', 'lahey', 'pg', 'nv', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor', 'fujitsu']'
    INFO: customize ArmFlangCompiler
    WARN: Could not locate executable armflang
    INFO: customize Gnu95FCompiler
    WARN: Could not locate executable gfortran
    WARN: Could not locate executable f95
    INFO: customize IntelFCompiler
    WARN: Could not locate executable ifort
    WARN: Could not locate executable ifc
    INFO: customize LaheyFCompiler
    WARN: Could not locate executable lf95
    INFO: customize PGroupFCompiler
    WARN: Could not locate executable pgfortran
    INFO: customize NVHPCFCompiler
    WARN: Could not locate executable nvfortran
    INFO: customize AbsoftFCompiler
    WARN: Could not locate executable f90
    WARN: Could not locate executable f77
    INFO: customize NAGFCompiler
    INFO: customize VastFCompiler
    INFO: customize CompaqFCompiler
    WARN: Could not locate executable fort
    INFO: customize IntelItaniumFCompiler
    WARN: Could not locate executable efort
    WARN: Could not locate executable efc
    INFO: customize IntelEM64TFCompiler
    INFO: customize GnuFCompiler
    WARN: Could not locate executable g77
    INFO: customize G95FCompiler
    WARN: Could not locate executable g95
    INFO: customize PathScaleFCompiler
    WARN: Could not locate executable pathf95
    INFO: customize NAGFORCompiler
    WARN: Could not locate executable nagfor
    INFO: customize FujitsuFCompiler
    WARN: Could not locate executable frt
    WARN: don't know how to compile Fortran code on platform 'posix'
    warning: build_ext: f77_compiler=None is not available.
    
    INFO: building '_glmnet' extension
    error: extension '_glmnet' has Fortran sources but no Fortran compiler found
    INFO:
    ########### EXT COMPILER OPTIMIZATION ###########
    INFO: Platform      :
      Architecture: x64
      Compiler    : gcc
    
    CPU baseline  :
      Requested   : 'min'
      Enabled     : SSE SSE2 SSE3
      Flags       : -msse -msse2 -msse3
      Extra checks: none
    
    CPU dispatch  :
      Requested   : 'max -xop -fma4'
      Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CNL
      Generated   : none
    INFO: CCompilerOpt.cache_flush[857] : write cache to path -> /tmp/pip-install-j1g6z3bg/glmnet_2a4b2922fe5a4660b6e4a169242c965e/build/temp.linux-x86_64-3.9/ccompiler_opt_cache_ext.py
    ----------------------------------------
ERROR: Command errored out with exit status 1: /zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-j1g6z3bg/glmnet_2a4b2922fe5a4660b6e4a169242c965e/setup.py'"'"'; __file__='"'"'/tmp/pip-install-j1g6z3bg/glmnet_2a4b2922fe5a4660b6e4a169242c965e/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ismq92dd/install-record.txt --single-version-externally-managed --compile --install-headers /zfs/projects/faculty/ldblatt-mistreatment/blackbox_2.0/assets/blackbox_v2/include/site/python3.9/glmnet Check the logs for full command output.

[kalashnov]

Downgrading numpy to 1.22.1 helped me

[amirhessam88]

Definitely, they gotta release a new version and adapt the changes.

[amirhessam88]

@jacksonlee-civis Would you let us know if you are able to release a new version with updated dependency and prolly support Python 3.8, 3.9, ...
I am not sure if lower versions of Python is necessary

[jacksonlee-civis]

I'm not on this project anymore. To my knowledge, this project isn't currently staffed for either maintenance or feature work. Since this codebase lives under the Civis corporate umbrella, I'm afraid there's not much else I could say or do.

[amirhessam88]

@jacksonlee-civis Thanks; how can I tag someone from Civis to get more info?

[bnaul]

For anyone else who comes across this: we have published wheels from our fork (built using the method from #72) at https://pypi.org/project/python-glmnet/; you should be able to install for Python 3.9, 3.10, or 3.11 on Mac or Linux with

pip install python-glmnet

[amirhessam88]

Thanks @bnaul

[nileshtrivedi]

@mattyb @aehrmann @mwpeterson @Naphier Any chance this issue could be taken up soon?

[amirhessam88]

@nileshtrivedi I am not sure if this library is being maintained anymore; I highly recommend to switch using the clone that @bnaul put together. https://pypi.org/project/python-glmnet/

[mheilman]

Hi, I just wanted to confirm that this repo is no longer maintained. I made a PR (#85) to make that clear in the README.