Deepmind Tree Bazel Build Portfile Help Request

Steven Smith steve.t.smith at gmail.com
Tue Jul 7 17:53:08 UTC 2020


Fixed the issue: requires creation of TMPDIR

xinstall -d ${worksrcpath}/tmp

Thanks again.

Steve

> On Jul 7, 2020, at 7:31 AM, Steven Smith <steve.t.smith at gmail.com> wrote:
> 
> Thanks to your help, I’ve been able to edit setup.py akin to what you’ve done in py-tensorflow and have solved the long timeout issue, but not the underlying cause of the hang.
> 
> The command:
> 
>> sudo port build py37-dm-tree
> 
> fails to build (with no reported errors, except that it fails). However, running the python build command by hand in ${worksrcpath} works:
> 
>> sudo -u macports bash -c 'python3.7 setup.py --no-user-cfg build -j6'
> 
> I don’t know enough about “the wonderful bazel build system …” to troubleshoot, but I have a few hypotheses:
> There’s some issue with handling bazel output when run within the port command
> There’s some issue with bazel downloading [???] things it needs when run within port (e.g. in this case it’s getting @com_google_absl from somewhere)
> 
> Any pointers would be appreciated. I don’t see <https://github.com/macports/macports-ports/search?q=bazel&unscoped_q=bazel> any existing Portfiles that use bazel from within a setup.py. I’ve attached the draft Portfile and setup.py.patch below.
> 
> 
> By the way, a successful by-hand build produces many SUBCOMMAND lines like this:
> 
>> SUBCOMMAND: # @com_google_absl//absl/numeric:int128 [action 'Linking external/com_google_absl/absl/numeric/libint128.a', configuration: e2b2526ee351b893fa655145c6fbfa76bc6002a7b6841de71a5ce32b31c693d5]
>> (cd /opt/local/var/macports/build/_opt_local_ports_python_py-dm-tree/py37-dm-tree/work/760db6fa436447021e976cb8885c1247/execroot/tree && \
>>   exec env - \
>>     APPLE_SDK_PLATFORM=MacOSX \
>>     APPLE_SDK_VERSION_OVERRIDE=10.15 \
>>     PATH='/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/opt/X11/bin:/Applications/Server.app/Contents/ServerRoot/usr/bin:/Applications/Server.app/Contents/ServerRoot/usr/sbin:/Library/Apple/usr/bin:/usr/local/bin' \
>>     XCODE_VERSION_OVERRIDE=11.5.0.11E608c \
>>   external/local_config_cc/libtool -no_warning_for_no_symbols -static -o bazel-out/darwin-opt/bin/external/com_google_absl/absl/numeric/libint128.a bazel-out/darwin-opt/bin/external/com_google_absl/absl/numeric/_objs/int128/int128.o)
> 
> 
> 
> 
> Draft Portfile:
> 
>> # -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4
>> 
>> PortSystem          1.0
>> PortGroup           compiler_blacklist_versions 1.0
>> PortGroup           github 1.0
>> PortGroup           java 1.0
>> PortGroup           python 1.0
>> PortGroup           xcode_workaround 1.0
>> 
>> github.setup        deepmind tree 2b81872
>> # no official release; version from __init__.py plus github commit date
>> version             0.1.6.20200524
>> name                py-dm-${github.project}
>> revision 0
>> 
>> platforms           darwin
>> license             Apache-2
>> maintainers         nomaintainer
>> 
>> description         Deepmind tree is a library for working with nested\
>>                     data structures.
>> long_description    ${description} In a way, tree generalizes the builtin\
>>                     map function which only supports flat sequences, and\
>>                     allows to apply a function to each "leaf" preserving\
>>                     the overall structure.
>> 
>> homepage            https://github.com/deepmind/tree <https://github.com/deepmind/tree>
>> distname            ${github.project}-${version}
>> 
>> checksums           rmd160  40518b306e8f5a80b12ef5e76ca9e2f8ef232de3 \
>>                     sha256  0748428f70fae2209b8763bf99cadaf22276a4391851ba919dce9d3d0bc047bf \
>>                     size    35176
>> 
>> python.versions     37 38
>> 
>> # Required java version
>> java.version        11+
>> # JDK port to install if required java not found
>> java.fallback       openjdk14
>> # JDK only needed at build time, but java PG sets lib dependency so
>> # declare no conflict to allow redistribution of binaries.
>> license_noconflict  ${java.fallback}
>> 
>> # require c++14
>> compiler.cxx_standard 2014
>> 
>> # The oldest Xcode version to use default Xcode compiler
>> # https://github.com/tensorflow/tensorflow/issues/39262 <https://github.com/tensorflow/tensorflow/issues/39262>
>> set tf_min_xcode    10.2
>> compiler.blacklist-append {clang < 1001}
>> # Work out if we should be using macports clang
>> set use_mp_clang [ expr ( [ string match macports-clang-* ${configure.compiler} ] || [ vercmp ${xcodeversion} ${tf_min_xcode} ] < 0 ) ]
>> 
>> proc python_listify {tcl_list} {
>>     set python_list {}
>>     foreach item [lreverse ${tcl_list}] {
>>         set python_list "'${item}', ${python_list}"
>>     }
>>     return ${python_list}
>> }
>> 
>> if {${name} ne ${subport}} {
>>     depends_build-append \
>>                     port:bazel \
>>                     port:py${python.version}-setuptools
>> 
>>     depends_run-append \
>>                     port:py${python.version}-six \
>> 
>>     depends_test-append \
>>                     port:py${python.version}-absl \
>>                     port:py${python.version}-attrs \
>>                     port:py${python.version}-numpy \
>>                     port:py${python.version}-pytest \
>>                     port:py${python.version}-wrapt
>> 
>>     # Limit the number of parallel jobs to the number of physical, not logical, cpus.
>>     # First current setting to ensure we would be reducing the current setting.
>>     set physicalcpus [sysctl hw.physicalcpu]
>>     if { ${build.jobs} > ${physicalcpus} } {
>>         build.jobs ${physicalcpus}
>>     }
>> 
>>     # Build using the wonderful bazel build system ...
>>     set bazel_cmd "bazel --max_idle_secs=60 --output_user_root=${workpath}"
>>     set bazel_build_opts "--subcommands --compilation_mode=opt --verbose_failures"
>>     # Limit bazel resource utilisation
>>     set bazel_build_opts "${bazel_build_opts} --jobs ${build.jobs} --local_ram_resources=HOST_RAM*0.75 --local_cpu_resources=HOST_CPUS*.75"
>>     # Explicitly pass SDK https://github.com/bazelbuild/rules_go/issues/1554 <https://github.com/bazelbuild/rules_go/issues/1554>
>>     # Check versioned SDK actually exists... https://trac.macports.org/ticket/60317 <https://trac.macports.org/ticket/60317>
>>     if {[string first ${configure.sdk_version} ${configure.sdkroot}] != -1} {
>>         set bazel_build_opts "${bazel_build_opts} --macos_sdk_version=${configure.sdk_version}"
>>     } else {
>>         ui_warn "configure.sdkroot='${configure.sdkroot}' does not match configure.sdk_version='${configure.sdk_version}'"
>>     }
>>     # hack to try and transfer MP c, c++ and ld options to bazel...
>>     foreach opt [list {*}${configure.cflags} ] {
>>         set bazel_build_opts "${bazel_build_opts} --conlyopt \"${opt}\""
>>     }
>>     foreach opt [list {*}${configure.cxxflags} ] {
>>         set bazel_build_opts "${bazel_build_opts} --cxxopt \"${opt}\""
>>     }
>>     foreach opt [list {*}${configure.ldflags} ] {
>>         set bazel_build_opts "${bazel_build_opts} --linkopt \"${opt}\""
>>     }
>>     if { ${use_mp_clang} } {
>>         set bazel_build_opts "${bazel_build_opts} --action_env CC=${configure.cc <http://configure.cc/>}"
>>         set bazel_cmd "BAZEL_USE_CPP_ONLY_TOOLCHAIN=1 ${bazel_cmd}"
>>     }
>>     ui_debug "Bazel build command : ${bazel_cmd}"
>>     ui_debug "Bazel build options : ${bazel_build_opts}"
>> 
>>     # diff -NaurdwB ./dm-tree-orig/setup.py ./dm-tree-new/setup.py | sed -E -e 's/\.\/dm-tree-(orig|new)*\/(setup\.py)(\.[[:alnum:]]+)*/\.\/setup.py/' > ./setup.py.patch
>>     patchfiles      setup.py.patch
>> 
>>     post-patch {
>>         reinplace \
>>             "s|@BAZEL_CMD@|[python_listify ${bazel_cmd}]|g" \
>>             ${worksrcpath}/setup.py
>>         reinplace \
>>             "s|@BAZEL_BUILD_OPTS@|[python_listify ${bazel_build_opts}]|g" \
>>             ${worksrcpath}/setup.py
>>     }
>> 
>>     build.env-append \
>>         TMPDIR=${workpath}/tmp \
>>         JAVA_HOME=${java.home}
>> 
>>     test.run        yes
>>     test.cmd        py.test-${python.branch}
>>     test.target
>>     test.env-append \
>>                     "PATH=$env(PATH):${workpath}/bin" \
>>                     PYTHONPATH=${worksrcpath}/build/lib
>> 
>>     post-destroot {
>>         set docdir ${prefix}/share/doc/${subport}
>>         xinstall -d ${destroot}${docdir}
>>         xinstall -m 0644 -W ${worksrcpath} LICENSE README.md \
>>             ${destroot}${docdir}
>>     }
>> 
>>     livecheck.type      none
>> } else {
>>     livecheck.type      pypi
>> }
> 
> Draft setup.py.patch:
> 
>> --- ./setup.py	2020-07-05 09:50:12.000000000 -0400
>> +++ ./setup.py	2020-07-07 06:55:37.000000000 -0400
>> @@ -90,11 +90,11 @@
>>        os.makedirs(self.build_temp)
>>  
>>      bazel_argv = [
>> -        'bazel',
>> +        @BAZEL_CMD@
>>          'build',
>> +        @BAZEL_BUILD_OPTS@
>>          ext.bazel_target,
>>          '--symlink_prefix=' + os.path.join(self.build_temp, 'bazel-'),
>> -        '--compilation_mode=' + ('dbg' if self.debug else 'opt'),
>>      ]
>>  
>>      if IS_WINDOWS:
> 
> 
> 
> 
> 
>> On Jul 6, 2020, at 1:22 PM, Steven Smith <steve.t.smith at gmail.com <mailto:steve.t.smith at gmail.com>> wrote:
>> 
>> I’ll go try the settings you’re using with “the wonderful bazel build system ...”
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20200707/f1d6f9e2/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3898 bytes
Desc: not available
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20200707/f1d6f9e2/attachment-0001.bin>


More information about the macports-dev mailing list