Deepmind Tree Bazel Build Portfile Help Request
Steven Smith
steve.t.smith at gmail.com
Mon Jul 6 16:22:50 UTC 2020
I am not using --max_idle_secs=60.
Also, neither is port py-tensorflow nor the repo tensorflow/tensorflow itself, nor any other MacPorts repo, and these all build fine without this option:
https://github.com/macports/macports-ports/blob/master/python/py-tensorflow/Portfile <https://github.com/macports/macports-ports/blob/master/python/py-tensorflow/Portfile>
https://github.com/tensorflow/tensorflow/search?q=max_idle_secs&type=Code <https://github.com/tensorflow/tensorflow/search?q=max_idle_secs&type=Code>
https://github.com/macports/macports-ports/search?q=max_idle_secs&type=Code <https://github.com/macports/macports-ports/search?q=max_idle_secs&type=Code>
I’m happy to try adding it, but my expectation is that some other issue is causing the hang when run within the port command (the build itself only takes few seconds when run by hand).
I must also add the bazel-based builds tensorflow-probability and others, so getting a robust bazel build approach would be very helpful and useful.
Steve
> On Jul 6, 2020, at 11:57 AM, Christopher Jones <jonesc at hep.phy.cam.ac.uk> wrote:
>
> Hi,
>
> Are you using this option
>
> bazel --max_idle_secs=60
>
> when building with bazel ?
>
> Bazel builds start an underlying ’server’ process, that by default hangs around for 3600 secs (yes, 1hour) before disappearing. Without the above your build will hang around waiting for this to happen…
>
> Chris
>
>> On 6 Jul 2020, at 2:57 am, Steven Smith <steve.t.smith at gmail.com <mailto:steve.t.smith at gmail.com>> wrote:
>>
>> I’m preparing a PR for the Google Magenta project <https://magenta.tensorflow.org <https://magenta.tensorflow.org/>>, and am running into this issue for the “dm-tree” dependent package with the bazel build.
>>
>> The command `port -dv test py37-dm-tree` hangs during the bazel build.
>> I can’t even Ctl-C out of the process—it requires a full ps command then a kill -9 to shut down everything.
>> However, running the build by hand works:
>> sudo -u macports bash -c '/opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 setup.py --no-user-cfg build'
>>
>> Any pointers or help would be appreciated. I’ve never used bazel, so I copied a bunch of relevant stuff from the py-tensorflow Portfile and attempted a few obvious hacks, but nothing changes the underlying problem of a hung port command.
>>
>> Relevant files:
>>
>> https://github.com/deepmind/tree/blob/master/setup.py <https://github.com/deepmind/tree/blob/master/setup.py>
>> https://github.com/deepmind/tree/blob/master/bazel/BUILD <https://github.com/deepmind/tree/blob/master/bazel/BUILD>
>>
>> Bazel process (hung when called from MacPorts):
>>
>>> bazel(tree-0.1.6.20200524) -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/private/var/tmp/_bazel_root/54b8ea352146bad8adc65c2e363d519a -Xverify:none -Djava.util.logging.config.file=/private/var/tmp/_bazel_root/54b8ea352146bad8adc65c2e363d519a/javalog.properties -Dcom.google.devtools.build.lib.util.LogHandlerQuerier.class=com.google.devtools.build.lib.util.SimpleLogHandler$HandlerQuerier -XX:-MaxFDLimit -Djava.library.path=/var/tmp/_bazel_root/install/e3bdf1a360e6d6cc38da72ec898bd905/embedded_tools/tools/objc:/var/tmp/_bazel_root/install/e3bdf1a360e6d6cc38da72ec898bd905/ -Dfile.encoding=ISO-8859-1 -jar /var/tmp/_bazel_root/install/e3bdf1a360e6d6cc38da72ec898bd905/A-server.jar --max_idle_secs=10800 --noshutdown_on_low_sys_mem --connect_timeout_secs=30 --output_user_root=/var/tmp/_bazel_root --install_base=/var/tmp/_bazel_root/install/e3bdf1a360e6d6cc38da72ec898bd905 --install_md5=e3bdf1a360e6d6cc38da72ec898bd905 --output_base=/private/var/tmp/_bazel_root/54b8ea352146bad8adc65c2e363d519a --workspace_directory=/opt/local/var/macports/build/_opt_local_ports_python_py-dm-tree/py37-dm-tree/work/tree-0.1.6.20200524 --default_system_javabase=/Library/Java/JavaVirtualMachines/openjdk13/Contents/Home --failure_detail_out=/private/var/tmp/_bazel_root/54b8ea352146bad8adc65c2e363d519a/failure_detail.rawproto --deep_execroot --expand_configs_in_place --idle_server_tasks --write_command_log --nowatchfs --nofatal_event_bus_exceptions --nowindows_enable_symlinks --client_debug=false --product_name=Bazel --noincompatible_enable_execution_transition --option_sources=
>>
>>
>> Draft Portfile:
>>
>>> # -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4
>>>
>>> PortSystem 1.0
>>> PortGroup compilers 1.0
>>> PortGroup compiler_blacklist_versions 1.0
>>> PortGroup github 1.0
>>> PortGroup java 1.0
>>> PortGroup python 1.0
>>> PortGroup xcodeversion 1.0
>>> PortGroup xcode_workaround 1.0
>>>
>>> github.setup deepmind tree 2b81872
>>> # no official release; version from __init__.py plus github commit date
>>> version 0.1.6.20200524
>>> name py-dm-${github.project}
>>> revision 0
>>>
>>> platforms darwin
>>> license Apache-2
>>> maintainers nomaintainer
>>>
>>> description Deepmind tree is a library for working with nested\
>>> data structures.
>>> long_description ${description} In a way, tree generalizes the builtin\
>>> map function which only supports flat sequences, and\
>>> allows to apply a function to each "leaf" preserving\
>>> the overall structure.
>>>
>>> homepage https://github.com/deepmind/tree <https://github.com/deepmind/tree>
>>> distname ${github.project}-${version}
>>>
>>> checksums rmd160 40518b306e8f5a80b12ef5e76ca9e2f8ef232de3 \
>>> sha256 0748428f70fae2209b8763bf99cadaf22276a4391851ba919dce9d3d0bc047bf \
>>> size 35176
>>>
>>> python.versions 37 38
>>>
>>> # Required java version
>>> java.version 11+
>>> # JDK port to install if required java not found
>>> java.fallback openjdk14
>>> # JDK only needed at build time, but java PG sets lib dependency so
>>> # declare no conflict to allow redistribution of binaries.
>>> license_noconflict ${java.fallback}
>>>
>>> # require c++14
>>> compiler.cxx_standard 2014
>>>
>>> # The oldest Xcode version to use default Xcode compiler
>>> # https://github.com/tensorflow/tensorflow/issues/39262 <https://github.com/tensorflow/tensorflow/issues/39262>
>>> set tf_min_xcode 10.2
>>> compiler.blacklist-append {clang < 1001}
>>> # Work out if we should be using macports clang
>>> set use_mp_clang [ expr ( [ string match macports-clang-* ${configure.compiler} ] || [ vercmp ${xcodeversion} ${tf_min_xcode} ] < 0 ) ]
>>>
>>> use_parallel_build no
>>>
>>> if {${name} ne ${subport}} {
>>> depends_build-append \
>>> port:bazel \
>>> port:py${python.version}-setuptools
>>>
>>> depends_run-append \
>>> port:py${python.version}-six \
>>>
>>> depends_test-append \
>>> port:py${python.version}-absl \
>>> port:py${python.version}-attrs \
>>> port:py${python.version}-numpy \
>>> port:py${python.version}-pytest \
>>> port:py${python.version}-wrapt
>>>
>>> # diff -NaurdwB ./dm-tree-orig/setup.py ./dm-tree-new/setup.py | sed -E -e 's/\.\/dm-tree-(orig|new)*\/(setup\.py)(\.[[:alnum:]]+)*/\.\/setup.py/' > ./setup.py.patch
>>> # patchfiles setup.py.patch
>>>
>>> # set bazel_output_base ${workpath}/tmp/_bazel_root
>>> # xinstall -d ${bazel_output_base}
>>> # post-patch {
>>> # reinplace \
>>> # "s|@BAZEL_OUTPUT_USER_ROOT@|${bazel_output_base}|g" \
>>> # ${worksrcpath}/setup.py
>>> # }
>>>
>>> build.env-append \
>>> BAZEL_USE_CPP_ONLY_TOOLCHAIN=1 \
>>> JAVA_HOME=${java.home}
>>>
>>> test.run yes
>>> test.cmd py.test-${python.branch}
>>> test.target
>>> test.env-append \
>>> "PATH=$env(PATH):${workpath}/bin" \
>>> PYTHONPATH=${worksrcpath}/build/lib
>>>
>>> post-destroot {
>>> set docdir ${prefix}/share/doc/${subport}
>>> xinstall -d ${destroot}${docdir}
>>> xinstall -m 0644 -W ${worksrcpath} LICENSE README.md \
>>> ${destroot}${docdir}
>>> }
>>>
>>> livecheck.type none
>>> } else {
>>> livecheck.type pypi
>>> }
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20200706/e1591bd4/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3898 bytes
Desc: not available
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20200706/e1591bd4/attachment-0001.bin>
More information about the macports-dev
mailing list