[MacPorts] #47056: nco/ncks and netcdf
MacPorts
noreply at macports.org
Sat Mar 7 16:08:12 PST 2015
#47056: nco/ncks and netcdf
--------------------------------+-----------------------
Reporter: barry.j.mcinnes@… | Owner: takeshi@…
Type: defect | Status: new
Priority: Normal | Milestone:
Component: ports | Version: 2.3.3
Resolution: | Keywords:
Port: nco netcdf |
--------------------------------+-----------------------
Changes (by ryandesign@…):
* owner: macports-tickets@… => takeshi@…
* keywords: nco netcdf =>
* cc: takeshi@… (removed)
Old description:
> The current nco and netcdf on macports, currently take more than 2 hours
> (killed), whereas the versions on linux for the
> same files and command take 13 seconds. A similar nccopy command takes 13
> seconds on the same mac for the same files.
>
> The linux setup is
> /usr/local/bin/ncks -r
> NCO netCDF Operators version "4.4.5" last modified 2014/07/15 built Aug
> 29 2014 on linux64.psd.esrl.noaa.gov by root
> ncks version 4.4.5
> Linked to netCDF library version 4.3.3, compiled Feb 24 2015 07:21:57
> Copyright (C) 1995--2014 Charlie Zender
> This program is part of NCO, the netCDF Operators.
> NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO
> WARRANTY
> You can redistribute and/or modify NCO under the terms of the
> GNU General Public License (GPL) Version 3 with exceptions described in
> the LICENSE file
> GPL: http://www.gnu.org/copyleft/gpl.html
> LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE
> Homepage: http://nco.sf.net
> User Guide: http://nco.sf.net/nco.html
> Configuration Option: Active? Meaning or Reference:
> Check _FillValue Yes http://nco.sf.net/nco.html#mss_val
> Check missing_value No http://nco.sf.net/nco.html#mss_val
> DAP clients Yes http://nco.sf.net/nco.html#dap
> Debugging: Custom No Pedantic, bounds checking (slowest
> execution)
> Debugging: Symbols No Produce symbols for debuggers (e.g., dbx,
> gdb)
> GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl
> HDF4 support Unknown http://nco.sf.net/nco.html#hdf4
> Internationalization No http://nco.sf.net/nco.html#i18n (pre-
> alpha)
> MPI parallelization No http://nco.sf.net/nco.html#mpi (beta)
> netCDF3 64-bit files Yes http://nco.sf.net/nco.html#lfs
> netCDF4/HDF5 available Yes http://nco.sf.net/nco.html#nco4
> netCDF4/HDF5 enabled Yes http://nco.sf.net/nco.html#nco4
> OpenMP SMP threading No http://nco.sf.net/nco.html#omp
> Optimization: run-time No Fastest execution possible (slowest
> compilation)
> Parallel netCDF3 No http://nco.sf.net/nco.html#pnetcdf (pre-
> alpha)
> Regular Expressions Yes http://nco.sf.net/nco.html#rx
> Shared libraries built No Small, dynamically linked executables
> Static libraries built Yes Large executables with private namespaces
> UDUnits conversions Yes http://nco.sf.net/nco.html#udunits
> UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits
>
> The macports is
> ncks -r
> NCO netCDF Operators version "4.4.7" last modified 2014/11/05 built Jan
> 6 2015 on mac73.psd.esrl.noaa.gov by root
> ncks version 4.4.7
> Linked to netCDF library version 4.3.2, compiled Feb 3 2015 08:18:38
> Copyright (C) 1995--2014 Charlie Zender
> This program is part of NCO, the netCDF Operators.
> NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO
> WARRANTY
> You can redistribute and/or modify NCO under the terms of the
> GNU General Public License (GPL) Version 3 with exceptions described in
> the LICENSE file
> GPL: http://www.gnu.org/copyleft/gpl.html
> LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE
> Homepage: http://nco.sf.net
> User Guide: http://nco.sf.net/nco.html
> Configuration Option: Active? Meaning or Reference:
> Check _FillValue Yes http://nco.sf.net/nco.html#mss_val
> Check missing_value No http://nco.sf.net/nco.html#mss_val
> DAP clients Yes http://nco.sf.net/nco.html#dap
> Debugging: Custom No Pedantic, bounds checking (slowest
> execution)
> Debugging: Symbols No Produce symbols for debuggers (e.g., dbx,
> gdb)
> GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl
> HDF4 support Unknown http://nco.sf.net/nco.html#hdf4
> Internationalization No http://nco.sf.net/nco.html#i18n (pre-
> alpha)
> MPI parallelization No http://nco.sf.net/nco.html#mpi (beta)
> netCDF3 64-bit files Yes http://nco.sf.net/nco.html#lfs
> netCDF4/HDF5 available Yes http://nco.sf.net/nco.html#nco4
> netCDF4/HDF5 enabled Yes http://nco.sf.net/nco.html#nco4
> OpenMP SMP threading No http://nco.sf.net/nco.html#omp
> Optimization: run-time No Fastest execution possible (slowest
> compilation)
> Parallel netCDF3 No http://nco.sf.net/nco.html#pnetcdf (pre-
> alpha)
> Regular Expressions Yes http://nco.sf.net/nco.html#rx
> Shared libraries built No Small, dynamically linked executables
> Static libraries built Yes Large executables with private namespaces
> UDUnits conversions Yes http://nco.sf.net/nco.html#udunits
> UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits
>
> The command is
> ncks -D 3 --fl_fmt=netcdf4_classic -L 5 --overwrite
> huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc
> /Projects/CMIP5A//Monthly/historical/ACCESS1-0//r1i1p1//huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc
>
> The file size is 200MB.
> Maybe an update to netcdf 4.3.3 will speed things up ?
> Maybe its the chunking size the macports cannot handle ?
> One thing I note is that the chunck size for the ncks for the huss
> variable on Mac ends up being:
> 1872, 145, 192
> but using nccopy, it's 1, 145, 192 which is more in line with what I
> would expect.
New description:
The current nco and netcdf on macports, currently take more than 2 hours
(killed), whereas the versions on linux for the
same files and command take 13 seconds. A similar nccopy command takes 13
seconds on the same mac for the same files.
The linux setup is
{{{
/usr/local/bin/ncks -r
NCO netCDF Operators version "4.4.5" last modified 2014/07/15 built Aug 29
2014 on linux64.psd.esrl.noaa.gov by root
ncks version 4.4.5
Linked to netCDF library version 4.3.3, compiled Feb 24 2015 07:21:57
Copyright (C) 1995--2014 Charlie Zender
This program is part of NCO, the netCDF Operators.
NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO
WARRANTY
You can redistribute and/or modify NCO under the terms of the
GNU General Public License (GPL) Version 3 with exceptions described in
the LICENSE file
GPL: http://www.gnu.org/copyleft/gpl.html
LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE
Homepage: http://nco.sf.net
User Guide: http://nco.sf.net/nco.html
Configuration Option: Active? Meaning or Reference:
Check _FillValue Yes http://nco.sf.net/nco.html#mss_val
Check missing_value No http://nco.sf.net/nco.html#mss_val
DAP clients Yes http://nco.sf.net/nco.html#dap
Debugging: Custom No Pedantic, bounds checking (slowest
execution)
Debugging: Symbols No Produce symbols for debuggers (e.g., dbx,
gdb)
GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl
HDF4 support Unknown http://nco.sf.net/nco.html#hdf4
Internationalization No http://nco.sf.net/nco.html#i18n (pre-
alpha)
MPI parallelization No http://nco.sf.net/nco.html#mpi (beta)
netCDF3 64-bit files Yes http://nco.sf.net/nco.html#lfs
netCDF4/HDF5 available Yes http://nco.sf.net/nco.html#nco4
netCDF4/HDF5 enabled Yes http://nco.sf.net/nco.html#nco4
OpenMP SMP threading No http://nco.sf.net/nco.html#omp
Optimization: run-time No Fastest execution possible (slowest
compilation)
Parallel netCDF3 No http://nco.sf.net/nco.html#pnetcdf (pre-
alpha)
Regular Expressions Yes http://nco.sf.net/nco.html#rx
Shared libraries built No Small, dynamically linked executables
Static libraries built Yes Large executables with private namespaces
UDUnits conversions Yes http://nco.sf.net/nco.html#udunits
UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits
}}}
The macports is
{{{
ncks -r
NCO netCDF Operators version "4.4.7" last modified 2014/11/05 built Jan 6
2015 on mac73.psd.esrl.noaa.gov by root
ncks version 4.4.7
Linked to netCDF library version 4.3.2, compiled Feb 3 2015 08:18:38
Copyright (C) 1995--2014 Charlie Zender
This program is part of NCO, the netCDF Operators.
NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO
WARRANTY
You can redistribute and/or modify NCO under the terms of the
GNU General Public License (GPL) Version 3 with exceptions described in
the LICENSE file
GPL: http://www.gnu.org/copyleft/gpl.html
LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE
Homepage: http://nco.sf.net
User Guide: http://nco.sf.net/nco.html
Configuration Option: Active? Meaning or Reference:
Check _FillValue Yes http://nco.sf.net/nco.html#mss_val
Check missing_value No http://nco.sf.net/nco.html#mss_val
DAP clients Yes http://nco.sf.net/nco.html#dap
Debugging: Custom No Pedantic, bounds checking (slowest
execution)
Debugging: Symbols No Produce symbols for debuggers (e.g., dbx,
gdb)
GNU Scientific Library Yes http://nco.sf.net/nco.html#gsl
HDF4 support Unknown http://nco.sf.net/nco.html#hdf4
Internationalization No http://nco.sf.net/nco.html#i18n (pre-
alpha)
MPI parallelization No http://nco.sf.net/nco.html#mpi (beta)
netCDF3 64-bit files Yes http://nco.sf.net/nco.html#lfs
netCDF4/HDF5 available Yes http://nco.sf.net/nco.html#nco4
netCDF4/HDF5 enabled Yes http://nco.sf.net/nco.html#nco4
OpenMP SMP threading No http://nco.sf.net/nco.html#omp
Optimization: run-time No Fastest execution possible (slowest
compilation)
Parallel netCDF3 No http://nco.sf.net/nco.html#pnetcdf (pre-
alpha)
Regular Expressions Yes http://nco.sf.net/nco.html#rx
Shared libraries built No Small, dynamically linked executables
Static libraries built Yes Large executables with private namespaces
UDUnits conversions Yes http://nco.sf.net/nco.html#udunits
UDUnits2 conversions Yes http://nco.sf.net/nco.html#udunits
}}}
The command is
{{{
ncks -D 3 --fl_fmt=netcdf4_classic -L 5 --overwrite
huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc
/Projects/CMIP5A//Monthly/historical/ACCESS1-0//r1i1p1//huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc
}}}
The file size is 200MB.
Maybe an update to netcdf 4.3.3 will speed things up ?
Maybe its the chunking size the macports cannot handle ?
One thing I note is that the chunck size for the ncks for the huss
variable on Mac ends up being:
1872, 145, 192
but using nccopy, it's 1, 145, 192 which is more in line with what I would
expect.
--
--
Ticket URL: <https://trac.macports.org/ticket/47056#comment:2>
MacPorts <https://www.macports.org/>
Ports system for OS X
More information about the macports-tickets
mailing list