Porting atf - violating the layout of the ports-filesystems

[Julio Merino]

[ Rainer, Ivan: Sorry for sending this twice to you, but the strict
posting rules of the mailing list caused the original mail to be
dropped.  I had to subscribe and send the email again. ]

On Mon, Jul 12, 2010 at 5:42 PM, Ivan "Rambius" Ivanov
<rambiusparkisanius at gmail.com> wrote:
> [...]
> I contacted ATF's author to inform him that I had include ATF into
> MacPorts. He raised concerns about moving the tests into
> ${prefix}/share/${name}/tests.
>
> His main objection was that the tests tree was supposed to be shared
> across all packages and more than one package can install ATF-based
> tests. The user can just go into prefix/tests/ (imagine you'd have
> prefix/tests/atf/, prefix/tests/bind/, prefix/tests/gtk2/, etc.) and
> run all tests at once for all packages. After he explained it to me, I
> am more inclined to move the tests back to $prefix/tests and set
> destroot.violate_mtree to yes.
>
> I CC'ed the author of ATF - Julio and he will provide more details.

Hello all,

Yes, what Ivan mentions is correct.

First of all, putting the tests anywhere under prefix/share/ is plain
wrong.  Tests are executables and therefore they are not shareable
across machines of different platforms.  Therefore, they do not belong
in share (per the traditional definition of share).  (A reasonable
possibility would be to stick them under prefix/libexec/.  This is
mostly OK, but is also wrong in the sense that they are not private
binary programs intended to be executed by some other binary/library.
They are really supposed to be exposed to the user.)

Second, putting the tests under prefix/share/atf/tests/ is also wrong.
 atf will create a subdirectory called 'atf' under the tests
directory, which means you'll end up with prefix/share/atf/tests/atf/,
which is obviously redundant.

So the place for tests must be somewhere else and this is why atf
creates a new top-directory layout under prefix.  The point of this
tree is to allow any package that uses atf to put its tests there
under a subdirectory named after the package name.  This way, you have
a central place in your system where you can go, run the tests, and
ensure things work.

Imagine if every single atf-ified package put its tests in
prefix/tests/<package-name>/ -- it'd then be trivial to run the tests
for all the software you have installed and know that the software
works flawlessly on your current OS/hardware platform.  And even
better, you could do that periodically to ensure that, e.g. OS
upgrades do not screw things up.

Now, I know that installing the tests as part of the package[1] is a
completely different paradigm to the one offered by most testing
frameworks (I am not aware of any other that intends to do something
similar).  Therefore, you cannot apply the same logic to the atf
package as you would for others, and I would really ask you to
consider allowing the creation of the prefix/tests/ directory.

1: http://www.netbsd.org/~jmmv/atf/docs/faq.html#faq-user-why-install-test-programs

-- 
Julio Merino