GSoC Proposal

Thu Mar 28 15:18:12 UTC 2019

Dear Karan,

On Thu, 28 Mar 2019 at 07:52, KARAN SHETH <karan.sheth at somaiya.edu> wrote:

> Resending the mail cause, previous mail was not sent to mailing list.
> On Wed, Mar 27, 2019 at 2:10 PM Mojca Miklavec <mojca at macports.org> wrote:
>
>> Dear Karan,
>>
>> I'm just answering a small part of your question here, a bit more about
>> upt while CC-ing its author & potential mentor:
>>
>> On Tue, 26 Mar 2019 at 18:15, KARAN SHETH <karan.sheth at somaiya.edu>
>> wrote:
>>
>>>
>>>
>>>> (Another option for a project in Python would be bringing universal
>>>> packaging tool [1] to MacPorts.)
>>>>
>>>
>>> I looked in to this option and it looks very exciting, this is what I
>>> have understood so far about bringing upt to MacPorts :
>>> - A UPT-Frontend and Backend will have to be created for macports.
>>>
>>
>> No, MacPorts in itself requires writing just a backend to support
>> creating packages for existing frontends (currently there are frontends for
>> pypi, ruby gems, cpan), but you usually need a separate backend for each of
>> the frontends. However, it would be great to maybe create new frontends for
>> npm, cabal, ... (npm would be particularly helpful). And if working on that
>> project, it would make sense to concentrate on at least one frontend (ruby
>> gems, npm, or something else) and try to make some awesome improvement of
>> our packages. Our ruby packages are so outdated that they are literally
>> useless, haskell support is horribly lagging behind as so much manual work
>> is needed (we would like to provide the latest version of pandoc), and npm
>> support is basically non-existent.
>>
>> The work would most likely include doing various improvements to UPT
>> itself. In particular, the biggest problem would be how to keep packages
>> up-to-date after the initial creation. We currently have 1000+ perl
>> packages, plenty of python packages etc. and it's a pain to keep them up to
>> date. When packages change, new dependencies are added, and none of that is
>> currently automated. The current version of UPT support creating the inial
>> version of the package, but there is no mechanism in place yet to keep
>> package updated afterwards (incorporating other changes you might have made
>> to the Portfile itself).
>>
>>
>>> So the Frontend will take the Portfile from macports and convert it
>>> using UPT.
>>>
>>
>> No, the frontend would take a package from pypi / cpan / ruby gems / npm
>> / ... and generate a Portfile.
>>
>>
>>> But my doubt is, what is the UPT-Backend in this case, I know it should
>>> be MacOS but like in the example given on UPT repo its converts json from
>>> PYPI into Guix Pxg Definition.
>>> So same way, what would UPT do in case of Macports?
>>>
>>
>> Generate a Portfile
>>
>>
>>> I may be completely wrong here as I have no prior knowledge about UPT.
>>>
>>
>> You aren't expected to have any prior knowledge about it. It's relatively
>> new, currently one-person project, but we had quite some GSOC projects in
>> the part to do various conversion from X to Portfiles (from pypi, cpan),
>> and supporting such a project would solve a lot more packaging nightmares
>> all at once.
>>
>> This is some example code that was cooked together in cca. two hours:
>>     https://github.com/mojca/upt-macports
>>
>> I would need to add some documentation, but once you set it up, the
>> command like
>>     upt package -f pypi -b macports upt
>> would create a MacPorts package for the "upt" package (that's the last
>> argument) as fetched from pypi (frontend pypi, backend macports, package
>> upt).
>>
>>
>>> Now seeing that I have two choices - first to continue making the django
>>> app or to make upt, I will first look into UPT after my doubts raised above
>>> have been clarified and then decide which route to take.
>>>
>>
>> The choice is yours, you can also submit two proposals, but it makes most
>> sense to really create one awesome proposal rather than trying to submit
>> three, none of which you had a substantial time to do properly. (Or make
>> one really awesome proposal, and a backup one.) If more than one student
>> applies for the same project, we can only select one, but it really makes
>> most sense to apply for the one that you are most likely to shine in.
>>
>> If you plan to apply for this project, what you might want to look into
>> is approximately the following:
>> - get familiar with cpan2port & pypi2port to generate perl and python
>> packages (install it via macports, figure out how to use it, try to create
>> a package, take a glimpse at the source code)
>> - figure out how to create and / or modify perl / python / ruby / ...
>> packages
>> - get familiar with using "port livecheck", maybe find some outdated
>> package that you are using and try to update it (if your interest is in
>> python, search for ports inside "python")
>> - try to install UPT (maybe via virtualenv & pip) and find out how to
>> generate a package with an existing backend (the macports backend on the
>> link above should also be able to generate super simplified python
>> packages, but it's not yet complete)
>> - you could check https://trac.macports.org/ticket/48324 /
>> https://lists.macports.org/pipermail/macports-dev/2018-March/037531.html just
>> to get an idea about haskell issues and lack of automation there
>>
>
>
> I looked into all that you had mentioned and gave a good thought about
> which project to work on, and even though I am good at django and JS, I
> would like to work on UPT as my main project as this seems more exciting to
> me and I feel it can help many people and will speed up the porting process
> (and if time permits, I will prepare a demo app for the Collect Build stats
> too). But first I'll try and make a rough npm frontend. And regarding the
> backend, I tried running your macport backend but got a few errors which I
> was able to kind of solve but now that I understand a bit more I guess
> I'll  give it one more try and fix any issues.
>

That's great!

> So, I am hoping that I have a rough code for the npm frontend and macport
> backend ready in a day or at max two, which I will submit to you for
> review.
>

We are looking forward. I hope that Cyril will also help with the process
(my encounter with UPM so far was for the full two hours while hacking on
it :).

> This is what I have thought to do :
> - For npm frontend:
>     - get the json for any package from
> https://registry.npmjs.org/<package-name>
>     - parse it as per upt-frontend template
>
> - For macport backend:
>    - First I will fix the issues with the Pypi code and then add npm to it
>
> Given all what I have tried out, I expect everything to go smoothly but if
> I have any doubt's, I will get in touch.
>

OK.

I've also read about port livecheck but haven't tried it yet.
>
> One concern/doubt I have is - does the Portfile generated always have to
> be correct as I don't think that's achievable given the complexity
> sometimes, but sure upt can be a starting point and with few manual edits
> Portfile can be perfected.
>

Yes, the Portfile should always be syntactically correct, even though it
may not always be the final / ultimate version. There are definitely cases
where external software is needed and the internal name of macports cannot
simply be deduced from pypi or npm index. So I expect that further changes
might often be needed.

Note that my version did not yet print the checksums. This functionality
has been added to upm after I played with it, so the checksums are
definitely still missing.

Now, this brings us to the question of how to keep these ports up to date,
which might be even more important than creating those ports in the first
place. This is something that needs to be investigated and needs some fresh
ideas.

We could in principle have some huge index file with all the data from
various packages, and then individual portfiles would only list the name
and version of each individual port, while the data (including all
dependencies) would be fed from a table somewhere else in the system. Or
some other clever method for updating the ports needs to be devoted /
invented.

See for example

https://github.com/macports/macports-ports/blob/master/_resources/port1.0/group/crossgcc-1.0.tcl
which simply lists various versions of gcc compiler, and then individual
ports only say which version they want:

https://github.com/macports/macports-ports/blob/master/cross/i386-elf-gcc/Portfile

I imagine that it would make sense to write MacPorts backend + some
selected frontents like npm, and then:

   - find an elegant and efficient way of updating ports while keeping in
   mind that manual additions to packages might be needed (there are multiple
   options)
   - (maybe present some proof-of-concept solution for automatic updates of
   perl ports)
   - find a way to recursively generate packages (if you need a package and
   thirty of its dependencies are still missing in our tree, it should be able
   generate all of them at once)
   - automate the testing to some extent (pypi2port offers a command that
   creates the port, fetches the sources, compiles the port and maybe even
   runs the tests; I did not recheck it, so maybe I remember some details
   wrong)
   - pick some areas in MacPorts with weak support (ruby, cabal, npm, ...)
   and try to improve multiple packages, bringing them up to date, adding
   dependencies so that software like pandoc, rails etc. could be built, ...

These are just random ideas, picking up your favourites is up to you.

Mojca
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20190328/5c491f29/attachment-0001.html>