<div dir="ltr"><div>Dear Karan,</div><div><br></div><div dir="ltr"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 28 Mar 2019 at 07:52, KARAN SHETH <<a href="mailto:karan.sheth@somaiya.edu" target="_blank">karan.sheth@somaiya.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Resending the mail cause, previous mail was not sent to mailing list.<br></div><div>On Wed, Mar 27, 2019 at 2:10 PM Mojca Miklavec <<a href="mailto:mojca@macports.org" target="_blank">mojca@macports.org</a>> wrote:</div><div class="gmail_quote"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Dear Karan,</div><div><br></div><div>I'm just answering a small part of your question here, a bit more about upt while CC-ing its author & potential mentor:</div><div><br></div><div class="gmail_quote"></div></div></div></div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 26 Mar 2019 at 18:15, KARAN SHETH <<a href="mailto:karan.sheth@somaiya.edu" target="_blank">karan.sheth@somaiya.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote" style="color:rgb(0,0,0)"><div></div><div>(Another option for a project in Python would be bringing universal packaging tool [1] to MacPorts.)</div></div></div></div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div><div class="gmail_quote"><div>I looked in to this option and it looks very exciting, this is what I have understood so far about bringing upt to MacPorts :<br></div><div>- A UPT-Frontend and Backend will have to be created for macports.<br></div></div></div></div></div></div></blockquote><div><br></div></div></div></div></div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div>No, MacPorts in itself requires writing just a backend to support creating packages for existing frontends (currently there are frontends for pypi, ruby gems, cpan), but you usually need a separate backend for each of the frontends. However, it would be great to maybe create new frontends for npm, cabal, ... (npm would be particularly helpful). And if working on that project, it would make sense to concentrate on at least one frontend (ruby gems, npm, or something else) and try to make some awesome improvement of our packages. Our ruby packages are so outdated that they are literally useless, haskell support is horribly lagging behind as so much manual work is needed (we would like to provide the latest version of pandoc), and npm support is basically non-existent.</div><div><br></div><div>The work would most likely include doing various improvements to UPT itself. In particular, the biggest problem would be how to keep packages up-to-date after the initial creation. We currently have 1000+ perl packages, plenty of python packages etc. and it's a pain to keep them up to date. When packages change, new dependencies are added, and none of that is currently automated. The current version of UPT support creating the inial version of the package, but there is no mechanism in place yet to keep package updated afterwards (incorporating other changes you might have made to the Portfile itself).</div></div></div></div></div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div><div class="gmail_quote"><div></div><div>So the Frontend will take the Portfile from macports and convert it using UPT.<br></div></div></div></div></div></div></blockquote><div><br></div></div></div></div></div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div>No, the frontend would take a package from pypi / cpan / ruby gems / npm / ... and generate a Portfile.</div></div></div></div></div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div><div class="gmail_quote"><div></div><div>But
my doubt is, what is the UPT-Backend in this case, I know it should be
MacOS but like in the example given on UPT repo its converts json from
PYPI int<code><font face="sans-serif">o Guix Pxg Definition.</font><br></code></div><div><code><font face="sans-serif">So same way, what would UPT do in case of Macports?</font></code></div></div></div></div></div></div></blockquote><div><br></div></div></div></div></div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div>Generate a Portfile</div></div></div></div></div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div><div class="gmail_quote"><div><code><font face="sans-serif">I may be completely wrong here as I have no prior knowledge about UPT.</font></code></div></div></div></div></div></div></blockquote><div><br></div></div></div></div></div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div>You aren't expected to have any prior knowledge about it. It's relatively new, currently one-person project, but we had quite some GSOC projects in the part to do various conversion from X to Portfiles (from pypi, cpan), and supporting such a project would solve a lot more packaging nightmares all at once.</div><div><br></div><div>This is some example code that was cooked together in cca. two hours:</div><div> <a href="https://github.com/mojca/upt-macports" target="_blank">https://github.com/mojca/upt-macports</a></div><div><br></div><div>I would need to add some documentation, but once you set it up, the command like</div><div> upt package -f pypi -b macports upt</div><div>would create a MacPorts package for the "upt" package (that's the last argument) as fetched from pypi (frontend pypi, backend macports, package upt).</div></div></div></div></div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div><div class="gmail_quote"><div>Now
seeing that I have two choices - first to continue making the django
app or to make upt, I will first look into UPT after my doubts raised
above have been clarified and then decide which route to take.</div></div></div></div></div></div></blockquote><div><br></div></div></div></div></div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div>The choice is yours, you can also submit two proposals, but it makes most sense to really create one awesome proposal rather than trying to submit three, none of which you had a substantial time to do properly. (Or make one really awesome proposal, and a backup one.) If more than one student applies for the same project, we can only select one, but it really makes most sense to apply for the one that you are most likely to shine in.</div><div><br></div><div>If you plan to apply for this project, what you might want to look into is approximately the following:</div><div>- get familiar with cpan2port & pypi2port to generate perl and python packages (install it via macports, figure out how to use it, try to create a package, take a glimpse at the source code)</div><div>- figure out how to create and / or modify perl / python / ruby / ... packages</div><div>- get familiar with using "port livecheck", maybe find some outdated package that you are using and try to update it (if your interest is in python, search for ports inside "python")</div><div>- try to install UPT (maybe via virtualenv & pip) and find out how to generate a package with an existing backend (the macports backend on the link above should also be able to generate super simplified python packages, but it's not yet complete)</div><div>- you could check <a href="https://trac.macports.org/ticket/48324" target="_blank">https://trac.macports.org/ticket/48324</a> / <a href="https://lists.macports.org/pipermail/macports-dev/2018-March/037531.html" target="_blank">https://lists.macports.org/pipermail/macports-dev/2018-March/037531.html</a> just to get an idea about haskell issues and lack of automation there</div></div></div></div></div></blockquote><div><br></div><div><br></div></div></div><div dir="ltr"><div class="gmail_quote"><div dir="auto"><div style="font-family:sans-serif;font-size:12.8px" dir="auto">I
looked into all that you had mentioned and gave a good thought about
which project to work on, and even though I am good at django and JS, I
would like to work on UPT as my main project as this seems more exciting to me and I feel it can help many people and will speed up the porting process (and if time permits, I
will prepare a demo app for the Collect Build stats too). But first I'll
try and make a rough npm frontend. And regarding the backend, I tried
running your macport backend but got a few errors which I was able to
kind of solve but now that I understand a bit more I guess I'll give it
one more try and fix any issues.</div></div></div></div></div></div></blockquote><div><br></div><div>That's great!</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_quote"><div dir="auto"><div style="font-family:sans-serif;font-size:12.8px" dir="auto">So,
I am hoping that I have a rough code for the npm frontend and macport
backend ready in a day or at max two, which I will submit to you for
review. <br></div></div></div></div></div></div></blockquote><div><br></div><div>We are looking forward. I hope that Cyril will also help with the process (my encounter with UPM so far was for the full two hours while hacking on it :).</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_quote"><div dir="auto"><div style="font-family:sans-serif;font-size:12.8px" dir="auto"></div><div style="font-family:sans-serif;font-size:12.8px" dir="auto"><span style="font-size:12.8px">This is what I have thought to do :</span><br></div><div style="font-family:sans-serif;font-size:12.8px" dir="auto">- For npm frontend:</div><div style="font-family:sans-serif;font-size:12.8px" dir="auto"> - get the json for any package from <a href="https://registry.npmjs.org/%3Cpackage-name" style="text-decoration-line:none;color:rgb(66,133,244)" target="_blank">https://registry.npmjs.org/<package-name</a>></div><div style="font-family:sans-serif;font-size:12.8px" dir="auto"> - parse it as per upt-frontend template <br></div><div style="font-family:sans-serif;font-size:12.8px" dir="auto"><br></div><div style="font-family:sans-serif;font-size:12.8px" dir="auto">- For macport backend:</div><div style="font-family:sans-serif;font-size:12.8px" dir="auto"> - First I will fix the issues with the Pypi code and then add npm to it <br></div><div style="font-family:sans-serif;font-size:12.8px" dir="auto"><br></div><div style="font-family:sans-serif;font-size:12.8px" dir="auto">Given all what I have tried out, I expect everything to go smoothly but if I have any doubt's, I will get in touch.</div></div></div></div></div></div></blockquote><div><br></div><div>OK.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_quote"><div dir="auto"><div style="font-family:sans-serif;font-size:12.8px" dir="auto"></div><div style="font-family:sans-serif;font-size:12.8px" dir="auto">I've also read about port livecheck but haven't tried it yet.</div><div style="font-family:sans-serif;font-size:12.8px" dir="auto"><br></div><div style="font-family:sans-serif;font-size:12.8px" dir="auto">One
concern/doubt I have is - does the Portfile generated always have to be
correct as I don't think that's achievable given the complexity
sometimes, but sure upt can be a starting point and with few manual
edits Portfile can be perfected.</div></div></div></div></div></div></blockquote><div><br></div><div>Yes, the Portfile should always be syntactically correct, even though it may not always be the final / ultimate version. There are definitely cases where external software is needed and the internal name of macports cannot simply be deduced from pypi or npm index. So I expect that further changes might often be needed.</div><div><br></div><div>Note that my version did not yet print the checksums. This functionality has been added to upm after I played with it, so the checksums are definitely still missing.</div><div><br></div><div>Now, this brings us to the question of how to keep these ports up to date, which might be even more important than creating those ports in the first place. This is something that needs to be investigated and needs some fresh ideas.</div><div><br></div><div>We could in principle have some huge index file with all the data from various packages, and then individual portfiles would only list the name and version of each individual port, while the data (including all dependencies) would be fed from a table somewhere else in the system. Or some other clever method for updating the ports needs to be devoted / invented.</div><div><br></div><div>See for example</div><div> <a href="https://github.com/macports/macports-ports/blob/master/_resources/port1.0/group/crossgcc-1.0.tcl">https://github.com/macports/macports-ports/blob/master/_resources/port1.0/group/crossgcc-1.0.tcl</a></div><div>which simply lists various versions of gcc compiler, and then individual ports only say which version they want:</div><div> <a href="https://github.com/macports/macports-ports/blob/master/cross/i386-elf-gcc/Portfile">https://github.com/macports/macports-ports/blob/master/cross/i386-elf-gcc/Portfile</a></div><div><br></div><div>I imagine that it would make sense to write MacPorts backend + some selected frontents like npm, and then:</div><div><ul><li>find an elegant and efficient way of updating ports while keeping in mind that manual additions to packages might be needed (there are multiple options)</li><li>(maybe present some proof-of-concept solution for automatic updates of perl ports)</li><li>find a way to recursively generate packages (if you need a package and thirty of its dependencies are still missing in our tree, it should be able generate all of them at once)</li><li>automate the testing to some extent (pypi2port offers a command that creates the port, fetches the sources, compiles the port and maybe even runs the tests; I did not recheck it, so maybe I remember some details wrong)</li><li>pick some areas in MacPorts with weak support (ruby, cabal, npm, ...) and try to improve multiple packages, bringing them up to date, adding dependencies so that software like pandoc, rails etc. could be built, ...</li></ul><div>These are just random ideas, picking up your favourites is up to you.</div></div><div><br></div><div>Mojca</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"></blockquote></div></div>
</div>