<div dir="ltr">Hi<div><br></div><div>I am not saying that my db and the exiting Db would be interdependent.</div><div>Rather i am saying just once in forever I can copy the content to my database.</div><div>Then do the code to keep on updating it whenever something is added or deleted or modified.</div><div><br></div><div>Like for exaple maintainers.</div><div>You already have a table of mainatainers and their ports.So i can copy the data to my database.</div><div>Then i will make a similar to <span style="font-size:12.8px">PortIndex2PGSQL.tcl script to keep on updating my database independent of the existing database.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Again and again processing the 15mb json file would be bothersome.</span></div><div><span style="font-size:12.8px">We definitely need some way of getting a differential json.</span></div><div><span style="font-size:12.8px">To update only the changes happened because of commit to the port file.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Or maybe we could make some changes in buildbot that.Whenever there is some change in portfile it could update my port table as well with the changes.</span></div><div><span style="font-size:12.8px">Not sure how to do this.But just an idea as of now.</span></div><div><span style="font-size:12.8px"><br></span></div><div>"<span style="font-size:12.8px">You need a separate glue table to connect maintainers with ports. But</span></div><span style="font-size:12.8px">that's already part of your database design. You should not store</span><br style="font-size:12.8px"><span style="font-size:12.8px">maintainers in you port table, but perform correct joins (or maybe</span><br style="font-size:12.8px"><div><span style="font-size:12.8px">create a view first?) instead.</span>"</div><div><br></div><div>You misunderstood me, Copying the data.I meant once in a lifetime.</div><div>Then keep on updating the table with whatever changes happens.</div><div>And i will definitely perform join or views.</div><div><br></div><div>I will be doing following things :</div><div>- Update Ticket descriptions</div><div>- Put the gsoc proposal in docs</div><div>- update the graph on the static website</div><div>- try to handle sensitive keys</div><div><br></div><div><br></div><div>Regarding heroku.</div><div>Don't worry about that. I got it figured out.</div><div><br></div><div>And regarding working on a seperate branch.Rainer suggested not to.Because daily pull requests have to merged by the mentor.</div><div><br></div><div>Would directly commiting to master be a problem ?<br>As only i will be dealing with the repo.</div><div>Probably no one will right now clone the repository.</div><div><br></div><div>It is also very easy to directly commit to master than creating a PR.</div><div><br></div><div><br></div><div><br></div><div>One doubt I still have is that whether <span style="font-size:12.8px">portindex2postgres.tcl again and again runs the entire MacPorts port tree on the database ?<br>Or just the port that has been changed ?</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Thanks</span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 11 May 2018 at 12:47, Mojca Miklavec <span dir="ltr"><<a href="mailto:mojca@macports.org" target="_blank">mojca@macports.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 11 May 2018 at 04:00, Vishnu wrote:<br>
> Hi<br>
><br>
> Also i wanted to know<br>
> For this link<br>
> <a href="https://www.macports.org/ports.php?by=name&substr=python27" rel="noreferrer" target="_blank">https://www.macports.org/<wbr>ports.php?by=name&substr=<wbr>python27</a><br>
><br>
> There is already existing database with loads of information.<br>
> Is it not updated?<br>
> is it static ?<br>
<br>
</span>It is not static, the database is updated via an external script from<br>
PortIndex and the page is rendered with php code, reading information<br>
from the database.<br>
<span class=""><br>
> If it is updated automatically...Then half of the work for database would<br>
> already be over.<br>
><br>
> Could i get access to the DB?<br>
<br>
</span>Once your application reaches a certain quality & completeness, it<br>
should render the above page obsolete and the above page will most<br>
likely be removed.<br>
<br>
While we could theoretically give you access to the database (not<br>
without opening some security holes): first of all this hardly makes<br>
any sense because you should always be able to reproduce the exact<br>
state of the database if you run PortIndex2PGSQL.tcl yourself, but<br>
most importantly we don't want to introduce a complex network of<br>
interdependencies.<br>
<br>
Let's say that I need to change the new app a few months from now.<br>
Before I can deploy the change I need to test it locally on my<br>
machine. If you app will require password-protected or ip-limited<br>
access to some third-party database, this would greatly hinder my<br>
ability to patch the new app. The app should be as standalone as<br>
possible. The next very important aspect is that you might want to<br>
include more information than what's currently available in that<br>
database. For example, you might decide to include the information<br>
about which port supports which version of macOS. That information is<br>
not available in the existing database, so you would then first need<br>
to patch who-knows-what-other-code in random mixture of tcl, php,<br>
bash, ... And you would not be able to do table joins etc. It's<br>
absolutely bad idea to depend on third-party database. You should of<br>
course collect information from various sources, but you need to keep<br>
all the required data in a single database somewhere close to your<br>
app.<br>
<br>
Some time ago you had ideas about updating portindex just once per<br>
year? Well, here's an example why you should ideally do it after each<br>
commit rather than once per year. That said, we should ideally find a<br>
way to optimise the PortIndex2json part (probably not your task).<br>
PortIndex already updates just those ports that have in fact been<br>
changed, but conversion to json updates the complete file and needs to<br>
feed the web app with full 15 MB of data after each commit. This is<br>
quite a bit suboptimal.<br>
<span class=""><br>
> I got the rough idea of the db from here:<br>
> <a href="https://github.com/macports/macports-infrastructure/blob/2129f0cd0eb80f207d2cc62542b65c197733ac51/jobs/PortIndex2PGSQL.tcl#L249" rel="noreferrer" target="_blank">https://github.com/macports/<wbr>macports-infrastructure/blob/<wbr>2129f0cd0eb80f207d2cc62542b65c<wbr>197733ac51/jobs/<wbr>PortIndex2PGSQL.tcl#L249</a><br>
<br>
</span>Sure, you can take this as inspiration for database design, but not as<br>
your definite source of data. Your definite source of data should be<br>
PortIndex from the latest git checkout of macports-ports.<br>
<span class=""><br>
> So is this updated regularly?How?<br>
<br>
</span>I'm not absolutely sure, but I believe the changes are deployed after<br>
each commit via<br>
<a href="https://build.macports.org/builders/jobs-portindex" rel="noreferrer" target="_blank">https://build.macports.org/<wbr>builders/jobs-portindex</a><br>
<span class=""><br>
> And there seems to be lot of data already existing. Which i could just copy.<br>
> Existing maintainer table,many other.<br>
<br>
</span>No. You should replicate that in your app.<br>
<br>
You need a separate glue table to connect maintainers with ports. But<br>
that's already part of your database design. You should not store<br>
maintainers in you port table, but perform correct joins (or maybe<br>
create a view first?) instead.<br>
<span class=""><br>
> On 11 May 2018 at 07:11, Vishnu <<a href="mailto:vishnum1998@gmail.com">vishnum1998@gmail.com</a>> wrote:<br>
>><br>
>> Hi<br>
>><br>
>> So should I start working on the same database?<br>
>><br>
>> Community does not have any suggestions I guess.<br>
>> So should i go ahead with the existing structure?<br>
<br>
</span>*I* do have lots of comments even if nobody else does. One of the<br>
problems is that the excell table and your md documents were not in<br>
sync last time I checked (some tables were missing etc.)<br>
<br>
While I'm working on them, can you please:<br>
<br>
- Update ticket descriptions (some of them were empty when I checked,<br>
some would say "Implement this feature" without actually telling<br>
anything else). Super short descriptions are in principle ok, but only<br>
if we have an additional document describing what will actually be<br>
done. You already spent quite some time writing your proposal, but the<br>
proposal is only visible to GSOC mentors. Can we put your proposal<br>
(probably in markdown?) next to database design document? The idea is<br>
that anyone looking at the repository should have an understanding of<br>
what will be done (and could in principle pick it up himself), but we<br>
might want to update the document during the summer. While you might<br>
know that feature X should be implemented, maybe you don't know yet<br>
how to implement it and before you spend a lot of time on coding<br>
something that would be a bad idea and would need to be dropped<br>
because it would be too inefficient or too insecure, it makes sense to<br>
have a very clear document of what will be done.<br>
<br>
- Remove fake build statistics from the dynamic app and instead<br>
complete the "static sample page" (or pages) that you started to work<br>
on during your coding challenge, and commit it somewhere to the<br>
repository. You can/should keep the fake data there, but also complete<br>
the page, at least with the info about *what* should be there. You had<br>
just one graph there. You don't need to actually implement lots of<br>
graphs, but you need to create placeholder (say, just a title and<br>
description about what the graph to come there should show).<br>
<span class=""><br>
>> I had a doubt regarding version.<br>
>><br>
>> How can i check the existing versions of any port.<br>
>> Say python27?<br>
>><br>
>> Any list of maintainers i can get?<br>
>><br>
>> Thanks<br>
>><br>
>> On 9 May 2018 at 04:06, Vishnu <<a href="mailto:vishnum1998@gmail.com">vishnum1998@gmail.com</a>> wrote:<br>
>>><br>
>>> Hi<br>
>>><br>
>>> And also i couldn't figure out any way to hide passwords/ Sensitive<br>
>>> information while creating app.<br>
<br>
</span>Here's one way:<br>
<a href="https://ultimatedjango.com/learn-django/lessons/handling-sensitive-keys/" rel="noreferrer" target="_blank">https://ultimatedjango.com/<wbr>learn-django/lessons/handling-<wbr>sensitive-keys/</a><br>
<br>
Of course you need that information on the server where you are<br>
running the application, but the secrets and passwords should not be<br>
stored in a public repository.<br>
What I often do is create something like<br>
settings.py.sample<br>
or perhaps just<br>
secrets.py.sample<br>
and commit that one with a fake password to repository. Then, whoever<br>
wants to run the app, should first copy the file (removing the .sample<br>
extension), enter the correct secret data and only then run the app.<br>
<br>
Again: you do need to have this information stored somewhere, it just<br>
may not leak to a public repository. If you commit settings.py with<br>
fake passwords and correct the password on that one file, you might<br>
accidentally commit the change one day, so it's better to have a<br>
separate file.<br>
<span class=""><br>
>>> On 9 May 2018 at 03:43, Vishnu <<a href="mailto:vishnum1998@gmail.com">vishnum1998@gmail.com</a>> wrote:<br>
>>>><br>
</span><span class="">>>>> I had one doubt.<br>
>>>> Should i switch the link in heroku account for integration with macports<br>
>>>> github ?<br>
<br>
</span>I'm not sure what is it that you are asking. If Heroku need special<br>
priviliges on GitHub (what permissions are required?), it might be<br>
best to create an additional user on GitHub for now. Can you provide<br>
some pointers? It's pointless that I theoreticise about the options<br>
before I know what is required.<br>
<span class=""><br>
>>>> Because i think then you need to give accesss to heroku of your account.<br>
>>>><br>
>>>> I think it would be wise for me to do the commit update in my local<br>
>>>> repository itself..<br>
>>>><br>
>>>> Once every 2 weeks or something ill push all the changes to macports<br>
>>>> repository.<br>
>>>> Do comment .What should be done?<br>
<br>
</span>I wanted to suggest that the app would be developed either in a<br>
separate branch or in your fork of repository (not in master branch,<br>
please!), and then you would make pull requests on regular basis and<br>
we would review the pull request and make sure that any code that ends<br>
up in repository has been fully reviewed / tested.<br>
<br>
If you commit everything (including tests) straight to the master<br>
branch of the main repository, it's more challenging to track which<br>
code reviews have already been taken into account and which ones were<br>
not. You should not make giant pull requests anyway (ideally you would<br>
make a PR on approximately daily basis or at least once per week in<br>
case you need to figure out a number of things and code is not yet<br>
ready; making one giant PR every two weeks or - god forbid - once per<br>
month or only at the end of GSOC might cause too many troubles).<br>
<span class="HOEnZb"><font color="#888888"><br>
Mojca<br>
</font></span></blockquote></div><br></div>