Prev: Installing makefiles for BSD Make
Next: Problems with sysutils/screen (GNU Screen)
From: Paul Macdonald on 23 Jul 2010 09:15
On 23/07/2010 09:39, Doug Barton wrote:
> I have a new version of portmaster in pretty good shape at the moment
> so I'd like to ask for volunteers to test it. The changes and new
> features are described below. You can get the new version from svn at:
> http://svn.freebsd.org/viewvc/base/user/dougb/portmaster/
> or
> $ svn co svn://svn.freebsd.org/base/user/dougb/portmaster

excellent Doug, thanks for the additional logging. Unfortunately i have
nothing that needs updating yet:)

--
-------------------------
Paul Macdonald
IFDNRG Ltd
Web and video hosting
-------------------------
t: 0131 5548070
m: 07534206249
e: paul(a)ifdnrg.com
w: http://www.ifdnrg.com
-------------------------
IFDNRG
40 Maritime Street
Edinburgh
EH6 6SA
-------------------------

_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

From: "Douglas Berry" on 23 Jul 2010 10:21
Thanks for the new code! Package updates for perl work fine here,
and no regressions found.

With respect to the --clean-* options, my earlier (a long time ago)
test with --clean-distfiles showed that valid distfiles would be
deleted if the corresponding port was not installed. I'd like to
only delete them if they are no longer valid (similar to portsclean
-D). I commonly rsync /usr/ports/distfiles to different build hosts,
thus saving multiple downloads from the 'net. I can't test the
validity of this at the moment, is it still the case? Would the
--clean-packages feature behave the same way?

cheers,
doug

_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

From: Doug Barton on 23 Jul 2010 15:00
On Fri, 23 Jul 2010, Douglas Berry wrote:

> Thanks for the new code! Package updates for perl work fine here, and
> no regressions found.

Excellent! Thanks for taking the time to look it over.

> With respect to the --clean-* options, my earlier (a long time ago)
> test with --clean-distfiles showed that valid distfiles would be
> deleted if the corresponding port was not installed. I'd like to only
> delete them if they are no longer valid (similar to portsclean -D).

The problem is that in order to accomplish that portmaster would have to
check every port in the tree. Assuming 22,000 ports, and that any given
port is equally likely to fall anywhere in the tree, on average you'd
have to search 11,000 ports for every distfile that is not related to a
port installed on that system.

Now what I _could_ do is use the same technique I use in
--clean-distfiles (create a text file with the distfile information) but
instead of limiting it to installed ports, do all of them. I have never
even tested that to see how long it would take, but I suppose I could
take a look.

> Would the --clean-packages feature behave the same way?

No, because the packages contain the ORIGIN in the ports tree that they
were created from, so it's trivial to test their validity. If they
package is not the most recent version there is a fallback test that
compares it to what is installed on the local host, but if the package
is up to date you'll never even get there.


hth,

Doug

--

Improve the effectiveness of your Internet presence with
a domain name makeover! http://SupersetSolutions.com/

Computers are useless. They can only give you answers.
-- Pablo Picasso

_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

From: jhell on 23 Jul 2010 18:29
On 07/23/2010 15:00, Doug Barton wrote:
> The problem is that in order to accomplish that portmaster would have to
> check every port in the tree. Assuming 22,000 ports, and that any given
> port is equally likely to fall anywhere in the tree, on average you'd
> have to search 11,000 ports for every distfile that is not related to a
> port installed on that system.
>
> Now what I _could_ do is use the same technique I use in
> --clean-distfiles (create a text file with the distfile information) but
> instead of limiting it to installed ports, do all of them. I have never
> even tested that to see how long it would take, but I suppose I could
> take a look.

Hi Doug Barton, Ports@,

Here is a suggestion that I have just played around with involving
locate(1) with environment LOCATE_PATH set in portmaster & ports-side
locate.updatedb(8) with LOCATE_CONFIG before the port snapshots are made.

Server side: ( at port snapshot creation )
LOCATE_CONFIG=/usr/ports/locate.rc /usr/libexec/locate.updatedb
locate.rc:
FCODES="/usr/ports/locate.database"
SEARCHPATHS="/usr/ports"

real 149.79
user 3.51
sys 27.53
2076 maximum resident set size
37 average shared memory size
3946 average unshared data size
134 average unshared stack size
2893 page reclaims
335 page faults
0 swaps
0 block input operations
0 block output operations
0 messages sent
0 messages received
0 signals received
35875 voluntary context switches
38020 involuntary context switches


In portmaster(8):
LOCATE_PATH=/usr/ports/locate.database

Generate a list of distinfo files and create a distfile index using
``SIZE''.

for file in $(locate "/usr/ports/*/*/distinfo"); do
grep "SIZE " $file >>DISTINDEX.8
done

Use the output from that to judge if the file in /usr/ports/distfiles
needs to exist or of the file does not meet the SIZE requirements & you
can kill two birds with one stone.

You could always use the MD5 or SHA256 sum that's recorded but that may
be IMO a slight bit of overhead.

This may not be an exact proccess right now but it sure would be nice to
have a locate database in ports that could be updated by (make
locatedb) or something similar.


Just for creating the list of distfiles from the SIZE contents:
$ du -sh DISTINDEX.8.1.gz
512B DISTINDEX.8.1.gz

time -alhp ./createindex.sh
real 490.99
user 28.03
sys 117.14
1820 maximum resident set size
228 average shared memory size
2297 average unshared data size
133 average unshared stack size
2353331 page reclaims
0 page faults
0 swaps
0 block input operations
0 block output operations
0 messages sent
0 messages received
0 signals received
77884 voluntary context switches
165392 involuntary context switches


Regards,

--

jhell,v

_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

From: Doug Barton on 23 Jul 2010 20:13
On Fri, 23 Jul 2010, jhell wrote:

> On 07/23/2010 15:00, Doug Barton wrote:
>>
>> Now what I _could_ do is use the same technique I use in
>> --clean-distfiles (create a text file with the distfile information) but
>> instead of limiting it to installed ports, do all of them. I have never
>> even tested that to see how long it would take, but I suppose I could
>> take a look.

Turns out this wasn't quite as painful as I thought it would be, so I've
added the feature to svn version r210440. On my system, without a full
ports tree, it takes between 3.5 and 4.5 minutes to build the list.

> Hi Doug Barton, Ports@,
>
> Here is a suggestion that I have just played around with involving
> locate(1) with environment LOCATE_PATH set in portmaster & ports-side
> locate.updatedb(8) with LOCATE_CONFIG before the port snapshots are made.

This is certainly an interesting idea, however it has a lot of potential
problems. Best case scenario on a default system with ports tree updated
just before locate.db is created, on average the information would be
3.5 days out of date.

There is also another problem, not all of the ports' distinfo files are
named "distinfo." Most of them are, but not all.

> Server side: ( at port snapshot creation )
> LOCATE_CONFIG=/usr/ports/locate.rc /usr/libexec/locate.updatedb
> locate.rc:
> FCODES="/usr/ports/locate.database"
> SEARCHPATHS="/usr/ports"
>
> real 149.79

It's not 100% clear to me what you're timing here, but it took 2.5
minutes to do it.

> Use the output from that to judge if the file in /usr/ports/distfiles
> needs to exist or of the file does not meet the SIZE requirements & you
> can kill two birds with one stone.

Currently --clean-distfiles doesn't attempt to validate that the file
matches the specs in the distinfo file, and I don't see a compelling
reason to add that feature. It would be very difficult (nearly
impossible) to accidentally save an invalid file, and totally impossible
to use one to build a port.

> Just for creating the list of distfiles from the SIZE contents:
> $ du -sh DISTINDEX.8.1.gz
> 512B DISTINDEX.8.1.gz
>
> time -alhp ./createindex.sh
> real 490.99

Sorry I'm so dense, but I'm still not following exactly what part of the
process you're referring to here, but this time it took over 8 minutes.
Given that the method I'm using to generate a 100% up to date version of
the list that's guaranteed to get all the distfiles takes less time, I
do not think I will be adding the feature you are suggesting, but I do
appreciate the suggestion. :)


Doug

--

Improve the effectiveness of your Internet presence with
a domain name makeover! http://SupersetSolutions.com/

Computers are useless. They can only give you answers.
-- Pablo Picasso

_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

 |  Next  |  Last
Pages: 1 2
Prev: Installing makefiles for BSD Make
Next: Problems with sysutils/screen (GNU Screen)