Prev: INDEX build failed for 6.x
Next: INDEX now builds successfully on 6.x
From: Alexander Leidinger on 3 Jun 2010 09:02
Quoting Peter Jeremy <peter(a)vk2pj.dyndns.org> (from Thu, 3 Jun 2010
22:37:28 +1000):

> On 2010-Jun-01 10:54:02 +0200, Jan Henrik Sylvester <me(a)janh.de> wrote:
>> Yesterday, I was chasing libintl.so.8, rebuilding all ports that got
>> bumped, checking with libchk for other libintl.so.8 dependencies, and
>> forcing a rebuild of all these packages: All but two of them had an
>> indirect dependency on devel/gettext (and I did email the maintainers of
>> devel/ccrtp and textproc/gsed linking without a dependency).
>
> This might be unrealistic but, IMHO, these "indirect dependencies" should
> not exist. IMHO, there should only be two situations:
>
> 1) Port X directly links against or dlopen's libY.so from port Y.
>
> In this case, port Y should be listed in LIB_DEPENDS or equivalent
> for port X and port X will need a portrevision bump and rebuild if
> the port Y ABI changes (eg a .so version bump)
>
> 2) Port X directly links against or dlopen's libZ.so and libZ.so pulls
> in libY.so from port Y.
>
> In this case, port X should not be directly accessing any symbols in
> libY.so. If the libY.so ABI changes, libZ.so will need to be rebuilt
> but unless the libZ.so ABI changes, there should be no need to rebuild
> port X.
>
> Are there any other situations that have to be considered?

The problem is a little bit more complex if you have a look at the big
picture.

Indirect dependencies get hardcoded into binaries in several
cases(*1), except you use a compiler switch which tells the
compile-time-linker to not record library dependencies when no symbols
of those libs are reference directly(*2).

If you want to know which libs are linked into binaries of a port, I
suggest you have a look at my
/usr/ports/Tools/scripts/explicit_lib_depends.sh script (note that its
name starts with "explicit", not with "direct", this is on purpose as
those two descriptions have a different semantic which is important
for this discussion).

(*1) Either by libtool as it does not has some hint enabled on
FreeBSD, or by pkg-config listing them directly even if the
corresponding library does not reference symbols of such a dependency
in its API.

(*2) This is used by some gnome ports (not all), and can not be
enabled globally, as some programs may depend upon this (e.g. when
loading plugins which are not linked correctly).

Bye,
Alexander.

--
Most people can do without the essentials, but not without the luxuries.

http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7
http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137
_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

From: Jan Henrik Sylvester on 4 Jun 2010 09:35
On 01/-10/63 20:59, Alexander Leidinger wrote:
> Quoting Peter Jeremy <peter(a)vk2pj.dyndns.org> (from Thu, 3 Jun 2010
> 22:37:28 +1000):
>
>> On 2010-Jun-01 10:54:02 +0200, Jan Henrik Sylvester <me(a)janh.de> wrote:
>>> Yesterday, I was chasing libintl.so.8, rebuilding all ports that got
>>> bumped, checking with libchk for other libintl.so.8 dependencies, and
>>> forcing a rebuild of all these packages: All but two of them had an
>>> indirect dependency on devel/gettext (and I did email the maintainers of
>>> devel/ccrtp and textproc/gsed linking without a dependency).
>>
>> This might be unrealistic but, IMHO, these "indirect dependencies" should
>> not exist. IMHO, there should only be two situations:
>>
>> 1) Port X directly links against or dlopen's libY.so from port Y.
>>
>> In this case, port Y should be listed in LIB_DEPENDS or equivalent
>> for port X and port X will need a portrevision bump and rebuild if
>> the port Y ABI changes (eg a .so version bump)
>>
>> 2) Port X directly links against or dlopen's libZ.so and libZ.so pulls
>> in libY.so from port Y.
>>
>> In this case, port X should not be directly accessing any symbols in
>> libY.so. If the libY.so ABI changes, libZ.so will need to be rebuilt
>> but unless the libZ.so ABI changes, there should be no need to rebuild
>> port X.
>>
>> Are there any other situations that have to be considered?
>
> The problem is a little bit more complex if you have a look at the big
> picture.
>
> Indirect dependencies get hardcoded into binaries in several cases(*1),
> except you use a compiler switch which tells the compile-time-linker to
> not record library dependencies when no symbols of those libs are
> reference directly(*2).
>
> If you want to know which libs are linked into binaries of a port, I
> suggest you have a look at my
> /usr/ports/Tools/scripts/explicit_lib_depends.sh script (note that its
> name starts with "explicit", not with "direct", this is on purpose as
> those two descriptions have a different semantic which is important for
> this discussion).
>
> (*1) Either by libtool as it does not has some hint enabled on FreeBSD,
> or by pkg-config listing them directly even if the corresponding library
> does not reference symbols of such a dependency in its API.
>
> (*2) This is used by some gnome ports (not all), and can not be enabled
> globally, as some programs may depend upon this (e.g. when loading
> plugins which are not linked correctly).

Thanks for your reply and blog posting. With direct and indirect, I was
refering to package dependencies and not library dependencies.

I have checked one of the missing dependencies with
explicit_lib_depends.sh, but it did not show up, because
explicit_lib_depends.sh makes some assumptions upon were binaries reside
that are not true in this case:

%objdump -x /usr/local/share/examples/telepathy-qt4/call/.libs/call |
grep NEEDED | grep intl
NEEDED libintl.so.9
%pkg_info -W /usr/local/share/examples/telepathy-qt4/call/.libs/call
/usr/local/share/examples/telepathy-qt4/call/.libs/call was installed by
package telepathy-qt4-0.3.2
%/usr/ports/Tools/scripts/explicit_lib_depends.sh telepathy-qt4-0.3.2
WARNING: your libtool records dependencies recursively, you can not
trust the following output.
QtCore:${PORTSDIR}/devel/qt4-corelib
QtDBus:${PORTSDIR}/devel/dbus-qt4
QtNetwork:${PORTSDIR}/net/qt4-network
QtXml:${PORTSDIR}/textproc/qt4-xml
USE_GNOME+= _glib20 (devel/glib20)
USE_GNOME+= libxml2 (textproc/libxml2)
gstinterfaces-0.10:${PORTSDIR}/multimedia/gstreamer-plugins
gstreamer-0.10:${PORTSDIR}/multimedia/gstreamer
telepathy-farsight:${PORTSDIR}/net-im/telepathy-farsight
telepathy-glib:${PORTSDIR}/net-im/telepathy-glib

Thus, explicit_lib_depends.sh cannot be relied upon what to rebuild,
either -- it missed the same that libchk does. Anyhow, this does not
matter at all to the main point I tried to raise:

Should _all_ explicit and direct dependencies (to use your vocabulary)
be accounted for in the LIBDEPENDS of a port? The LIBDEPENDS are used to
bump PORTREVISIONs to trigger rebuilds. Not having all explicit and
direct dependencies listed there, reduces the use of PORTREVISION bumps
so much that their negative side effect (at least when they are not done
at the same moment as the original commit) becomes dominant (people
relying on libchk, bsdadminscripts, or the like are forced to rebuild
ports that are already consistent).

Peter seems to be of my opinion by what he said above, marcus@ did
introduce USE_GETTEXT for many ports that already have indirect port
dependencies, so I assume that he agrees, too. On the other hand, pav@
(who is among portmgr) seems to disagree (at least a year ago), as I
stated in my first mail:
http://lists.freebsd.org/pipermail/cvs-ports/2009-May/172173.html And he
is not the only one with that opinion.

Yes, the architecture of ports is insufficient for a good solution of
any kind, but as long as there is not even an agreement, what LIBDEPENDS
are supposed to contain, following port updates is harder than it should
be, because of the different strategies used to bump shared libraries
that affect many ports.

Considering the few responses my posting got, either not many see the
need to reach an agreement or the posing was not clear or not concise
enough. (It is not the first time I tried to raise this issue.)

Cheers,
Jan Henrik
_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

From: "b. f." on 4 Jun 2010 15:59
>Thus, explicit_lib_depends.sh cannot be relied upon what to rebuild,
>either -- it missed the same that libchk does. Anyhow, this does not
>matter at all to the main point I tried to raise:
>

Maybe the specific results that you mentioned don't matter, but the
fact that these programs fail to obtain satisfactory results in real
cases, due to some of the complicating factors Alexander mentioned,
_does_ matter -- because if there were a cheap, foolproof method of
determining which ports needed to be updated in all cases, without
introducing further changes in the ports infrastructure, then we could
implement the method in the ports infrastructure, or in portmaster and
portupgrade, and forget about the question that you have raised.


>Should _all_ explicit and direct dependencies (to use your vocabulary)
>be accounted for in the LIBDEPENDS of a port? The LIBDEPENDS are used to
>bump PORTREVISIONs to trigger rebuilds. Not having all explicit and
>direct dependencies listed there, reduces the use of PORTREVISION bumps
>so much that their negative side effect (at least when they are not done
>at the same moment as the original commit) becomes dominant (people
>relying on libchk, bsdadminscripts, or the like are forced to rebuild
>ports that are already consistent).
>

It is better in most cases to err on the side of caution and rebuild
too much, than to rebuild too little, and have a broken system. But
you seem to be conflating two different problems here, as I read it:
the effectiveness of the PORTREVISION bumps is reduced when all direct
dependencies aren't recorded in *_DEPENDS and a simplistic method like
that in ports/Tools/scripts/bump_revision.pl is used to determine
which ports need to be bumped. But in that case too little is being
rebuilt, not too much.


>Peter seems to be of my opinion by what he said above, marcus@ did
>introduce USE_GETTEXT for many ports that already have indirect port
>dependencies, so I assume that he agrees, too. On the other hand, pav@
>(who is among portmgr) seems to disagree (at least a year ago), as I
>stated in my first mail:
>http://lists.freebsd.org/pipermail/cvs-ports/2009-May/172173.html And he
>is not the only one with that opinion.

There are some strident portlint warnings that may cause some
maintainers, rightly or wrongly, to add a direct dependency on
devel/gettext, even when there is already an indirect dependency. So
I wouldn't read too much into J. Marcus' actions. I suspect that Pav
is weighing the costs of refactoring all of the *_DEPENDS in Ports to
follow what you seem to be suggesting, and the added overhead of
processing more dependencies during builds, versus the concern that a
few users are going to have trouble during major upgrades if they are
building from source on a running system (this doesn't seem to be a
problem on tinderboxes and package-building clusters), and finding
that it isn't worth the effort from his perspective. With the current
practice, all dependencies may not be correctly recorded as direct
dependencies, but at least (barring some other mistakes) they _are_
recorded.

>Yes, the architecture of ports is insufficient for a good solution of
>any kind, but as long as there is not even an agreement, what LIBDEPENDS
>are supposed to contain, following port updates is harder than it should
>be, because of the different strategies used to bump shared libraries
>that affect many ports.
>
>Considering the few responses my posting got, either not many see the
>need to reach an agreement or the posing was not clear or not concise
>enough. (It is not the first time I tried to raise this issue.)

Just because one thinks that, ideally, LIB_DEPENDS should contain all
direct library dependencies, and that such a change may help resolve
some problems, doesn't necessarily mean that one thinks that, in
practice, all of them should be added -- because there is a cost to
doing so. In principle, I agree with what I think that you are
suggesting, that they should be added, but I can see why others may be
hesitant to do this, and I haven't tried to estimate the added
overhead on a system with a lot of ports.

b.
_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

From: Alexander Leidinger on 5 Jun 2010 17:39
On Fri, 04 Jun 2010 15:35:06 +0200 Jan Henrik Sylvester <me(a)janh.de>
wrote:

> I have checked one of the missing dependencies with
> explicit_lib_depends.sh, but it did not show up, because
> explicit_lib_depends.sh makes some assumptions upon were binaries
> reside that are not true in this case:

The heuristics are far from perfect. I just took the obvious ones. As
the scripts are intended to write out values for LIB_DEPENDS, and the
fact that indirect dependencies shall not be recorded there, the
scripts where not useful when I wrote them (I noticed this after
writting as much as you can see in the ports collection), and are still
not useful because indirect dependencies are still recorded in libs
and programs. For this reason I did not invest more time in improving
which parts of a pkg-plist to check or not.

Feel free to submit imrpovements to the scripts.

[recording indirect deps or not]
> Considering the few responses my posting got, either not many see the
> need to reach an agreement or the posing was not clear or not concise
> enough. (It is not the first time I tried to raise this issue.)

The best solution would be to fix the ports to not link explicitely to
indirect deps (by improving libtool and by improving the .pc files
for pkg-config). Then we could even switch from recording indirect
dependencies in /var/db/pkg/<port>-<version>/+{CONTENTS,REQUIRED_BY} to
only record direct deps.

Bye,
Alexander.
_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

From: Alex Dupre on 7 Jun 2010 10:35
Alexander Leidinger ha scritto:
> The best solution would be to fix the ports to not link explicitely to
> indirect deps (by improving libtool and by improving the .pc files
> for pkg-config). Then we could even switch from recording indirect
> dependencies in /var/db/pkg/<port>-<version>/+{CONTENTS,REQUIRED_BY} to
> only record direct deps.

How hard is it? What prevents us in doing it? Later we modify libtool
upstream, later we could switch to record only direct dependencies.

--
Alex Dupre
_______________________________________________
freebsd-ports(a)freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscribe(a)freebsd.org"

First  |  Prev  |  Next  |  Last
Pages: 1 2 3 4
Prev: INDEX build failed for 6.x
Next: INDEX now builds successfully on 6.x