Discussion:
Bug#841294: Overrule maitainer of "global" to package a new upstream version
(too old to reply)
Didier 'OdyX' Raboud
2016-11-16 14:23:47 UTC
Permalink
Hi there,

I've been mostly VAC, and only now found enough time to properly read
through this bug log. In the interest of transparency and to help the TC
reach a consensus (also through understanding what the other members'
understanding, ideas and positions are), here comes my current
understanding of the problem at hand.

As preamble, I will upfront state that I have _not_ looked in the precise
details of the so-called 'htags' functionality, as, the rest will show, my
current viewpoint on the problem is that it doesn't matter.

* src:global hasn't had a Debian upload since October 2010, for wheezy ,
and no new upstream-release Debian upload since September 2007, for
lenny (although there was on upload yesterday)
* there was no src:global upload to experimental
* its maintainer, Ron Lee, failed to provide timely and comprehensive answers
to various requests through bugreports over the years. On #574947
specifically, a bug filed in March 2010, Ron's first answer in April 2013
boiled down to "we're in freeze now, no".
* Ron claims to have had various private conversations about src:global with
various parties about the problems that a new upstream upgrade would pose.
* How the 'htags' functionality is implemented in newer upstream releases
seems to be at the heart of what Ron sees as making these "unsuitable for
release with the distro".
* There is arguably frustration on all sides of the discussion about upgrading
src:global to newer upstream releases, both sides arguing that they are
doing the right thing for Debian.

Now. I'd like to write down some considerations about the Debian namespace.

Debian, as a Free Software distribution, ships various software made by
various upstream authors, thereby filling the Debian source packages, binary
packages, and binaries namespaces. Upstream's global_*.tar.gz is shipped as
src:global source package, producing the 'global' binary package, shipping the
/usr/bin/global binary [0].

The TC has had various discussions about uses and abuses of these various
namespaces (especially the 'binary packages' and 'binaries' namespaces) in the
past, usually in cases of conflict. Various developers also often safeguard
the namespaces by launching discussions upon ITPs proposing too generic names.
How we handle these namespaces of ours is certainly an important topic for the
project. (But don't mistake me, the 'global' name isn't the problem here).

All these namespaces are also interfaces of what we as a project create, with
the community of our users. The most important of these three is arguably the
binary packages namespace, which is the usual interface with the software we
ship. It has been customary to have a one-to-one mapping between upstream
projects, and binary package names, and in all cases where we decided
otherwise (iceweasel, cupsys, nodejs, 
), it has imposed a lot of
documentation and convincing for our users to get used to these.

All in all, I think there is a reasonable expectation from our users to have
our namespaces match the rest of the ecosystem; there is an expectation that a
given 'foobar' upstream, iff packaged for Debian, gets to be shipped as the
'foobar' binary package. And I think that extends to versions as well; it is
expected that our releases ship with 'reasonably recent' versions, as well.

(
I know, Debian stable releases aren't known for the freshness of their
packages. There's still an expectation from our users
)

Outside of special times with regards to releases (that is, outside frozen
times), it is quite customary for packages to 'catch back' on new upstream
releases. This has multiple effects: it prepares the new _functionalities_,
and bug fixes for the next stable release; it also often comes with
regressions and new bugs; annoyances and headaches for users and maintainers
alike. Uploading new upstream releases and letting them be consumed by a
portion of our userbase (unstable & testing users), is a way to share the
burden, and _discover_ what those regressions and bugs are; identify, then try
to fix or document these.

As you have understood by now, I think it is a reasonable expectation from
package users to get new upstream releases between stable releases. The
upstream software evolves independently of Debian, and if we're not forking
the upstream software, we owe updates to our users as part of our "package
maintainers" duty. It's also our role to ship the newer works of our upstreams
to our users. (To clarify: I'm not saying there can't be exceptions.)

Now, coming back to src:global, there was a first request in March 2010
(#574947, 5.8.1) to get a new upstream release, before the squeeze freeze,
only answered late during the wheezy freeze (at that time, 6.2.8 was
released). The other request (#816924, 6.5.2) landed in March 2016, 9 months
before the stretch release. For me, it looks like the squeeze, jessie _and_
stretch release cycles have offered _vast_ opportunity to upload new upstream
versions, iron the bugs out in unstable (blocking migration to newer stable
for as long as needed to fix these supposed bugs), and land in testing.

I understand Ron's approach as follows: "I see this potential regression in
new upstream releases, which I want addressed before (even considering)
uploading it to unstable". This seems to have been the argument against any
new upstream uploads in the last 6+ years, deferring the resolution of these
potential regressions onto others. This, frankly, defeats the whole point of
having an 'unstable' suite, in which real users get their hands on new
packages, file bugs and find regressions; in which maintainers followup on
these, pipe them up to upstream, collaborate with both upstream and Debian
users to find satisfactory solutions, to provide the best of upstream works
combined with out Debian-specific expertise to our users.

Now, from the users' point of view, the src:global shipped in Debian has
various bugs, which upstream addressed (#590053, fixed in 5.9.1; #756367,
fixed in 6.3, #844356 fixed in 6.5.5).

To conclude with my current thoughts:

* I think our user's expectation of having a reasonably recent global when
using our 'global' package is justified.
* I think there was plenty of time and opportunities given to the current
src:global maintainer to ship the new upstream releases to our users (the
package could have been 6.1 in squeeze, 6.2 in wheezy, 6.3 in jessie; 6.5
could be in stretch), at the _very_ least in experimental, at least in
unstable.
* Without assuming malice, I observe that despite action promises, there were
no steps towards a 6.* src:global upload, and that the 'upcoming' or
'current' freezes have been repeatedly used as postponers multiple times
already; work also hadn't resumed after the releases.
* In general, as well as specifically for the src:global package, I don't
think it's reasonable for a maintainer to indefinitely [1] hold new upstream
releases back; if new releases have bugs, we ought to identify and fix them,
idealy in collaboration with upstream. If the new releases have immediately
identifiable flaws, then the initial new releases should be uploaded to
experimental, and Debian as well as upstream bugs filed. Uploads to unstable
with serious bugs (as in "makes the package unsuitable for release in the
package maintainer's opinion") are also acceptable, of course.
* In general, I think the namespace should be reserved to the most recent
package for a given upstream. I don't think it's reasonable to force the
maintenance of a src:global6 because the src:global maintainer insists on
staying on an old version.
* I'm not happy with delaying the landing of a 6.x version of global for
another release, and despite the somehwat short time before the stretch full
freeze (2017-Feb-05), we're talking about a single binary package without
reverse dependencies. I'm really afraid that a side-effect of the TC
discussion will be yet-another release without an up-to-date src:global.

I see the following good ways out of this problem:

A) Ron stays maintainer of src:global and uploads a reasonably recent 6.x
version to unstable;
B) (With or without an explicit decision by the TC), src:global is handed over
to new maintainers and they'd upload a reasonably recent 6.x version to
unstable;

(
In both A) and B) cases, whoever is maintainer of src:global would be in
charge of handling the subsequent (so far hypothetical) bugs in time for
the release. As usual, everyone is welcome to help with finding,
forwarding, and fixing bugs.
)

I see this as a suboptimal outcome, because it rewards inertia and stop-
energy:

C) Ron stays maintainer of src:global and uploads a reasonably recent 6.x
version to experimental, with the explicit goal of making that version later
available through stretch-backports.
--
Cheers,
OdyX

P.S. Sorry for the wall of text, including too many repetitions :/

[0] I know, it's all obvious.
[1] Yes, 6+ years, even at the Debian rhythm, is indefinite.
Wookey
2016-11-17 19:16:50 UTC
Permalink
On 2016-11-16 15:23 +0100, Didier 'OdyX' Raboud wrote:

[offlist]
Post by Didier 'OdyX' Raboud
* I'm not happy with delaying the landing of a 6.x version of global for
another release, and despite the somehwat short time before the stretch full
freeze (2017-Feb-05), we're talking about a single binary package without
reverse dependencies. I'm really afraid that a side-effect of the TC
discussion will be yet-another release without an up-to-date src:global.
Thank you for some sanity on this matter.

Wookey
--
Principal hats: Linaro, Debian, Wookware, ARM
http://wookware.org/
Ron
2016-11-21 06:46:34 UTC
Permalink
Hi OdyX,
Post by Didier 'OdyX' Raboud
Hi there,
I've been mostly VAC, and only now found enough time to properly read
through this bug log. In the interest of transparency and to help the TC
reach a consensus (also through understanding what the other members'
understanding, ideas and positions are), here comes my current
understanding of the problem at hand.
As preamble, I will upfront state that I have _not_ looked in the precise
details of the so-called 'htags' functionality, as, the rest will show, my
current viewpoint on the problem is that it doesn't matter.
If we run with your proposal, what are you actually suggesting we tell
the people who'd be upset by the loss of htags without notice in Stretch?
Because I don't really see how you've addressed that here.

AFAICS, there's just either an implicit "Sucks to be you", or an
implication that this is a simple "regression" which might be fixed
by sending patches upstream.

But since people have actually tried the latter, and it's not a "simple
regression", but rather upstream rejecting such discussion and then
actively removing features needed to implement it sanely in a distro
package context - it seems that only leaves the former ...


FWIW, I actually agree with a lot of the general rules of thumb that
you outlined here, about how things should work in the normal case.
But this isn't really a "normal" case, if it was we wouldn't be talking
about this here at all. What to do would be obvious to everyone.

If anything, it probably has more in common with the cdrtools situation
where upstream actively opposed making it suitable for distro use, while
insisting we accept those changes ... and we're split between some users
which want what has been crippled, and some which need what appears to be
just a few small changes which have been made upstream since.

And all of those people are supposedly software developers (as there's
not a lot of use for this package for people who aren't).


The group complaining loudly now have basically squandered the entire
release cycle by not reporting actionable bugs about what they need,
and haven't sent any patches to remedy that. And they are proposing
to solve their problem by penalising the other group, who so far wasn't
complaining, without any notice or real opportunity to contribute to
improving that for themselves if they end up on the losing side of this.

Which is why I suggested what I proposed in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=841294#155
better summarised by Tollef in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=841294#245

That gives the people who care about htags time to (re)act, and as soon
as the backports suite for Stretch opens, it means we get the essential
essence of what Phil had suggested for free too.

For Stretch people would be able to pick what they want in the same way
as he suggested, without triggering the concerns I expressed about
forking this in a single release and juggling about with which one owns
'global' as an unqualified name, and having to decide when to 'unfork'
it again, and face the fallout that might bring on.


How is that worse than telling the people who want htags, who do have
a history of sending thought out patches, that "your only option is to
create your own packages locally if you want to use this in Stretch"?

Maybe I missed something in what you're suggesting, but I don't see
how you suggest we defuse that from blowing up in our faces through
some reasonable and fair and justifiable action. Because their
complaint wouldn't really be substantively any different from what
the people who don't use htags are currently making.

"You're on the wrong side of a version number" isn't a very compelling
or satisfying or technically astute rationale, if that's all it boils
down to. I'd like to have something a bit more substantial and fair
than that to offer the people who'd get burned without notice by this.


Cheers,
Ron
Uoti Urpala
2016-11-21 15:09:28 UTC
Permalink
Post by Ron
If we run with your proposal, what are you actually suggesting we tell
the people who'd be upset by the loss of htags without notice in Stretch?
Because I don't really see how you've addressed that here.
 
AFAICS, there's just either an implicit "Sucks to be you", or an
implication that this is a simple "regression" which might be fixed
by sending patches upstream.
Assuming the htags functionality really can't be supported with a newer
upstream version: tell people that the functionality is no longer
available in current GLOBAL. If someone - including you - thinks this
is a major problem and wants to provide an alternative, a fork
providing this functionality can be packaged. Under a name other than
"global".
Post by Ron
FWIW, I actually agree with a lot of the general rules of thumb that
you outlined here, about how things should work in the normal case.
But this isn't really a "normal" case, if it was we wouldn't be talking
about this here at all.  What to do would be obvious to everyone.
There's nothing particularly "abnormal" about disagreeing with upstream
decisions. What is unusual, and is the reason why this has been
escalated here, is how badly you have handled the situation in your
packaging.
Post by Ron
The group complaining loudly now have basically squandered the entire
release cycle by not reporting actionable bugs about what they need,
and haven't sent any patches to remedy that.  And they are proposing
If anyone has "squandered the release cycle", it's you. You already
knew, or should have known, that the package was in an untenable state.
You've failed to fix the situation for years. You don't get to blame
other people for that.
Sam Hartman
2016-11-30 16:39:08 UTC
Permalink
Ron> Hi OdyX,
Post by Didier 'OdyX' Raboud
Hi there,
I've been mostly VAC, and only now found enough time to properly
read through this bug log. In the interest of transparency and to
help the TC reach a consensus (also through understanding what
the other members' understanding, ideas and positions are), here
comes my current understanding of the problem at hand.
As preamble, I will upfront state that I have _not_ looked in the
precise details of the so-called 'htags' functionality, as, the
rest will show, my current viewpoint on the problem is that it
doesn't matter.
Ron> If we run with your proposal, what are you actually suggesting
Ron> we tell the people who'd be upset by the loss of htags without
Ron> notice in Stretch? Because I don't really see how you've
Ron> addressed that here.

Ron, thanks for working with the process.

I think that one thing you sometimes find when you try and build
consensus is that your conception of the problem and even the values by
which a solution should be judged is not shared.

As I understand it, you believe we should be evaluating the technical
options and that preserving things that work is of high value.

I think a lot of us believe that get newest code is a presumptive value
especially over the multiple releases time frame.

That is, I disagree with you that I need to address the question of
htags and figure out whether htags users are being impacted.
That might have been true for one release.
But over a longer time frame, the really strong presumption is that we
prefer a version that is being actively developed to one that isn't.
That if people won't step forward and maintain htags, it goes awy.

That's a presumption. If someone gets evidence that htags is heavily
used, we can consider that.
We might even go so far given sufficient evidence to decide that forking
global and never taking a new upstream was the right answer for our
users. That would take a lot of evidence.

I disagree with your approach that the people wanting to remove htags
need to show that it is being used. I disagree with your view that over
the timescales involved, people wanting a new upstream need to justify
that.

Yes, removing htags creates a regression.

Yes, I'm effectively saying "If you're using htags, sucks to be you. If
you're willing to spend effort maintaining a version of htags that is
secure, then we might be able to bring it back.

Yes, it's possible that we'll learn we broke things and need to revert.

But I think what you're getting here from a lot of people is a belief
that our community norm strongly favors active development and new
software. And sometimes when features are effectively not maintained in
a manner that we can package them, they go away.

I don't think it's reasonable to defer this to upstream and wait until
upstream removes htags.

I have tried to understand the technical issues. I'm not sure I'm 100%
in agreement in your analysis there, but I'm fairly close. However, I
haven't found the technical issues tend to affect my analysis of what
Debian should do.


I'd strongly urge you to work on a global6 package. I don't care
whether it's called global or global6. I don't think it should include
insecure cgi scripts that are enabled by default. I'd be fine if it
didn't include htags at all.
I'm not saying upload that package now; I'm not saying where to upload
it.
(Although I wouldn't object to an upload to experimental or unstable)
I think having that package ready and keeping options open as long as
possible would help preserve our ability to work through this process.
I hope that would go a long way to addressing people's feelings of
frustration.

Thanks for your consideration,

--Sam
Ron
2016-12-05 01:08:57 UTC
Permalink
Hi Sam,
Post by Sam Hartman
Ron> Hi OdyX,
Post by Didier 'OdyX' Raboud
Hi there,
I've been mostly VAC, and only now found enough time to properly
read through this bug log. In the interest of transparency and to
help the TC reach a consensus (also through understanding what
the other members' understanding, ideas and positions are), here
comes my current understanding of the problem at hand.
As preamble, I will upfront state that I have _not_ looked in the
precise details of the so-called 'htags' functionality, as, the
rest will show, my current viewpoint on the problem is that it
doesn't matter.
Ron> If we run with your proposal, what are you actually suggesting
Ron> we tell the people who'd be upset by the loss of htags without
Ron> notice in Stretch? Because I don't really see how you've
Ron> addressed that here.
Ron, thanks for working with the process.
Thanks for engaging with the question :)

I was a little disappointed that all OdyX had to say about it was:

<OdyX> I've sent a long mail with my opinion, and Ron's answer hasn't
altered my conviction yet.

Because "lalala, I'm not listening" isn't really an answer to a simple
direct question. And definitely not something that I can wrap with
"Sorry, but a group of Debian Developers considered this all in careful
detail, and this is what we have decided", when breaking the news to
affected people.
Post by Sam Hartman
I think that one thing you sometimes find when you try and build
consensus is that your conception of the problem and even the values by
which a solution should be judged is not shared.
I think you and I shared some opinions about the merits of the IETF
model for consensus on #-devel back when people were last exasperated
with Ian's antics and looking to get some new blood and new ways of
thinking into the ctte. So yes, I don't have a problem with my view
of things ending up being what's in the rough - the important part is
that the questions raised have been answered, and that there's rough
consensus those answers are sufficient to actually address the problem
in some manner.

Or more concisely:

"Rough consensus is achieved when all issues are addressed, but not
necessarily accommodated"

https://tools.ietf.org/html/rfc7282, if people aren't already familiar
with that process.
Post by Sam Hartman
As I understand it, you believe we should be evaluating the technical
options and that preserving things that work is of high value.
I think a lot of us believe that get newest code is a presumptive value
especially over the multiple releases time frame.
It's probably a bit more subtle than that, though my context is certainly
going to be coloured by knowing the history of this particular package.

This is a package which required significant patching to bring it up to
what was considered acceptable practice for distro software from its very
first release with debian back in 1999. Some of the issues were just
general problems and bugs, and upstream took patches for those. And some
of them were things he just didn't care about (or understand) at all,
but he welcomed them being maintained separately as distro patches.

So I don't see it as an unusual thing for us to patch the problems that
are effecting distro users.

And I've looked at what the newer versions have added, and it's not like
there's a lot there which could really be described as earth shattering
must-haves.

And I don't think the only answer to fixing what Vincent is complaining
about, or the issues that Wookey looked into more recently is "new
upstream or bust". I'm pretty sure that we could probably fix those
relatively simply with patches to what we have now.

I even asked for enough information to be able to assess and/or do that.
But there was no interest in providing that from Vincent or Punit, and
no other user of ggtags has surfaced to put their hand up, so like most
things with insufficient information and interest, that went no further
so far.


So as a one sentence answer, "my belief" is probably more like "we can
have the best of both worlds for Stretch". If the people expending
energy arguing here, spent just a tiny portion of that on looking into
their *problem* instead of fighting over their line-of-least-work
solution.


I'm not insisting that's what we should do. But it's certainly an
option, and it dodges the bullet of having to say "Sucks to be you"
without any notice at all. And it doesn't take anything away from
the people who want "new upstream or bust" for Stretch, because it
can still be available to them in backports. Without going scorched
earth on the other users, who would have no other option but to
package their own local versions for the duration of Stretch.
Post by Sam Hartman
That is, I disagree with you that I need to address the question of
htags and figure out whether htags users are being impacted.
That might have been true for one release.
But over a longer time frame, the really strong presumption is that we
prefer a version that is being actively developed to one that isn't.
That if people won't step forward and maintain htags, it goes awy.
There's clearly users of it recorded in the BTS, and historically it was
the thing that actually made this software interesting. Far more so than
just the tag search function (which lots of things do at least as well).

And I've already said here previously that personally, I think there are
now things which do what htags does better than htags too, so I'm not
irrationally attached to keeping it at all costs.
Post by Sam Hartman
That's a presumption. If someone gets evidence that htags is heavily
used, we can consider that.
We might even go so far given sufficient evidence to decide that forking
global and never taking a new upstream was the right answer for our
users. That would take a lot of evidence.
The only bit we apparently don't agree on yet, is the best way to test
what the evidence has to say.

I proposed we see who screams early in the Stretch+1 cycle, so that both
affected users and us had time to react if that was a mistake, and/or if
it provoked more people into actively fixing things that won't have
actually bitten them until we do that.

In the meantime, people wanting global6 not-negotiable, already have it
available, just in a different suite. And we can still fix real problems
in this version too if people provide real bug reports about them.

What you are suggesting is we burn our bridges in all suites, and if
that turns out to be a mistake, say "Oops. Too late to fix that for
Stretch now. Come back in a few years time (with patches?)."


But again, what I'm looking for here is a technically sound consensus.
That when people complain, I can refer them to with a straight face.
If I'm really in the rough on that, that's fine, I've never argued here
that it's "my way else we fight bitterly to the death over it".

I just don't want it to be embarrassingly dumb, having no better
justification than some dogma about the importance of version numbers
over all else.

"Sucks to be you, but my hands are tied because 6 > 5" isn't what I
imagine draws people to using Debian, or what we should expect from
diligent maintainers.
Post by Sam Hartman
I disagree with your approach that the people wanting to remove htags
need to show that it is being used. I disagree with your view that over
the timescales involved, people wanting a new upstream need to justify
that.
Yes, removing htags creates a regression.
Yes, I'm effectively saying "If you're using htags, sucks to be you. If
you're willing to spend effort maintaining a version of htags that is
secure, then we might be able to bring it back.
I appreciate you acknowledging and taking ownership of that.

It has been frustrating to watch the argument ignoring it go from
the plainly false "nobody uses it" through the privileged "ok, maybe
people do use it, but they aren't as important as me and my friends".

I found this particularly boggling:

<OdyX> It seems everyon is assuming this 'htags' thing is un-patchable
by anyone. We're patching software on a routine basis in Debian...

Since it uses that argument to say "it's htags users' fault if they
didn't patch it" (when they are on the record for repeatedly trying
to do so upstream) - while at the same time insisting that the only
way to address other bugs for a few niche use cases is to _not_ patch
them, but instead bring in a new upstream warts and all (when those
people are on the record as saying they aren't interested in even
reporting their issues with sufficient detail to help us do that).

And then to express worry about "rewarding bad behaviour" ...
Post by Sam Hartman
Yes, it's possible that we'll learn we broke things and need to revert.
But I think what you're getting here from a lot of people is a belief
that our community norm strongly favors active development and new
software. And sometimes when features are effectively not maintained in
a manner that we can package them, they go away.
I don't think it's reasonable to defer this to upstream and wait until
upstream removes htags.
I have tried to understand the technical issues. I'm not sure I'm 100%
in agreement in your analysis there, but I'm fairly close. However, I
haven't found the technical issues tend to affect my analysis of what
Debian should do.
I'd strongly urge you to work on a global6 package. I don't care
whether it's called global or global6. I don't think it should include
insecure cgi scripts that are enabled by default. I'd be fine if it
didn't include htags at all.
I'm not saying upload that package now; I'm not saying where to upload
it.
(Although I wouldn't object to an upload to experimental or unstable)
I think having that package ready and keeping options open as long as
possible would help preserve our ability to work through this process.
I hope that would go a long way to addressing people's feelings of
frustration.
I'm likewise primarily interested in seeing us arrive at a well
considered consensus about how to proceed.

I'm not particularly worried about whether I "win" any argument here
or not - my ideal is to see the concerns of all sides of this actually
be addressed in the best manner our collective wisdom is able.

In the best case, we'll have a good solution offering something real
for everyone.

In the not so ideal case, you give me a scapegoat to blame when people
scream about what a stupid thing we did.

In the worst case, the consensus is something I consider such an insane
waste of time for such a tiny number of users, that I'll hand it to
someone else to be their nightmare instead of mine. It's not like I
can't maintain my own private version if that's what I ultimately need
to do.


But either way, as long as it's a genuine rough consensus, that doesn't
simply ignore the hard questions, then I'll be happy to facilitate it,
whatever it is. I'm certainly not going to get EXTREMELY ANGRY about
it if things don't go "my way", and do something hostile or malicious
or stupid.

Understanding of the issues and constructive suggestions to address
them are _all_ I've ever asked for here. It saddens me that we still
have people who think the only way to solve things is to rally a bigger
lynch mob than the people who see things differently to what you do,
and try to get what you want by force and attrition and unfounded
rhetorical accusations.

That's not how collaborative projects work.
Post by Sam Hartman
Thanks for your consideration,
Likewise!
Ron
Philip Hands
2016-12-05 21:13:05 UTC
Permalink
Ron <***@debian.org> writes:

...
Post by Ron
I'm not insisting that's what we should do. But it's certainly an
option, and it dodges the bullet of having to say "Sucks to be you"
without any notice at all. And it doesn't take anything away from
the people who want "new upstream or bust" for Stretch, because it
can still be available to them in backports.
Perhaps you'd be kind enough to either confirm or correct my perceptions
of the current situation:

Version 6 includes a CGI script that one is expected to install in a
manner so hopelessly insecure that we'd not accept it in Debian.

However, it is possible to generate static content that achieves the
same aims, but at the cost of approximately doubling the disk usage,
and presumably also requiring more time to generate.

Also, for people that want personal access to htags there is a
htags-server command that brings up a dedicated server to run the CGI
as the invoking user, by default bound to a localhost port.

Version 6 fixes some bugs that are still present in your version 5
package and/or provides features that are missing, but bug reports of
sufficient quality to allow you to fix/backport to v5 are lacking.

Is that about right?

Are there any other regressions that you are aware of in v6?

Your suggestion, as I understand it, is that v6 should hit unstable
after stretch's release, and that people who are currently complaining
about bugs/missing-features in v5 (or are overly focused on numbers) can
then grab v6 out of stretch-backports.

Could you consider the relative merits of instead putting v6 into
stretch now, and dealing with the people that are currently clinging to
v5 by pointing out that:

0) there are now other alternatives to htags that might suit them better.

1) htags-server lets them run the CGI for local access.

2) htags can generate static content, and thus safely provide general
access while avoiding the need for a CGI

3) If there is anyone that cannot do either for some reason, they can
install global v5 from jessie, pin it to avoid upgrades, and report
the reasons why they had to do this to the BTS.

Have I missed some vital aspect of this?

Is there a compelling reason to favour the theoretical users that might
want to stick with v5 over the actual users that we've been hearing ask
for v6?

If the TC were to achieve consensus that v6 should be in stretch, will
it be sufficient for us to inform you of that in order to make it so?

I gather from what you just wrote that it would be sufficient.

If however you are likely to insist on resolutions to override the
maintainer, or transfer the maintainership, I think you ought to be
up-front about that in order to avoid any accusations of intentional
time-wasting later on.

If you can keep your answers brief, you'll earn my gratitude.

Cheers, Phil.
--
|)| Philip Hands [+44 (0)20 8530 9560] HANDS.COM Ltd.
|-| http://www.hands.com/ http://ftp.uk.debian.org/
|(| Hugo-Klemm-Strasse 34, 21075 Hamburg, GERMANY
Ron
2016-12-08 13:02:44 UTC
Permalink
Post by Philip Hands
Post by Ron
I'm not insisting that's what we should do. But it's certainly an
option, and it dodges the bullet of having to say "Sucks to be you"
without any notice at all. And it doesn't take anything away from
the people who want "new upstream or bust" for Stretch, because it
can still be available to them in backports.
Perhaps you'd be kind enough to either confirm or correct my perceptions
I'll try to be concise, but I might fail at 'brief', since I think it's
important to fairly summarise all sides of this. Leaving people to
somehow reconcile narrowly polarised, conflicting sound bites isn't very
helpful. And since I don't have the option to abstain from an opinion,
and aren't starting from an immutable one, I want to be as sure as I can
that _I've_ got the whole picture clear in my head before settling on
what I think is really best too.

If I show how I got there, and you disagree with my rationale, then I
have something that _I_ can rethink to maybe agree with you. If all
I do is state a conclusion, and handwave the details of how I got there,
then at best we can agree to disagree, and at worst we get more random
trolls throwing mindless insults at me again.
Post by Philip Hands
Version 6 includes a CGI script that one is expected to install in a
manner so hopelessly insecure that we'd not accept it in Debian.
For the version that Punit and Vincent wanted me to let them upload
and that people complained that I nacked, which is where this appeal
to the ctte started from, that's absolutely true. Not only did it
have the 'chmod 777' interface to enable it, it had little gems in it
like this too:

open(PIPE, '@globalpath@' . " --result=ctags-xid $flags $pattern |");

Which for those who don't speak it, is perl for "anyone can execute
arbitrary shell commands by typing them into a web browser", since
$pattern is an unsanitised, untrusted, input from the query string.
The version Punit published and suggested people should use had all
of that enabled in it - and he did that after I'd warned him there
was trouble that shouldn't just be ignored.


That was the shared state and history when this ctte bug was opened.
In the interests of seeing what new options we had _today_, I then did
an initial review of the most recent upstream state, which nobody else
had yet looked at, to see if anything material had changed which might
improve the situation. My second post here included details of that,
but the short answer is "it got more complicated".

That one had removed the ability to run it from a secure system CGI
location at all, and changed the way that it's generated yet again.
It also made changes that guarantee everyone will need to fork it
for distro use, because it now hardcodes a fun mix of paths, like
/opt/local/bin for perl and /usr/local for global et al.

It does comment out that line from above with a note:
"To use this code, please uncomment in your own responsibility."
But it also outputs a .htaccess enabling execution in the directory
where the output is generated, whether you want to use it from there
or not (and adds a second CGI, and a bunch of jquery doing AJAX too).

It has a bunch of immediately obvious problems:

- it still passes the unsanitised $pattern to global, which then
passes it to popen (which again lets it be interpreted by the
shell).

- it won't run with perl's taint mode enabled, because that
simply kills it warning of unsafe operations on untrusted data.

- perl's warnings aren't enabled, and it complains of other
rookie code problems if you do.

- Enabling 'use strict' makes it scream with pain and completely
fail to compile and run.


It would need more thorough auditing to confirm that there is(n't) an
exploitable path through that, but given that it ignores even the most
basic advice which perl has strongly recommended since perl4 was new
and shiny, then if there isn't, it would be because of luck more than
obvious care.

I'd not think it was a Good Plan to push this out as a "rush to beat the
freeze" upload of this source without a much more careful look at it,
and some effort spent actually fixing the already obvious deficiencies,
but the size of the truck you might drive through its holes is a bit
different to where we started from.

It might not be utterly unfixable, but you'd have to convince upstream
that security is important, or completely fork it for debian, which
brings us back to exactly where we are - unless we just remove it all.
But that would need Time, whoever does it.

It is what we now have enabled in the package that Wookey uploaded to
experimental.
Post by Philip Hands
However, it is possible to generate static content ...
Yes.
Post by Philip Hands
... that achieves the
same aims, but at the cost of approximately doubling the disk usage,
and presumably also requiring more time to generate.
No. This isn't just a time-space tradeoff for equivalent functionality
as Punit claimed. You lose the ability to actually search the code,
which is the whole point of doing this in/with global in the first
place.

If you don't have that, you can get much better results by using
doxygen to do the formatting, even if you don't have any doxygen
specific markup in your source at all, and it will also give you
client-side searchability (which these days might well be better
than global's is anyway) ...

So the options for global6 really boil down to:

- Do we try to fork/fix the (new) existing CGI, to make it at
least vaguely secure, modulo not being able to actually
install it to a secure system CGI location.

- Do we try to fork/fix htags to remove that, leaving it with
just (IMO) relatively useless (compared to other options)
functionality to create static markup.

- Do we just remove htags completely, and only ship the tagging
part of global (for Stretch at least, until someone does fix
the problems with it).
Post by Philip Hands
Also, for people that want personal access to htags there is a
htags-server command that brings up a dedicated server to run the CGI
as the invoking user, by default bound to a localhost port.
htags-server is a shell script that generates python or ruby code
to use the HTTP server functionality they can provide.

I have no idea offhand what the security properties of either of
those is is really like, but AIUI the python version at least
claims to run the CGIs as 'nobody' - but I'm not sure how that
works when run as a user unless something is SUID somewhere ...
or if that's patched to work differently in Debian to what the
python docs say it does. If that's true it would mean you need
everything to be at least world readable to use this.

It still unconditionally drops a .htaccess file into the output
though, so if you're running a real web server, that may or may
not grant CGI execute access to that directory, depending on
how it's otherwise configured - whether you use htags-server for
this or not.


That script doesn't really change any of the things that we'd be
concerned about in practice. It just means people who don't have
apache (or another web server) installed can install python or
ruby instead. Either can limit it to localhost or allow access
on a public IP.
Post by Philip Hands
Version 6 fixes some bugs that are still present in your version 5
package and/or provides features that are missing, but bug reports of
sufficient quality to allow you to fix/backport to v5 are lacking.
There appears to be two main issues that we 'know' using v6 'fixes'.

One is whatever it is that the third-party ggtags wrapper needs, which
aiui is what Vincent and Punit are most annoyed about. But I don't
use emacs, and ggtags isn't even in Debian - and they haven't even
told me what error they see, let alone what operation(s) trigger it.

Vincent just gave me the output of global's short --help, and said
"look, there's new options" - but we don't even know if it's actually
a 'missing' command line option that it fails on (or which one that
might be), or something else entirely. My hunch is that one would
probably be pretty trivial to fix - either in ggtags or global, or
both - if someone who uses it engages with what I asked originally,
to file a separate bug from the 'new upstream' one, detailing the
actual problem, and is willing to test any proposed fixes even if
they aren't up to actually submitting a patch for that themselves.
I don't mind writing a patch if we know what actually needs patching.

Or if that really did prove to be intractable to backport, we'd have
a real data point for the question we're looking at here too. But
right now, the odds are good that it's something quite minor, because
not that much has really changed.


The other one, Wookey has now given us a good report for as part of
his recent digging - and at first blush it appears to be 'trivial'
in the sense that it doesn't actually appear to be a bug in global's
code, but rather a hard limit in the version of flex that was used
to generate the scanner. (I do consider it a bug in global that it
isn't regenerated at build time - upstream doesn't even provide
makefile rules to do it, they generate it 'manually' and bundle
the generated file in the source). That one is only still open
because it's going to be an intrusive patch, even if it's 'easy',
and he wasn't keen to try regenerating it himself, and by that
point I wasn't sure if it was going to just be wasted effort, if
the consensus was going to be to kill off v5 right now.

I've got a bunch of things I want to get done that I do expect to
be released, but I can fit this one onto that priority list if we
do decide that keeping v5 for Stretch is the right choice.

I already uploaded a fix for the other good actionable report that
Wookey made while he was looking at all this.
Post by Philip Hands
Are there any other regressions that you are aware of in v6?
In terms of the upstream code, the issues with htags are the main
'showstopper' I'm currently aware of. But I haven't tested the
rest of it anywhere near exhaustively yet either.

In terms of the package currently in experimental, there's a bunch
of issues with it that would need to be fixed if we were going to
want it in Stretch. Wookey already identified some. And in the
quick look that I've already had at it so far, there's others
which were immediately obvious. There's a bunch of upstream stuff
that is simply missing from it, the man page for htags-server, the
icons and javascript and css for htags (which it actually complains
loudly about when you try to use it, and makes me wonder how much
they did actually test that or pay attention to what it reported)
but that becomes moot if we go for the "remove htags completely"
option too.

There's little things like it still having .la files in it, and
static libs for things that are supposedly 'plugins'. Which aren't
a big deal on their own to fix, but again suggests that if 'obvious'
things like that were missed, there's a good chance there will be
more issues than what I've already seen in a cursory review.
Post by Philip Hands
Your suggestion, as I understand it, is that v6 should hit unstable
after stretch's release, and that people who are currently complaining
about bugs/missing-features in v5 (or are overly focused on numbers) can
then grab v6 out of stretch-backports.
If we're going to say that our patience is up with waiting for sanity
to return to htags (and mine pretty much is), then I think that's the
most orderly way we can drain the swamp. With the least risk of having
people step on the unexploded ordinance we'll expose, in ways that we
can't easily fix under stable release constraints.

I don't think it's the only option, but it looks like the least painful
overall for everyone that I can so far think of, which maximises our
options in the event of something unexpected (which ironically, are
black swans we probably should expect ;)
Post by Philip Hands
Could you consider the relative merits of instead putting v6 into
stretch now, and dealing with the people that are currently clinging to
0) there are now other alternatives to htags that might suit them better.
I genuinely think that's actually the best advice we/I can give anyone,
regardless of what our "for the short term" choice ends up being. At
least unless somebody suddenly starts making major improvements to htags
to do things that doxygen can't/doesn't already do on its own.

But I'm at that awkward age, where I'm too old to imagine everyone will
take my advice, but not so cranky to think that anyone who doesn't should
have abuse and scorn heaped upon them before being nuked from orbit ...
So I'm pretty sure someone will still want spacebar heating, whatever we
might say. The difference is just how much warning we give them that we
are taking it away, as to whether we can feel that we were polite and
considerate enough about how we did that or not.
Post by Philip Hands
1) htags-server lets them run the CGI for local access.
2) htags can generate static content, and thus safely provide general
access while avoiding the need for a CGI
I think I already covered those above? (so I'll go back to trying to
be brief and let you ask if there's something I didn't :)
Post by Philip Hands
3) If there is anyone that cannot do either for some reason, they can
install global v5 from jessie, pin it to avoid upgrades, and report
the reasons why they had to do this to the BTS.
That is another option. Though it seems a bit backward from what would
seem like 'normal advice', and means they will fall out of having
security support before Stretch is EOL. It's basically equivalent to
saying "just maintain your own local build". I'm not sure I'd really
want to *suggest* that to people, or get them in the habit of thinking
that it's an ok thing to do when upgrading Debian - if it wasn't an
option they figured out for themselves (and hopefully properly
understood the consequences of).
Post by Philip Hands
Have I missed some vital aspect of this?
Knowing what other people don't know is one of the hardest questions
in the world :) So I hope you'll ask if there's something I didn't
shed the sort of light you were looking for on.

But you're asking good questions, and exploring the same sort of options
that that I'd been mulling over, so I think we're at least in the same
universe, even if we don't quite see it the same.

Optimising for multiple variables is one of the other hardest things
in the world, so I don't see it as a failure of anybody if we don't
(ever) all completely agree on what would be 'best' here. I'll be
happy if enough of us have at least thought about it hard enough to
show their answer is based on the actual variables at play here.
The center of gravity of those options should at least be a local
maxima, if not the ideal solution.
Post by Philip Hands
Is there a compelling reason to favour the theoretical users that might
want to stick with v5 over the actual users that we've been hearing ask
for v6?
That kind of depends on what we decide to do about htags when we move
to v6 (now or later).

That there are users of htags is definitely not theoretical, there was
a new bug reported against it just yesterday - and unlike some of the
more strident voices in this bug log, it's from an actual user of
global, who I've never spoken to before, and wasn't astroturfed in as
'reinforcement'.

If we tell him, "we fixed your bug, by removing htags", I'm pretty
sure his first thought isn't going to be "thank you!". If we tell
him, "we fixed your bug, and we removed the insecure CGI, but you
can still generate static markup that you can't search" - I don't
know what his answer will be, because he hasn't told us if he does
use that or not. Similarly if we rush a 'blind' patch into Stretch
to remove that and accidentally break something it would be hard to
fix because there wasn't enough time spent on testing it all, we're
somewhere on the spectrum of Suboptimal.
Post by Philip Hands
If the TC were to achieve consensus that v6 should be in stretch, will
it be sufficient for us to inform you of that in order to make it so?
I gather from what you just wrote that it would be sufficient.
Yeah, if we've got enough people willing to put their neck on the block
to say "I think we're better off with v6-only in Stretch, rather than
both via the magic of -backports, and here's why the people who might
get burned by that, and the problems that can/will create, aren't
important enough to exercise any patience to minimise them", then I'd
be a fool not to jump at the chance to let them take the rap for
whatever backlash might occur :)

What I'd _like_ to see a good consensus on though, is a little more
than that - because I don't think "should we ship v6 in Stretch" is
the right question, or rather a _sufficient_ question. Because if
the answer to that is "yes" - then we still have the question of
"what should we do with htags in v6", and that's actually the one
where things go all pear-shaped if we get it wrong.

And even if the answer to it is "no", then that question _still_
remains as "what should we do with htags in v6 for stretch+1" ...


So if we instead ask "what should we do with htags for Stretch
(and beyond)", then everything else pretty much follows on
automatically from that.

If we say it's important to keep it as fully functional as it is
now through the Stretch cycle, then really the only practical
option is to stay with v5 for that. v6 has gutted too much of
that to realistically fix in the time we have (or perhaps ever
if upstream continues to fight that).

If we say it's ok to lose the search function of it, or to just
lose it altogether since the search was what made it useful -
then we have nothing to gain by staying with v5 and should ship
v6, and accept whatever fallout occurs from us doing that without
any notice.


Maybe that's what I somehow failed to communicate earlier when I
kept saying htags was the important question I wanted to see get
addressed.

Because people saying "it's irrelevant, uploading 'something newer'
is overwhelmingly more important" completely misses the point that
uploading something newer unavoidably involves someone answering
that question.

And ignoring the issues around it totally leads to fun paradoxes
like Odyx saying that one of the reasons it's important to have v6
is that it "fixes bugs like #590053" - when what the reporter of
that bug wanted fixed is _exactly_ what we would lose by going to
the htags from v6, and he was one of the people who also spent a
considerable amount of time trying to work with upstream to have
a secure system install of the CGI supported ...

Or like Ian's faster than light "no brainer", where if we'd followed
his advice, and made a snap decision without thinking to just let
Vincent upload what Punit had prepared, because Clearly they were
more capable than Evil Me of making an appropriate decision - then
we'd have had a package in stretch with a trivial remote shell
exploit in it ...

Sam's argument that "we don't care" might be justifiable given the
time that's elapsed and the direction upstream has trended in, but
he hasn't yet given any preference for whether we remove htags
entirely, or just try to amputate the CGI - which only gets us
halfway to _what_ we should upload if we go with v6.

I think Tollef and I are somewhere near to on the same page, in that
we think it would suck to burn existing users on short notice, but
agree we can't prioritise them forever, and are open about exactly
how we proceed from here.

I get the feeling that you are leaning toward preferring v6, but
you're asking good questions, so I am interested in how you weigh
all this up.


But I think if we can get a sense of people's preference for what
to do with htags, then we collapse the waveform of what logically
follows from that - and if we don't have as many opinions as we
have people, a reasonable consensus should be fairly quickly
reachable from there, with a self-evident plan of action. And
faster than it has while each person has focussed on their own
personal subset of What Is Most Important here.
Post by Philip Hands
If however you are likely to insist on resolutions to override the
maintainer, or transfer the maintainership, I think you ought to be
up-front about that in order to avoid any accusations of intentional
time-wasting later on.
I'm pretty sure that if wasting time was my intention, there'd be
far more fun ways to do it than this!

I know people here love their FAOD clarifications, so it helps you
for us to be crystal clear about that, but it does make me sad that
people do project those sort of accusations onto others. Especially
onto others who *are* being diligent about making good decisions.

I'm taking the time to give detailed answers to good questions
because that's how good decisions get made. Anyone who thinks that
'threats' of "over my dead body" end with anything better than
dead bodies, still has a lot to learn about how the world works.

I'm pretty sure we can agree on some reasonable plan without the
need for a vote, or to invoke the constitution to force anything.
And I can guarantee you that on the remote chance that I'm wrong
about that, and do violently disagree with some insane consensus
that emerges - that I won't for a moment stand in the way of
letting someone else be the person whose face that blows up in.
If it comes to that, they'll have earned it!
Post by Philip Hands
If you can keep your answers brief, you'll earn my gratitude.
Sorry :( I'll have to find some other way to earn that, if proving
that your diligence in learning about this before deciding was well
founded doesn't do it.

Best,
Ron
Vincent Bernat
2016-12-08 13:39:44 UTC
Permalink
Post by Ron
One is whatever it is that the third-party ggtags wrapper needs, which
aiui is what Vincent and Punit are most annoyed about. But I don't
use emacs, and ggtags isn't even in Debian - and they haven't even
told me what error they see, let alone what operation(s) trigger it.
Vincent just gave me the output of global's short --help, and said
"look, there's new options" - but we don't even know if it's actually
a 'missing' command line option that it fails on (or which one that
might be), or something else entirely. My hunch is that one would
probably be pretty trivial to fix - either in ggtags or global, or
both - if someone who uses it engages with what I asked originally,
to file a separate bug from the 'new upstream' one, detailing the
actual problem, and is willing to test any proposed fixes even if
they aren't up to actually submitting a patch for that themselves.
I don't mind writing a patch if we know what actually needs patching.
Or, more likely, just like for #587489, #613011, #182188, #212040,
#218111, #669617, #756367 and #130091 (more than half of the open bugs
for global), you will just ignore the bug report. I have not included
#715980 and #716006 which are automated. I don't understand how you can
try to paint yourself as someone who cares about your users while there
is this public track record of your unwilligness to discuss with
anybody.

And for this particular case, you didn't asked for anything more. You
just said nothing.
--
The surest protection against temptation is cowardice.
-- Mark Twain
Ron
2016-12-08 14:02:13 UTC
Permalink
Post by Vincent Bernat
Post by Ron
One is whatever it is that the third-party ggtags wrapper needs, which
aiui is what Vincent and Punit are most annoyed about. But I don't
use emacs, and ggtags isn't even in Debian - and they haven't even
told me what error they see, let alone what operation(s) trigger it.
Vincent just gave me the output of global's short --help, and said
"look, there's new options" - but we don't even know if it's actually
a 'missing' command line option that it fails on (or which one that
might be), or something else entirely. My hunch is that one would
probably be pretty trivial to fix - either in ggtags or global, or
both - if someone who uses it engages with what I asked originally,
to file a separate bug from the 'new upstream' one, detailing the
actual problem, and is willing to test any proposed fixes even if
they aren't up to actually submitting a patch for that themselves.
I don't mind writing a patch if we know what actually needs patching.
And for this particular case, you didn't asked for anything more. You
just said nothing.
Post by Ron
I am using gg-tags in Emacs and the current version of global in
Debian just doesn't work with this mode.
What changed incompatibly to make it not work? And what would need
patching to fix that?
I'd really much rather see problems get fixed than layered under even
more problems. If someone familiar with Emacs has some details of
what doesn't work, and what needs to be done so that it will, that
sounds like a separate bug to be addressed to me.
Some arguments seem to not exist in previous versions. I did not
investigate more


How much am I supposed to hound you when you give a non-answer?
I've asked you again several times here, and each time you put in a
bunch of work to trot out some new accusations, but do nothing to
actually answer the question.
Vincent Bernat
2016-12-08 14:41:14 UTC
Permalink
Post by Ron
How much am I supposed to hound you when you give a non-answer?
Maybe assume good faith and tell me that the answer doesn't fit you?
But, no, that was the last time you were willing to say anything in the
bug report.
--
Don't compare floating point numbers just for equality.
- The Elements of Programming Style (Kernighan & Plauger)
Ron
2016-12-08 16:05:42 UTC
Permalink
Post by Vincent Bernat
Post by Ron
How much am I supposed to hound you when you give a non-answer?
Maybe assume good faith and tell me that the answer doesn't fit you?
You said you weren't interested in debugging it further, and so did
Punit - how is it not assuming good faith to take what you said at
face value? You're the only two users of ggtags that I know of.

But I can repeat it again one more time. Pretty please with sugar
on top can we have a real bug report, with real information to let us
assess whether your problem can be fixed with a trivial patch or not?

I'm assuming in good faith that you do know what a proper bug report
should look like, but I can give you some pointers in good faith
if you don't. Looking at the bugs that Wookey filed should be a
good guide.
Vincent Bernat
2016-12-09 06:33:50 UTC
Permalink
Post by Ron
Post by Vincent Bernat
Post by Ron
How much am I supposed to hound you when you give a non-answer?
Maybe assume good faith and tell me that the answer doesn't fit you?
You said you weren't interested in debugging it further, and so did
Punit - how is it not assuming good faith to take what you said at
face value? You're the only two users of ggtags that I know of.
I said I weren't interested in debugging it further two months ago (and
I am still not interested). I didn't say such a thing during my initial
participation more than two years ago.

There is no amount of distraction that you can do _now_ that would hide
the inactivity you had during all those years. Even people reporting
easily fixable packaging-related problems with patches were ignored:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=587489
--
The lunatic, the lover, and the poet,
Are of imagination all compact...
-- Wm. Shakespeare, "A Midsummer Night's Dream"
Didier 'OdyX' Raboud
2016-12-08 16:03:40 UTC
Permalink
Just commenting to some specific points as, despite an explicit request, you
have failed (and spectacularly so) to provide brief answers. That's not
helping a quick and clear path towards resolution.
Post by Philip Hands
Perhaps you'd be kind enough to either confirm or correct my perceptions
Version 6 includes a CGI script that one is expected to install in a
manner so hopelessly insecure that we'd not accept it in Debian.
For the version (
) that I nacked, which is where this appeal to the ctte
started from, that's absolutely true. Not only did it have the 'chmod 777'
Which for those who don't speak it, is perl for "anyone can execute
arbitrary shell commands by typing them into a web browser", since
$pattern is an unsanitised, untrusted, input from the query string.
If you haven't yet, I urge you to use our standard interface to report such
bugs; please make sure issues like this one are public on our bugtracker, with
correct found/notfound version markers.

This also applies to group who has uploaded the experimental version: please
version-close bugs that this version fixes.

For that specific Perl problem, I'd love to be enlightened in how the version
in 6.5.5 is significantly worse than the code in 5.7.1-3's global.cgi.tmpl:

http://sources.debian.net/src/global/5.7.1-3/htags/global.cgi.tmpl/?
hl=152#L152
It also made changes that guarantee everyone will need to fork it
for distro use, because it now hardcodes a fun mix of paths, like
/opt/local/bin for perl and /usr/local for global et al.
That's a _very_ typical distro maintainer's job, really.
It is what we now have enabled in the package that Wookey uploaded to
experimental.
At least now we have a version in Debian we can compare with. Noone has
claimed it would be perfect, but uploading this new version (to whichever
suite, really) was your duty as maintainer and you failed to do so in 6+
years.
Post by Philip Hands
Are there any other regressions that you are aware of in v6?
In terms of the upstream code, the issues with htags are the main
'showstopper' I'm currently aware of. But I haven't tested the
rest of it anywhere near exhaustively yet either.
In terms of the package currently in experimental, there's a bunch
of issues with it that would need to be fixed if we were going to
want it in Stretch.
Please file these as bugreports, with appropriate severities. This is the only
way we will all be able to have a clear picture of what makes each version the
most suitable to release in Stretch.
There's little things like it still having .la files in it, and
static libs for things that are supposedly 'plugins'. Which aren't
a big deal on their own to fix, but again suggests that if 'obvious'
things like that were missed, there's a good chance there will be
more issues than what I've already seen in a cursory review.
I don't really appreciate how you're sniping at the maintainers and uploaders
of the version currently in experimental, while doing the job to keep up with
recent versions of global was _your_ duty all along. What about actually
_helping_ making this global version as good as you think it should be?
What I'd _like_ to see a good consensus on though, is a little more
than that - because I don't think "should we ship v6 in Stretch" is
the right question, or rather a _sufficient_ question. Because if
the answer to that is "yes" - then we still have the question of
"what should we do with htags in v6", and that's actually the one
where things go all pear-shaped if we get it wrong.
And even if the answer to it is "no", then that question _still_
remains as "what should we do with htags in v6 for stretch+1" ...
The question was supposed to be asked, and answered in september 2011, when
global 6.0 was released. This question should have been answered for squeeze,
eventually wheezy, really.
Because people saying "it's irrelevant, uploading 'something newer'
is overwhelmingly more important" completely misses the point that
uploading something newer unavoidably involves someone answering
that question.
Uploading newer versions of upstream software is constantly about compromises,
functionality losses, and new functionalities. That's just how software
development works, and Debian is constantly following up on the new versions
of all sorts of software. With your approach, we'd have Gnome 2, KDE 3,
PostgreSQL 8.4 and GCC 4.4 in stretch, just with some compatibility patches,
because we'd have been too conservative. That's not the sort of Debian
releases I want to see.
And ignoring the issues around it totally leads to fun paradoxes
like OdyX saying that one of the reasons it's important to have v6
is that it "fixes bugs like #590053" - when what the reporter of
that bug wanted fixed is _exactly_ what we would lose by going to
the htags from v6, and he was one of the people who also spent a
considerable amount of time trying to work with upstream to have
a secure system install of the CGI supported ...
Actually, that bug is a very good example: the request is about a problem
fixed by upstream in 5.9. We still have 5.7, and this bug (filed in 2010,
before the freeze of _squeeze_) is not fixed in the version we're about to
ship in Stretch.

If all the problems come from "the htags from v6", what is blocking you from
at least providing the latest 5.x versions? If you _had_ backported whatever
fixed the #590053 bug from 5.9 to your 5.7 version, I could accept your global
version freeze. Point is, you haven't (we're talking about a 5+ years old
bug), and you continue to pretend there are unfixable bugs in 6.x. It seems to
me you're just _not_ interested in providing any other version than this 5.7.1
one, and I don't understand why.
I'm pretty sure we can agree on some reasonable plan without the
need for a vote, or to invoke the constitution to force anything.
According to popcon and apt, 'global' is a unimportant package, with few users
and no reverse dependencies, and we all have now spent way too much time
discussing its maintainership, frankly.

I can sympathize with some of your arguments:
- version numbers are not silver-bullets, and updating to new versions is not
a goal per-se;
- there are hard problems to solve in new versions of 'global'
- waiting another release will give you time to think a transition plan
through.

But I disagree that:
- it's fine to keep the Debian package several versions behind, over multiple
releases;
- the hard problems are impossible to fix;
- regressions or functionality losses brought by upstream changes must not be
inflicted upon our users;

I don't know yet what the good decision for the future maintenance of
src:global is, but I'm convinced that you have actually made a disservice to
'global' users, for way too long, and I want this to stop.

Sniping at the people who have gone through some effort to actually upload a
v6 version is not helping, at all. You should be grateful that they have
produced this effort, and _help_ them making this version of a package you're
the maintainer of. This was your duty all along.

I think a better service would be done by other maintainers, deferring the 'v6
in stretch' question to them. I'll start drafting a proposal along these lines
tomorrow.

Wookey, Vincent, Punit: would any of you be willing to receive the 'global'
package maintainer hat? (which would of course come with the possibility to
share and change the maintainer status afterwards.)

OdyX
Tollef Fog Heen
2016-12-08 17:14:12 UTC
Permalink
]] Didier 'OdyX' Raboud
Post by Didier 'OdyX' Raboud
Post by Philip Hands
Perhaps you'd be kind enough to either confirm or correct my perceptions
Version 6 includes a CGI script that one is expected to install in a
manner so hopelessly insecure that we'd not accept it in Debian.
For the version (…) that I nacked, which is where this appeal to the ctte
started from, that's absolutely true. Not only did it have the 'chmod 777'
Which for those who don't speak it, is perl for "anyone can execute
arbitrary shell commands by typing them into a web browser", since
$pattern is an unsanitised, untrusted, input from the query string.
If you haven't yet, I urge you to use our standard interface to report such
bugs; please make sure issues like this one are public on our bugtracker, with
correct found/notfound version markers.
This also applies to group who has uploaded the experimental version: please
version-close bugs that this version fixes.
For that specific Perl problem, I'd love to be enlightened in how the version
http://sources.debian.net/src/global/5.7.1-3/htags/global.cgi.tmpl/?
hl=152#L152
It's completely different. It's basically system(3) on a concatenated
string with partial user-defined content vs execve(2) on a list of
arguments (some of which are user-provided).

perldoc -f exec and perldoc -f open might be useful.

Using open like in the code snippet above is pretty much inexcusable in
this day and age.
--
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are
Didier 'OdyX' Raboud
2016-12-08 17:24:32 UTC
Permalink
Post by Tollef Fog Heen
Using open like in the code snippet above is pretty much inexcusable in
this day and age.
Fair enough, thanks for the explanation.

Ron: could you point us to your report of this problem to the upstream
bugtracker or list? What was the answer you got?
--
Cheers,
OdyX
Ron
2016-12-08 18:45:11 UTC
Permalink
Post by Didier 'OdyX' Raboud
Post by Tollef Fog Heen
Using open like in the code snippet above is pretty much inexcusable in
this day and age.
Fair enough, thanks for the explanation.
Ron: could you point us to your report of this problem to the upstream
bugtracker or list? What was the answer you got?
I didn't audit that code exhaustively when Punit proposed uploading it,
there were already enough things obviously wrong with what he was
suggesting to go through all of it with a fine toothed comb to find more
before he'd shown any interest in addressing the first lot.

But it stood out like a sore thumb when I was fact checking the answer
to Phil's question about the CGI being a hopeless case, to be sure that
my answer was as accurate as possible over the range of changes that
have happened to it.

It certainly seems like something that anyone professing that they
should be trusted to maintain this probably should have been looking
at when the red flags went up about upstream's idea of what is
adequately secure ...
Ron
2016-12-08 18:25:20 UTC
Permalink
Post by Didier 'OdyX' Raboud
Just commenting to some specific points as, despite an explicit request, you
have failed (and spectacularly so) to provide brief answers. That's not
helping a quick and clear path towards resolution.
Now I feel like goldilocks, first I'm bad because I didn't respond enough,
now I'm bad because I respond too much.

But apparently, I should have actually said just a little more in this
one too, to explain to you how perl works :) So I'll do that now!
Post by Didier 'OdyX' Raboud
Post by Philip Hands
Perhaps you'd be kind enough to either confirm or correct my perceptions
Version 6 includes a CGI script that one is expected to install in a
manner so hopelessly insecure that we'd not accept it in Debian.
For the version (…) that I nacked, which is where this appeal to the ctte
started from, that's absolutely true. Not only did it have the 'chmod 777'
Which for those who don't speak it, is perl for "anyone can execute
arbitrary shell commands by typing them into a web browser", since
$pattern is an unsanitised, untrusted, input from the query string.
If you haven't yet, I urge you to use our standard interface to report such
bugs; please make sure issues like this one are public on our bugtracker, with
correct found/notfound version markers.
Do you really want entries in the tracker for buggy code that was never
in Debian, because I nacked Punit uploading things he didn't understand
with a vague promise to maybe look at them later?
Post by Didier 'OdyX' Raboud
This also applies to group who has uploaded the experimental version: please
version-close bugs that this version fixes.
For that specific Perl problem, I'd love to be enlightened in how the version
http://sources.debian.net/src/global/5.7.1-3/htags/global.cgi.tmpl/?hl=152#L152
Ok, you don't speak perl ...

http://perldoc.perl.org/functions/open.html
http://perldoc.perl.org/perlsec.html

But in this case, it's also not that different to shell best practice.

"You would want to use the list form of the pipe so you can pass
literal arguments to the command without risk of the shell
interpreting any shell metacharacters in them."
Post by Didier 'OdyX' Raboud
It also made changes that guarantee everyone will need to fork it
for distro use, because it now hardcodes a fun mix of paths, like
/opt/local/bin for perl and /usr/local for global et al.
That's a _very_ typical distro maintainer's job, really.
But still a regression in this code where that previously wasn't needed.
Post by Didier 'OdyX' Raboud
There's little things like it still having .la files in it, and
static libs for things that are supposedly 'plugins'. Which aren't
a big deal on their own to fix, but again suggests that if 'obvious'
things like that were missed, there's a good chance there will be
more issues than what I've already seen in a cursory review.
I don't really appreciate how you're sniping at the maintainers and uploaders
of the version currently in experimental, while doing the job to keep up with
recent versions of global was _your_ duty all along. What about actually
_helping_ making this global version as good as you think it should be?
How is pointing out real issues with it "sniping at them"?

We have people here saying what a wonderful and perfect job they are
doing, while slagging off at me. Including yourself.

I didn't see you telling them that sort of thing is not appreciated.

Last I heard it was considered bad form to change the package format
in an NMU too if you want to talk about "duty".
Post by Didier 'OdyX' Raboud
What I'd _like_ to see a good consensus on though, is a little more
than that - because I don't think "should we ship v6 in Stretch" is
the right question, or rather a _sufficient_ question. Because if
the answer to that is "yes" - then we still have the question of
"what should we do with htags in v6", and that's actually the one
where things go all pear-shaped if we get it wrong.
And even if the answer to it is "no", then that question _still_
remains as "what should we do with htags in v6 for stretch+1" ...
The question was supposed to be asked, and answered in september 2011, when
global 6.0 was released. This question should have been answered for squeeze,
eventually wheezy, really.
The question *was* asked (by me), repeatedly. Nothing material changed
to alter the problem and hence the answer.

Now we're talking about what to do among a wider group of people, given
that it still looks like nothing material will change. The system works?
Post by Didier 'OdyX' Raboud
And ignoring the issues around it totally leads to fun paradoxes
like OdyX saying that one of the reasons it's important to have v6
is that it "fixes bugs like #590053" - when what the reporter of
that bug wanted fixed is _exactly_ what we would lose by going to
the htags from v6, and he was one of the people who also spent a
considerable amount of time trying to work with upstream to have
a secure system install of the CGI supported ...
Actually, that bug is a very good example: the request is about a problem
fixed by upstream in 5.9. We still have 5.7, and this bug (filed in 2010,
before the freeze of _squeeze_) is not fixed in the version we're about to
ship in Stretch.
And this _again_ is precisely why you thinking that it's unimportant to
understand the problem, and that even if you don't you can leap to wild
conclusions, is troublesome.

That report led to both me and the reporter having a (very) long
discussion with upstream about how to resolve the real problem that
you keep assuming we never tried to do anything about.

Nothing material changed to improve this after that discussion either.


I don't blame you for not knowing that. But leaping to your own
conclusions without asking isn't helping anyone understand better.
Post by Didier 'OdyX' Raboud
If all the problems come from "the htags from v6", what is blocking you from
at least providing the latest 5.x versions?
We are providing the latest 5.x version that didn't break the interfaces
we need. I'm talking about v6 here, because v6 is what we are talking
about moving to next.
Post by Didier 'OdyX' Raboud
If you _had_ backported whatever fixed the #590053 bug from 5.9 to your
5.7 version, I could accept your global version freeze.
Yeah, that one is a fair cop. It probably just slipped through the
cracks because both Taisuke and I were utterly exhausted after the
discussion with upstream. I forgot about it, and he never chimed
in again to remind me (though we have talked separately again since).

I can add that one to the list if we do look at keeping v5 for a
bit longer.
Post by Didier 'OdyX' Raboud
Point is, you haven't (we're talking about a 5+ years old bug), and
you continue to pretend there are unfixable bugs in 6.x.
Why do you think it's "pretending"? I really would like to know
which bit of this you really still don't understand?
Post by Didier 'OdyX' Raboud
It seems to me you're just _not_ interested in providing any other
version than this 5.7.1 one, and I don't understand why.
What I'm not interested in, is repeating the same shitfight over and
over, with it never getting even an inch closer to addressing the
problems that have us stalled there.

But I don't see why you claim I'm not interested in finding a way
to actually move past this. I said right from the very beginning
that while it was clear there was no easy way to move to a later
version without breaking significant functionality, that historically
was a very important (even the most important) part of this package,
I thought that having this come to the ctte was actually a good
opportunity to get a broader consensus about how we ought to deal
with that.

Which part of that don't you understand? I'd like to fix that,
because it seems to be a major wall you've put up to us having
the sort of conversation we should be having, instead of me
having to fend off accusations based on imagined things.
Post by Didier 'OdyX' Raboud
I'm pretty sure we can agree on some reasonable plan without the
need for a vote, or to invoke the constitution to force anything.
According to popcon and apt, 'global' is a unimportant package, with few users
and no reverse dependencies,
Which is another reason it's suffering for attention on many fronts
including upstream. It has always been a very niche package, with
very few users, all of whom are supposedly software developers.

So if people like that can't or don't send patches, and/or are unable
to get them accepted upstream - and it's not even like they report a
lot of bugs (and that's not because they don't exist) ...

And that it is no longer 'state of the art' at what it does ...

It's state shouldn't really be all that surprising. And somehow
blaming me for that seems a bit disingenuous.


Punit promised to maintain his fork too, and it never saw even a
single commit after the day he announced that, when he'd scratched
his original itch. He promised to deal with the insecure CGI in
his fork, but never did. Vincent claimed that he already had, and
he couldn't remember whether he had or hadn't. And they still
didn't do that before uploading to experimental.

But somehow that is ok, but I'm the bad guy?
Post by Didier 'OdyX' Raboud
- version numbers are not silver-bullets, and updating to new versions is not
a goal per-se;
- there are hard problems to solve in new versions of 'global'
- waiting another release will give you time to think a transition plan
through.
- it's fine to keep the Debian package several versions behind, over multiple
releases;
I've never said this was 'fine'. Quite the opposite really.
I've said we didn't have a good answer. And the only alternative that
anyone previously proposed was "the problem doesn't effect me personally,
what if we just pretend it doesn't exist"?
Post by Didier 'OdyX' Raboud
- the hard problems are impossible to fix;
I've laid out the options we have. You can't pull a fish hook out
of your finger without it hurting. Someone is going to get hurt.
Previously, I chose to avoid hurting the people who _had_ been
contributing to trying to improve this, and to preserve a feature
that historically was The Thing this was good at and for.

Some people who weren't contributing complained that we should hurt
those people anyway if it means they didn't need to help with their
own patches. They said the other group of people didn't exist or
weren't important.

I've suggested a way that spreads the pain as fairly as we can.
I've been asking if we have consensus for that, or if not, what
this group of people is prepared to take responsibility for
deciding as a way forward.
Post by Didier 'OdyX' Raboud
- regressions or functionality losses brought by upstream changes must not be
inflicted upon our users;
We solve that problem with Debian specific things all the time.
Even systemd diverges from upstream to avoid inflicting those things
on our users.

It's not _always_ possible, but when we can do it, it is one of the
things that makes Debian awesome. There are _always_ tradeoffs.
You don't like the one I picked, that fine.

But you don't get to complain about that if this is the first time
you've ever talked to me about it, _and_ you've not bothered to
actually learn the reasons why things are like they are.


And you're still not _actually_ addressing the problem at all.
You're just scapegoating me, and hoping that if you give the problem
to someone else, then you won't have to understand it and make a
decision about what _you_ think is the right tradeoff.

That's not really a way to encourage anyone else to consult the
ctte for advice on how to break hard stalemates in the future.
And I was really hoping we could kill that bird with this stone
too, and show people by example that there is a better way to
handle these sort of things... That the ctte could actually offer
_technical_ advice arrived at by consensus, rather than just be
arbiters of schoolyard spats.

Wouldn't that be a genuinely nice outcome all round?
Didier 'OdyX' Raboud
2016-12-09 10:58:02 UTC
Permalink
Post by Ron
Post by Didier 'OdyX' Raboud
If you haven't yet, I urge you to use our standard interface to report such
bugs; please make sure issues like this one are public on our bugtracker,
with correct found/notfound version markers.
Do you really want entries in the tracker for buggy code that was never
in Debian, because I nacked Punit uploading things he didn't understand
with a vague promise to maybe look at them later?
That code is now in Debian (experimental), so yes, I do expect you to act in
good faith and report bugs you see. You are obviously quite versed in how
'global' works, and that's undoubtedly valuable to produce the best possible
'global' package.
Post by Ron
Now we're talking about what to do among a wider group of people, given
that it still looks like nothing material will change. The system works?
It doesn't: it shouldn't take 3 stable releases to get a new upstream release
for a leaf package.
Post by Ron
That report led to both me and the reporter having a (very) long
discussion with upstream about how to resolve the real problem that
you keep assuming we never tried to do anything about.
By "(very) long discussion", do you mean these 8 mails ?

http://lists.gnu.org/archive/html/bug-global/2010-08/threads.html#00006

As far as I could see, this is the only thread in all of bug-global's (public)
history in which you contributed, hence your only public input to upstream
about how >> 5.7.1 versions have problems in your opinion.

Is there another public bug tracker somewhere that I missed?

Did you have other conversations with upstream? If so, where can we find them?
Post by Ron
Post by Didier 'OdyX' Raboud
If all the problems come from "the htags from v6", what is blocking you
from at least providing the latest 5.x versions?
We are providing the latest 5.x version that didn't break the interfaces
we need. I'm talking about v6 here, because v6 is what we are talking
about moving to next.
Fair enough. Sorry for assuming the breakage was in from 6.0 on.

I'm purposedly not answering to the rest of your email, as I think the TC now
has enough information to issue a decision.
--
Cheers,
OdyX
Sam Hartman
2016-12-09 13:58:10 UTC
Permalink
Didier> That code is now in Debian (experimental), so yes, I do
Didier> expect you to act in good faith and report bugs you see. You
Didier> are obviously quite versed in how 'global' works, and that's
Didier> undoubtedly valuable to produce the best possible 'global'
Didier> package.

Ron, I would prefer that Didier use a different tone.
However, it's my opinion as someone who will be voting on this that
Didier is essentially right that your view of this situation and of the
responsibilities of a Debian maintainer are inconsistent with the
project as a whole.
You have continued to try and frame the discussion in the terms you
would prefer.
I have considered those terms, and I do not find framing the discussion
in those terms compelling.

In the language similar to the IETF, used only because we're both familiar with
it, the technical issues surrounding global-6 and the question of
evidence regarding htags have been considered. My judgment of the
discussion is that the rough consensus here is that those issues are not
significant compared to failing to upgrade global in six years. That
is, we in this discussion have reach an informed opinion that those
issues are not significant enough to block. It is my opinion you are in
the rough.

I think the question before the TC is (and is properly) how should
global-6 be maintained and whether you are the person to do that.

You have tried to frame it arguing that the version number doesn't
matter.
I absolutely agree.
However, the time at which Debian has last synced with upstream does
matter.
Six years is a long time.
Moreover, I believe that the standard you've used to evaluate whether
failing to sync for six years was acceptable is inconsistent with best
practices of the project.

--Sam
Colin Watson
2016-12-09 15:03:14 UTC
Permalink
Post by Sam Hartman
In the language similar to the IETF, used only because we're both familiar with
it, the technical issues surrounding global-6 and the question of
evidence regarding htags have been considered. My judgment of the
discussion is that the rough consensus here is that those issues are not
significant compared to failing to upgrade global in six years. That
is, we in this discussion have reach an informed opinion that those
issues are not significant enough to block. It is my opinion you are in
the rough.
I think the question before the TC is (and is properly) how should
global-6 be maintained and whether you are the person to do that.
You have tried to frame it arguing that the version number doesn't
matter.
I absolutely agree.
However, the time at which Debian has last synced with upstream does
matter.
Six years is a long time.
Moreover, I believe that the standard you've used to evaluate whether
failing to sync for six years was acceptable is inconsistent with best
practices of the project.
As a maintainer who has sometimes had cause to do similar things, I'm
concerned at the standard being applied here. Could you perhaps review
the history around groff 1.18.1.1 -> 1.20 for comparison? This is a
case that's all over and done with seven years ago, so should be free of
emotive associations by now, and the history can be read through
reasonably quickly.

Here are some references:
https://bugs.debian.org/196762
https://bugs.debian.org/374569
https://lists.gnu.org/archive/html/groff/2007-11/msg00011.html

Now, my perception of this case is that there was a complex blocking
issue with the new upstream release that affected a minority of users,
and therefore I chose to hold back the new upstream until it could be
handled in a way that did not cause serious regressions. Eventually, by
dint of help from quite a few people, we managed to get it sorted out to
everyone's satisfaction. I think I acted as a responsible maintainer,
even though I wasn't perhaps as energetic as might have been ideal and
not everyone was happy with me along the way.

But, if you so chose, you could take the history and frame it in quite a
different way. In that view, I held back substantial upstream changes
for over six years despite many requests from users to upgrade, in bug
reports, mailing list posts, and personal appeals at conferences,
pleading regressions affecting only a small minority of users. Despite
presenting a superficial appearance of wanting help, I dithered for many
months between status updates, causing RC bugs along the way. A
different maintainer should have been appointed who would have uploaded
groff 1.19, perhaps as a separate source package, and left the CJK users
to maintain the old fork for themselves.

Does that sound familiar?

Now, it's clear which view of events I prefer in the groff case, and I'm
not trying to say that the same thing applies in the global case (to be
quite honest, I've not spent the time required to read the history there
well enough to form a view; one advantage to having quit the TC is that
I get to ignore this sort of thing if I feel like it :-) ). However, I
am concerned about the "don't sync with upstream for six years so you're
a negligent maintainer" principle apparently being formed here, because
from my own history I feel that there can in fact be good reasons for
that.

If the TC thinks my actions in the past were reasonable, then I would
like any ruling here to be a bit more nuanced about cases of delayed
syncing with upstream, reflecting whatever important differences you see
between the two cases.

Thanks,
--
Colin Watson [***@debian.org]
Didier 'OdyX' Raboud
2016-12-09 16:13:48 UTC
Permalink
Post by Colin Watson
Post by Sam Hartman
However, the time at which Debian has last synced with upstream does
matter.
Six years is a long time.
Moreover, I believe that the standard you've used to evaluate whether
failing to sync for six years was acceptable is inconsistent with best
practices of the project.
As a maintainer who has sometimes had cause to do similar things, I'm
concerned at the standard being applied here. Could you perhaps review
the history around groff 1.18.1.1 -> 1.20 for comparison? This is a
case that's all over and done with seven years ago, so should be free of
emotive associations by now, and the history can be read through
reasonably quickly.
Thank you for the example case, including its references.
Post by Colin Watson
Now, my perception of this case is that there was a complex blocking
issue with the new upstream release that affected a minority of users,
and therefore I chose to hold back the new upstream until it could be
handled in a way that did not cause serious regressions.
(
)
If the TC thinks my actions in the past were reasonable, then I would
like any ruling here to be a bit more nuanced about cases of delayed
syncing with upstream, reflecting whatever important differences you see
between the two cases.
By a quick glance, your past actions _were_ indeed reasonable. And commenting
on what differentiates this example with 'global' is a worthwile exercise.

What I see as fundamental difference here was your use of #196762 as a single
point of contact for the problem you were facing with groff 1.19, in which you
explained, commented and followed up on what the problem was, what were your
thoughts, asking for help, and updating this bug regularly. You also used this
Post by Colin Watson
Unfortunately, for technical reasons (see bug #196762), it is extremely
difficult to upgrade to the new upstream release.
All-in-all, although there are appeareances of equivalence between the two
packages' histories, I certainly see fundamental differences in how the "new
upstream version has problems that are hard to fix" question was addressed.

There can be valid, strong and solid technical reasons to hold back on new
upstream versions, but what matters for us as a distribution, is how we
collaborate and communicate about these blockers. Our SC's "We will not hide
problems" is not to be understood strictly as "our bugtracker is public" only;
it is a much larger concern that we should share, so as to make our
collaboration as frictionless as possible, as well as make the technical
problems not only reside in the package maintainer's heads, but to share them
with the project.

That's really the thing I miss most in 'global's history: it shouldn't need a
TC bug to get the 'htags' problem described properly, on a public, standalone
bug.

Things like
Post by Colin Watson
(we) are currently discussing changes to the CGI mechanism which may alter
several things about how this is currently arranged.
in #590053, or the discussion in #574947 where Ron claims to have reached a
private agreement that is not confirmed by upstream authors are un-
comprehensible for outsiders, which are left without a clue upon what needs to
be fixed, or done.

The problem I have with how 'global' is maintained is not at all technical.
Holding back on an old version because of 'htags' breakage _was_ a good
technical decision back then. But the way this hurdle was (not) documented is
just not how I want to see maintainers collaborate and "not hide problems" for
packages in Debian.

The fact that the hurdle is _still not_ properly and publicly documented
doesn't open room for helping the maintainer, doesn't bring clarity to
newcomers and therefore puts the maintainer in sole responsiblity. Wei Lui
Post by Colin Watson
I'm aware of the disagreement between Ron and Shigio, but it's so
frustrating that no resolution has been found for so many years. There
were talks about possible designs proposed by Ron and / or Shigio but
it is just impossible for us users to do archaeology dating back to
1999 with no useful references whatsoever.
Another difference I see is this urge you felt. Two months after groff 1.19
was released, you filed this bug documenting your concerns with it and no less
Post by Colin Watson
I'm beginning to get a bit itchy about falling too far behind upstream
here.
As a matter of fact, I do _expect_ maintainers to get itchy when they fall too
far behind upstream. If Ron had documented the problem from the start (or at
any later point in time, really), users would have found it, and debated ways
out, patches, etc. This bug would have included upstream developers for a
discussion _in public_. That bug would have helped joining forces, helped the
maintainer determine the best consensus to adopt.

And it wouldn't have needed a long TC discussion.
--
Cheers,
OdyX
Ron
2016-12-09 16:37:21 UTC
Permalink
Post by Didier 'OdyX' Raboud
What I see as fundamental difference here was your use of #196762 as a single
point of contact for the problem you were facing with groff 1.19, in which you
explained, commented and followed up on what the problem was, what were your
thoughts, asking for help, and updating this bug regularly. You also used this
...
Post by Didier 'OdyX' Raboud
That's really the thing I miss most in 'global's history: it shouldn't need a
TC bug to get the 'htags' problem described properly, on a public, standalone
bug.
So you're saying "if only #574947 existed which did that"?
Oh wait.
Sam Hartman
2016-12-09 21:00:04 UTC
Permalink
Colin> As a maintainer who has sometimes had cause to do similar
Colin> things, I'm concerned at the standard being applied here.
Colin> Could you perhaps review the history around groff 1.18.1.1 ->
Colin> 1.20 for comparison? This is a case that's all over and done
Colin> with seven years ago, so should be free of emotive
Colin> associations by now, and the history can be read through
Colin> reasonably quickly.

Colin> Here are some references: https://bugs.debian.org/196762
Colin> https://bugs.debian.org/374569
Colin> https://lists.gnu.org/archive/html/groff/2007-11/msg00011.html

Reading this first reference was enough for me to identify several
differences I believe are key.

First, upgrading to new upstream is presumed good, not always good.
My concern with Ron's position is mostly that he wants the people
requesting a new upstream to justify that rather than wanting the htags
users to justify not breaking htags.

I think it was fairly obvious to everyone involved in the groff
discussion that the multibyte patch was required for Japanese man pages
among other things.
That is, I think there is evidence of the importance of the groff
multibyte patch in a way there's not evidence of heavy htags CGI use
here.
Put another way, the record presents evidence sufficient to overcome the
presumption that upgrading is good.

Second, you were responsive and pro-active. You reached out to the
multibyte patch author.
You tried to forward port it yourself.

Third, there was an eventual way forward. It was clear that groff would
eventually get to UTF-8 support good enough that we could use it.

So for these reasons, I think the groff situation is different in ways
that matter at least to my analysis.
Ron
2016-12-09 21:54:07 UTC
Permalink
Post by Sam Hartman
First, upgrading to new upstream is presumed good, not always good.
My concern with Ron's position is mostly that he wants the people
requesting a new upstream to justify that rather than wanting the htags
users to justify not breaking htags.
You understand the testability difference between those two though
right? And that proving a negative is a logical impossibility?

It's relatively easy for someone to report "I have this problem"
and to assess whether that can reasonably be fixed with a backport
or not. We do that for stable and security updates all the time.

But the only practical way to "ask" htags users to do what you ask
is to first break it anyway, and then hope you still have the
ability to backpedal once they realise you have.

Which is exactly what I was proposing the time was finally ripe
for us to do in an orderly way. And it was further changes that
happened in only the most recent upstream releases of global
which really made that look convincing. That still wasn't the
case with the source Punit originally proposed.
Ron
2016-12-09 14:48:46 UTC
Permalink
Post by Didier 'OdyX' Raboud
Post by Ron
Post by Didier 'OdyX' Raboud
If you haven't yet, I urge you to use our standard interface to report such
bugs; please make sure issues like this one are public on our bugtracker,
with correct found/notfound version markers.
Do you really want entries in the tracker for buggy code that was never
in Debian, because I nacked Punit uploading things he didn't understand
with a vague promise to maybe look at them later?
That code is now in Debian (experimental), so yes, I do expect you to act in
good faith and report bugs you see. You are obviously quite versed in how
'global' works, and that's undoubtedly valuable to produce the best possible
'global' package.
No, the code in experimental has that 'fixed', by commenting it out and
inviting the user to uncomment it themselves.

The context for this, was that was the code which was proposed to be
uploaded, and which was last discussed, at the time this was brought
to the ctte. It never was uploaded to any suite, just published in
collab-maint, and I think Punit provided packages somewhere else.

The code in experimental does have some eye raising things in it, but
nothing that I've yet traced through as being definitely exploitable.
But I also haven't given it a serious audit yet, just eyeballed it
quickly for obvious things.
Post by Didier 'OdyX' Raboud
Post by Ron
Now we're talking about what to do among a wider group of people, given
that it still looks like nothing material will change. The system works?
It doesn't: it shouldn't take 3 stable releases to get a new upstream release
for a leaf package.
There's a difference between blindly uploading a new upstream and
actually having a solution to the problem which is the reason that
it wasn't.

I made that reason very clear, and invited proposed solutions in
the original 'new upstream' bug, #574947. Nobody else, except
Taisuke and I ever made any effort to deal with that.

Taisuke and I both considered the secure use of htags to be an
important use case. But given the time that has gone by, and the
fact that the upstream code in what is currently in experimental
has completely eliminated any possible use from a secure system
location now, and how doxygen's seach facility has improved in
the last couple of years - my opinion has likewise changed in line
with that changed circumstance.

But it's taken all of "3 stable releases" for that to actually
change ... this wasn't all the case at the time of the freeze
for Jessie, or before.


I still think it would be rude to burn the remaining users on
such short notice - but I don't think we should delay doing that
any longer than the end of the Stretch freeze. And if there is
sufficient consensus to say "burn them immediately", I've already
said I'm ok with that too. But I would want there to be a consensus
of people who'd have my back about doing that. Else we just have
the same situation as we do now, where people abuse me for not doing
exactly what _they_ would have preferred.
Post by Didier 'OdyX' Raboud
Post by Ron
That report led to both me and the reporter having a (very) long
discussion with upstream about how to resolve the real problem that
you keep assuming we never tried to do anything about.
By "(very) long discussion", do you mean these 8 mails ?
http://lists.gnu.org/archive/html/bug-global/2010-08/threads.html#00006
No. That was one thread of many. But aside from what's also in
the BTS, and on -devel (or was it -project?), the vast majority
were private emails, and span many years of trying to move this
forward one way or another.
Ron
2016-12-09 20:24:17 UTC
Permalink
Post by Didier 'OdyX' Raboud
I'm purposedly not answering to the rest of your email, as I think the TC now
has enough information to issue a decision.
It's ok, you purposely not answering things I asked you to comment on,
which didn't support your personal agenda, had already stopped being
a surprise some time ago.

So I'm going to save the people on the ctte who haven't yet trawled
through this quagmire to sort the facts from the misdirection and
abuse, from wasting their precious time like I apparently have. And
I'm going to cut my losses here and not waste any more of my time on
it either.

I promised Phil and Sam, that if an effort was made to come to a
consensus on the technical questions, that I'd respect it, even if
that consensus wasn't formed around my own preferences.

And I likewise promised them, that if I thought what was being suggested
was ignorantly insane, that I'd just walk away without a fight.

So congratulations. You officially bounded across the threshold of insane
when, even given the context of what Vincent would have happily signed
blindly and uploaded, you were happy to offer him the responsibility of
being the maintainer of this with your TC hat on. This would be the same
Vincent who has never looked at this code, has never contributed anything
to it, who couldn't even muster the effort to understand the problems
himself or file proper bugs about them - and instead delegated that to the
TC, the same way he delegated doing the work to Punit, and delegated a lot
of plainly false accusations toward me. And who couldn't even follow the
simple instructions here: https://www.debian.org/devel/tech-ctte
without taking short cuts and ignoring the parts of that process which do
require some minimal extra effort to be put in.

Obviously an excellent candidate to maintain a package with hard problems!
Everything becomes easy if your ability to ignore things is strong enough.

You then had the gall to angrily insist that while you thought he might
be a better maintainer than me, it was still my responsibility to do the
work to fix all the obvious things that others had missed in their fork
(which he hadn't contributed anything to either). I'm afraid that's not
how encouraging volunteers to contribute their time for you works ...
sorry if this is news to you.


As you yourself noted, global is an unimportant package, and realistically
pretty much every part of it has been obsoleted for a long time now, by
alternatives that are better maintained upstream, that take real issues
more seriously, and that are more usefully functional.

I've never been shy about stepping up to take on 'hard' packages, whether
they are hard because of the code, or hard due to 'difficult' upstreams,
or some other reason. And nobody ever thanks the people who do that, but
I was fine with that too. But I'm not fine with the culture of arrogant
entitlement that some groups of users seem to be infected with, "you gave
me your work for free, now you owe me!". And I'm not fine with being
abused and slandered by people who themselves have contributed absolutely
nothing toward helping improve the things they are complaining about.

I stayed engaged with this process against my better judgement, because I,
like many others, had high hopes that ending Ian's reign of terror on the
TC was a huge opportunity for a new culture of mediated cooperation and
consensus to emerge. And regardless of what was decided about global,
this seemed like a good opportunity to demonstrate and cultivate that as
an example for others.

But it's now quite clear that experiment has failed quite dismally too.
Maybe because Vincent is your mate. Maybe because Ian was living up to his
reputation to cause infinite pain to anyone who ever disagreed with him.
Maybe because it is just easier, and less potentially embarrassing, to punt
on hard problems than to have a solution of your own to put on the table.
Maybe because of something else entirely - but that's your problem to debug
and solve now. It doesn't matter what the outcome of what you're proposing
to vote on would be, you were asked by a package maintainer to offer advice,
and the best you could offer was "we'll let somebody other than us take the
blame for whatever unfolds with this".

It is a 'vote' that frankly I'd even be embarrassed to 'win'. I'm exhausted
and demoralised by this being just another long road to nowhere. Not because
some people disagreed with me - I don't mind that, that's how you learn
things. But because the increasingly ill-named technical committee has once
again refused to stick its collective necks out to actually offer technical
advice when explicitly asked to. We chopped some heads off the hydra, but
it's apparently still just the same old beast, trudging the same old trails.
Only some different people get to pad their CVs with extra job titles.

Explaining things in careful detail has had every appearance of being a
complete waste of my time whichever way this might have ended up. The only
problem you've engaged with is making it be someone else's problem again.
So I give up. Maybe that means the terrorists with their simpler mantras
have won, and the cheering and firing into the air can begin.
But at least I've learned not to make that mistake again.


Anyone who doesn't understand:
https://lists.debian.org/debian-project/2016/12/msg00062.html
https://lists.debian.org/debian-devel/2016/11/msg00458.html
will be doomed to repeat it. Again.

I really do wish we'd all learn that. One day.


Personally, I'm going to take the remaining time I'd have wasted on this
and put it to good use improving my own infrastructure to maintain a set
of local packages for private use by myself and inside our company. If
this is the future of maintaining hard packages, then I'd much rather
just do that completely privately, and not contribute that work somewhere
that will just attract ungrateful people to rudely bitch at me about it.

I'll join the people who've taken the soft option to only maintain easy
packages in the distro archives, with nice polite users and upstreams
who are a genuine joy to work with. Which aren't still in CVS.


I'm feeling marginally better about this knowing that Wookey has put up
his hand to offer to be on point, even though he doesn't actually use it.
He really has been the only person who actually put in the effort to make
researched and actionable bug reports, and showed a willingness to
cooperate and fix things, even despite having an obvious prejudice for an
option different to what I preferred. That's how real respect is earned.

If he'd done that earlier, or before this became yet-another adversarial
melee in the ctte arena, maybe this would have all gone much differently
and more pleasantly - but you can't uncrash a trainwreck.

So I've now orphaned this package, and he has my blessing and sympathy
for being responsible for whatever happens with it from here. I haven't
filed a WNPP bug for that as we don't need to offer it to someone random.

Wookey: if you want the complete git history, right back to the very first
package in 1999, you can grab it from the Vcs-Git URL in the sid package.
I'm not going to go Full Bruce and rage delete it, but eventually I should
decruft alioth and remove it from there, so if you want it you should
probably clone it somewhere that works for you.

Good luck!
Ron
Uoti Urpala
2016-12-10 01:00:05 UTC
Permalink
Post by Ron
You then had the gall to angrily insist that while you thought he might
be a better maintainer than me, it was still my responsibility to do the
work to fix all the obvious things that others had missed in their fork
(which he hadn't contributed anything to either).  I'm afraid that's not
how encouraging volunteers to contribute their time for you works ...
sorry if this is news to you.
It was perfectly reasonable to ask that if you have any pretense at all
of actually doing the work expected of your maintainer position, you'd
contribute to creating packaging for the new upstream version, instead
of only attacking the people working on it.
Post by Ron
things.  But because the increasingly ill-named technical committee has once
again refused to stick its collective necks out to actually offer technical
advice when explicitly asked to.  We chopped some heads off the hydra, but
Explaining things in careful detail has had every appearance of being a
complete waste of my time whichever way this might have ended up. The only
The fundamental problem with most of your technical arguments was that
they were arguments about upstream development, not about packaging.
Even if they were 100% true, they still would not justify how you have
handled the Debian package. If you disagree with upstream that way, you
should try to convince them, and if that fails and you care enough,
create a new upstream fork.

Instead, you turned the Debian packaging into a practical fork. That's
not the right place for hosting a new fork. You also obviously did a
bad job of maintaining your fork (given the complete lack of
development). Your attitude seems to have been that you insist on
keeping the original GLOBAL out of Debian, do no development at all
yourself, and if anyone has problems with your fork you insist that
they do the work of developing it. That's not reasonable at all.
Wookey
2016-12-10 02:50:59 UTC
Permalink
Post by Ron
So I've now orphaned this package, and he has my blessing and sympathy
for being responsible for whatever happens with it from here. I haven't
filed a WNPP bug for that as we don't need to offer it to someone random.
OK. Cheers. Warm potato accepted :-)
Post by Ron
Wookey: if you want the complete git history, right back to the very first
package in 1999, you can grab it from the Vcs-Git URL in the sid package.
I'm not going to go Full Bruce and rage delete it, but eventually I should
decruft alioth and remove it from there, so if you want it you should
probably clone it somewhere that works for you.
OK. I don't use git unless I have to, so I'll let Punit worry about that.

Wookey
--
Principal hats: Linaro, Debian, Wookware, ARM
http://wookware.org/
Wookey
2016-12-09 14:39:28 UTC
Permalink
On 2016-12-08 17:03 +0100, Didier 'OdyX' Raboud wrote:

Sorry missed this in all yesterday's mail.
Post by Didier 'OdyX' Raboud
Wookey, Vincent, Punit: would any of you be willing to receive the 'global'
package maintainer hat? (which would of course come with the possibility to
share and change the maintainer status afterwards.)
Punit is happy to be maintainer. I am happy to co-maintain with him
(now that I'e gone the trouble of finding out a reasonable amount
about the package).

Wookey
--
Principal hats: Linaro, Debian, Wookware, ARM
http://wookware.org/
Wookey
2016-12-08 19:05:55 UTC
Permalink
[copyoing to the bug, which I failed to do originally]
Thanks for comprehensive reply, Ron.

I wonder if it would make more sense to have this discussion in
#816924 which is the actual bug for 'should we upgrade global to v6,
and if not, why not?' You have to be in the know to find this on the
ctte mailing list.
Post by Ron
Post by Philip Hands
Version 6 includes a CGI script that one is expected to install in a
manner so hopelessly insecure that we'd not accept it in Debian.
That one had removed the ability to run it from a secure system CGI
location at all, and changed the way that it's generated yet again.
It also made changes that guarantee everyone will need to fork it
for distro use, because it now hardcodes a fun mix of paths, like
/opt/local/bin for perl and /usr/local for global et al.
The .cgi scripts are generated from .in files which are correctly
parameterised with @PERLPATH@ and @GLOBALPATH@ etc. Upstream
unhelpfully ships pre-generated versions with the above arbitrary
local paths, but we kicked the build to force regeneration of these so
that all the scripts come out with correct debian paths. That was in
6.5.5-0.1 and is in 6.5.5-0.2 (with ctags path set correctly
too). Please file a bug if we missed any.

So this is not an outstanding issue, and no fork needed,but it would
be nice to improve upstrema's build and reduce debian patching here.
Post by Ron
"To use this code, please uncomment in your own responsibility."
OK, so as shipped, that's not actually an active problem.
Post by Ron
But it also outputs a .htaccess enabling execution in the directory
where the output is generated, whether you want to use it from there
or not (and adds a second CGI, and a bunch of jquery doing AJAX too).
The world is absolutely full of jquery doing AJAX. That's not bad in
itself (much as some of us prefered the 'old web'. We should make it
use system jquery.
Post by Ron
- it still passes the unsanitised $pattern to global, which then
passes it to popen (which again lets it be interpreted by the
shell).
So $pattern used to be sanitised, because that's what CVE-2000-0952 is
about. Ah and in fact it still is on line 148. Hmm, but that's after
using it in a pipe, which probably isn't much use... That is pretty
shoddy.
Post by Ron
- it won't run with perl's taint mode enabled, because that
simply kills it warning of unsafe operations on untrusted data.
- perl's warnings aren't enabled, and it complains of other
rookie code problems if you do.
- Enabling 'use strict' makes it scream with pain and completely
fail to compile and run.
It would need more thorough auditing to confirm that there is(n't) an
exploitable path through that, but given that it ignores even the most
basic advice which perl has strongly recommended since perl4 was new
and shiny, then if there isn't, it would be because of luck more than
obvious care.
I don't claim any web security expertise, so will ask a
potentially-dumb question. How much of a real world problem is the
shoddiness of this script if it is only used locally, using
htags-server (which is the only functionality now provided by the
package)? It exploses the script only to localhost unless you
explicitly configure it to do more, but not 'the net'. Nothing is done
as root, but it is run as 'you' so potentially gives access to 'you's
data, rather than all of www-data's data.

The only report of an actual vulnerability I can find online ('global'
is a very unhelpful name to search for vulnerbilities!) is:
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2000-0952 (from
2000 for global prior to v4.0.1

Which claimed to be fixing this very issue, but apparently it didn't stay fixed?

And this same code exists in global v5.7.1 (indeed both usages are
there, including the now-commented-out one), so how exactly was that
safer?
Post by Ron
It might not be utterly unfixable, but you'd have to convince upstream
that security is important, or completely fork it for debian, which
brings us back to exactly where we are - unless we just remove it all.
But that would need Time, whoever does it.
I have not grokked why the shoddy code in 5.7.1 is safe but the same
shoddy code in v6 cannot be let out. Did htmake+htconfig stop people
entering $pattern in the form?
Post by Ron
It is what we now have enabled in the package that Wookey uploaded to
experimental.
Indeed. Bugs (and even better patches) are welcome.
Post by Ron
Post by Philip Hands
Also, for people that want personal access to htags there is a
htags-server command that brings up a dedicated server to run the CGI
as the invoking user, by default bound to a localhost port.
htags-server is a shell script that generates python or ruby code
to use the HTTP server functionality they can provide.
I'd describe it as a script that runs the ruby or python2/3
HTTP-servers, as covered in the NEWS file, but I think we mean the
same thing.
Post by Ron
I have no idea offhand what the security properties of either of
those is is really like, but AIUI the python version at least
claims to run the CGIs as 'nobody' - but I'm not sure how that
works when run as a user unless something is SUID somewhere ...
or if that's patched to work differently in Debian to what the
python docs say it does. If that's true it would mean you need
everything to be at least world readable to use this.
All the files under the generated 'HTML' directory (using --suggest2)
are 644 (except rebuild.sh which is 640). That seems reasonable to me,
and it works as expected (with 6.5.5-0.2) The python process is run as
'wookey', and if I chmod everything o-r it still seems to work OK, so
not convinced that it is really runnning the cgi as 'nobody'.
Post by Ron
It still unconditionally drops a .htaccess file into the output
though, so if you're running a real web server, that may or may
not grant CGI execute access to that directory, depending on
how it's otherwise configured - whether you use htags-server for
this or not.
Is this an unreasonable default? Anyone who has set up their webserver
to give .cgi execute permissions because there is a .htaccess file
saying so either knows what they are doing, or gets what they deserve.
If you want to use htags locally then this is a sensible way to make
it work.
Post by Ron
That script doesn't really change any of the things that we'd be
concerned about in practice. It just means people who don't have
apache (or another web server) installed can install python or
ruby instead. Either can limit it to localhost or allow access
on a public IP.
And it's more convenient to use (and safer) than configuring apache to
allow .htacces enabling of CGI execution.
Post by Ron
In terms of the package currently in experimental, there's a bunch
of issues with it that would need to be fixed if we were going to
want it in Stretch. Wookey already identified some. And in the
quick look that I've already had at it so far, there's others
which were immediately obvious. There's a bunch of upstream stuff
that is simply missing from it, the man page for htags-server,
fixed in 6.5.5-0.2
Post by Ron
the icons and javascript and css for htags (which it actually complains
loudly about when you try to use it, and makes me wonder how much
they did actually test that or pay attention to what it reported)
Good point. I looked at the web-page, not the output from the server
end. I run requestpolicy, so am used to bare-looking web pages :-)
That's an easy fix.
Post by Ron
There's little things like it still having .la files in it, and
static libs for things that are supposedly 'plugins'. Which aren't
a big deal on their own to fix, but again suggests that if 'obvious'
things like that were missed, there's a good chance there will be
more issues than what I've already seen in a cursory review.
This is probably a result of letting libtool have its way. It loves
that ancient cruft. There is certainly quite a lot of tidying that
could be done with this, although I don't think it is in bad shape in
comparison to a great many debian packages. It would be good to talk
with upstream about whether there is any concrete benefit in using
libtool here anymore.

Wookey
--
Principal hats: Linaro, Debian, Wookware, ARM
http://wookware.org/
Wookey
2016-12-09 14:24:31 UTC
Permalink
Post by Wookey
Post by Ron
But it also outputs a .htaccess enabling execution in the directory
where the output is generated, whether you want to use it from there
or not (and adds a second CGI, and a bunch of jquery doing AJAX too).
The world is absolutely full of jquery doing AJAX. That's not bad in
itself (much as some of us prefered the 'old web'. We should make it
use system jquery.
OK. I tried making it use system jquery and some of the icons in htags
seem to go missing. So the internal jquery-treeview and
jquery-suggests may be incompatible. Needs a bit more research, so
we'll ship the internal one in 6.5.5-0.3 for now.
Post by Wookey
Post by Ron
the icons and javascript and css for htags
Good point.
That's an easy fix.
Included in 6.5.5-0.3 which I'm about to upload. So that fixes
#587489, (which has only been pending with a patch since May 2011).

(currently shipped in /usr/share/gtags rather than
/use/share/global/gtags because upstream needs patching to sort that
properly. One thing at a time.)

And now htags browsing has nice little icons for moving about. shiny :-)

We checked that 6.5.5 also fixes #847303. It does.

I have a query pending with Ron for if it's OK with him to move to
0-day uploads to experimental as I don't think the conventional NMU
delay is actually helping anyone in this case. It just makes things
slow. I'll just replace the pending 0.2 upload with this 0.3 one for
now.

Wookey
--
Principal hats: Linaro, Debian, Wookware, ARM
http://wookware.org/
Nikolaus Rath
2016-12-05 03:09:13 UTC
Permalink
Post by Ron
Hi Sam,
Post by Sam Hartman
Ron> Hi OdyX,
Post by Didier 'OdyX' Raboud
Hi there,
I've been mostly VAC, and only now found enough time to properly
read through this bug log. In the interest of transparency and to
help the TC reach a consensus (also through understanding what
the other members' understanding, ideas and positions are), here
comes my current understanding of the problem at hand.
As preamble, I will upfront state that I have _not_ looked in the
precise details of the so-called 'htags' functionality, as, the
rest will show, my current viewpoint on the problem is that it
doesn't matter.
Ron> If we run with your proposal, what are you actually suggesting
Ron> we tell the people who'd be upset by the loss of htags without
Ron> notice in Stretch? Because I don't really see how you've
Ron> addressed that here.
Ron, thanks for working with the process.
Thanks for engaging with the question :)
<OdyX> I've sent a long mail with my opinion, and Ron's answer hasn't
altered my conviction yet.
Because "lalala, I'm not listening" isn't really an answer to a simple
direct question. And definitely not something that I can wrap with
"Sorry, but a group of Debian Developers considered this all in careful
detail, and this is what we have decided", when breaking the news to
affected people.
...whose existence has only been postulated so far though.

In any case, it seems there are plenty of people who'd be willing to
relieve you of that burden and take over maintenance of global -
including dealing with any fallout from updating to an up-to-date
version.


Best,
-Nikolaus
--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«
Punit Agrawal
2016-11-22 23:39:19 UTC
Permalink
On Wed, Nov 16, 2016 at 2:23 PM, Didier 'OdyX' Raboud <***@debian.org> wrote:

[...]
Post by Didier 'OdyX' Raboud
A) Ron stays maintainer of src:global and uploads a reasonably recent 6.x
version to unstable;
B) (With or without an explicit decision by the TC), src:global is handed over
to new maintainers and they'd upload a reasonably recent 6.x version to
unstable;
(
In both A) and B) cases, whoever is maintainer of src:global would be in
charge of handling the subsequent (so far hypothetical) bugs in time for
the release. As usual, everyone is welcome to help with finding,
forwarding, and fixing bugs.
)
One suggestion to keep extra debian (non-upstream) functionality in
global is to move it to another package package. The package would
build on upstream htags and allows users to share their html
cross-indexing via cgi-bin.

As I understand it, the extra feature pertains to enabling access via
web-server to html index of sources generated by htags.
Post by Didier 'OdyX' Raboud
I see this as a suboptimal outcome, because it rewards inertia and stop-
C) Ron stays maintainer of src:global and uploads a reasonably recent 6.x
version to experimental, with the explicit goal of making that version later
available through stretch-backports.
--
Cheers,
OdyX
P.S. Sorry for the wall of text, including too many repetitions :/
[0] I know, it's all obvious.
[1] Yes, 6+ years, even at the Debian rhythm, is indefinite.
Didier 'OdyX' Raboud
2016-11-23 17:19:48 UTC
Permalink
The TC had an IRC meeting yesterday [0], and I (was) volunteered to wrap up
the different outcome possibilities, which would help forming our opinions.
Not all these options are exclusive, or would need an actual TC decision.

Here we go:

A) 'global' stays maintained as it currently is.

This would imply:
- no new 'global' upstream release before stretch,
- no 'htags removal' warning in stretch,
- possibility for the maintainer (and/or other interested parties) to start
maintaining a 'global6'; this wouldn't offer any guarantee for a more
recent version of 'global' in stretch. This would also imply having two
'global' packages in certain suites.

B) A fresher version of 'global' is uploaded to experimental 'soon'
(with or without interested parties' help; with or without a TC decision)

This would imply:
- any interested party would file (and close) Debian bugs for issues and
regressions (with appropriate severities), to make the 'fitness for a
stable release' assessment easier, and earlier.

C) After the release of stretch, a fresher version of 'global' is uploaded to
unstable with the explicit goal of making it available in buster.

This would imply:
- any interested party would file (and close) Debian bugs for issues and
regressions (with appropriate severities), to make the 'fitness for a
stable release' assessment easier.
- after migration to testing, this would make the fresher version of 'global'
available for backporting to stretch-backports.
- the version of 'global' released in stretch could carry 'htags removal'
warnings;

D) A fresher version of 'global' is uploaded to unstable 'soon', targetting
stretch (with or without interested parties' help; with or without a TC
decision)

This would imply:
- overruling the 'global' maintainer's decision (§6.1.4, implies 3:1 majority
in a TC vote);
- any interested party (including the maintainer) would file (and close)
Debian bugs for issues and regressions (with appropriate severities), to
make the 'fitness for a stable release' assessment easier.
- that this fresher version of 'global' would reach 'fit for release' status
before the Stretch release.

E) the 'global' package is handed to other maintainer(s)

This would imply:
- overruling the 'global' maintainer's decision (§6.1.4, implies 3:1 majority
in the TC vote);
--
Cheers,
OdyX

[0] http://meetbot.debian.net/debian-ctte/2016/debian-ctte.
2016-11-22-16.59.html
Wookey
2016-11-26 03:37:47 UTC
Permalink
Post by Didier 'OdyX' Raboud
The TC had an IRC meeting yesterday [0], and I (was) volunteered to wrap up
the different outcome possibilities, which would help forming our opinions.
Not all these options are exclusive, or would need an actual TC decision.
B) A fresher version of 'global' is uploaded to experimental 'soon'
(with or without interested parties' help; with or without a TC decision)
- any interested party would file (and close) Debian bugs for issues and
regressions (with appropriate severities), to make the 'fitness for a
stable release' assessment easier, and earlier.
OK, as Punit and I have prepared a current 6.5.5 which would be a
candidate for a 'modern' release, and I think it's useful to have such
a version available for people to test and file bugs against, I will
do a 5-day delayed NMU to experimental. Putting it in experimental
does not imply automatic transtion to stretch, nor block uploads via
unstable, so seems a reasonable thing to do.

This version is not perfect, but it is current with updated
packaging. I'll include details of the known issues in one of the
'please can we have a new version' bugs. I think it's more useful to
have the current state of play avalable than for me to keep messing
with it privately.

Wookey
--
Principal hats: Linaro, Debian, Wookware, ARM
http://wookware.org/
Ian Jackson
2016-11-29 13:03:59 UTC
Permalink
Post by Wookey
OK, as Punit and I have prepared a current 6.5.5 which would be a
candidate for a 'modern' release, and I think it's useful to have such
a version available for people to test and file bugs against, I will
do a 5-day delayed NMU to experimental. Putting it in experimental
does not imply automatic transtion to stretch, nor block uploads via
unstable, so seems a reasonable thing to do.
This version is not perfect, but it is current with updated
packaging. I'll include details of the known issues in one of the
'please can we have a new version' bugs. I think it's more useful to
have the current state of play avalable than for me to keep messing
with it privately.
Hooray, thank you.

Ian.
--
Ian Jackson <***@chiark.greenend.org.uk> These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Ian Jackson
2016-11-29 13:03:08 UTC
Permalink
Post by Didier 'OdyX' Raboud
E) the 'global' package is handed to other maintainer(s)
- overruling the 'global' maintainer's decision (§6.1.4, implies
3:1 majority in the TC vote);
No, handing the `global' package to other maintainer(s) would be a
decision under 6.1(2). Quoting that section:

In cases where Developers need to implement compatible ... stances
(for example, if they [Developers] disagree about [various things],
** or about who should be the maintainer for a package) **
the technical committee may decide the matter.

6.1(2) does not have a 3:1 requirement.

Ian.
Punit Agrawal
2016-11-30 15:29:24 UTC
Permalink
Post by Didier 'OdyX' Raboud
The TC had an IRC meeting yesterday [0], and I (was) volunteered to wrap up
the different outcome possibilities, which would help forming our opinions.
Not all these options are exclusive, or would need an actual TC decision.
A) 'global' stays maintained as it currently is.
- no new 'global' upstream release before stretch,
- no 'htags removal' warning in stretch,
- possibility for the maintainer (and/or other interested parties) to start
maintaining a 'global6'; this wouldn't offer any guarantee for a more
recent version of 'global' in stretch. This would also imply having two
'global' packages in certain suites.
B) A fresher version of 'global' is uploaded to experimental 'soon'
(with or without interested parties' help; with or without a TC decision)
- any interested party would file (and close) Debian bugs for issues and
regressions (with appropriate severities), to make the 'fitness for a
stable release' assessment easier, and earlier.
C) After the release of stretch, a fresher version of 'global' is uploaded to
unstable with the explicit goal of making it available in buster.
- any interested party would file (and close) Debian bugs for issues and
regressions (with appropriate severities), to make the 'fitness for a
stable release' assessment easier.
- after migration to testing, this would make the fresher version of 'global'
available for backporting to stretch-backports.
- the version of 'global' released in stretch could carry 'htags removal'
warnings;
D) A fresher version of 'global' is uploaded to unstable 'soon', targetting
stretch (with or without interested parties' help; with or without a TC
decision)
- overruling the 'global' maintainer's decision (§6.1.4, implies 3:1 majority
in a TC vote);
- any interested party (including the maintainer) would file (and close)
Debian bugs for issues and regressions (with appropriate severities), to
make the 'fitness for a stable release' assessment easier.
- that this fresher version of 'global' would reach 'fit for release' status
before the Stretch release.
E) the 'global' package is handed to other maintainer(s)
- overruling the 'global' maintainer's decision (§6.1.4, implies 3:1 majority
in the TC vote);
In offline discussions with Wookey, we came to the realisation that
reading the various bug reports (including this one) it is not very
clear how global (upstream) is structured, the functionality it
provides and bits that are holding back the debian update.

Gnu Global is a source code tagging system similar in functionality to
ctags or cscope. It allows searching for symbol definitions and use in
the codebase by splitting the functionality into two steps -

* Indexing - This is a offline step where an index of source code
symbols definitions and references for a code base is created. This
functionality is provided by "gtags" binary (part of upstream global).
Indexes (or is it indices?) typically live in the source tree (they
don't need to).

* Searching - Once the code base is indexed, the indexes can be used
to search for symbol definitions as well as query locations where the
symbol is referenced. This can be done using the "global" binary
(again provided by upstream). Also there are various integrations with
commonly used editors - emacs, vim, elvis, etc - that internally
invoke global for their symbol lookup requirements. So if you are a
developer who would like to use global to navigate your way through a
large code base a common way to use it would be to use "gtags" and the
editor integration of your choice.

In addition to the above mechanisms, upstream also provides "htags"
which can be used to generate an HTML version of the index. "htags"
uses the index created by "gtags" and the source tree as input and
generates html files which allow you to navigate the source code in
the browser. By default, htags generates static pages which means you
can browse the code in a local browser without requiring a web server.

Optionally, "htags" can generate a dynamic index (which reduces disk
space usage) but requires an http server setup to be able to serve the
pages. In this scenario, you will also need to be able to execute CGI
scripts as the symbol lookup is done when serving the http request (as
opposed to pre-processed when using static pages).

And this last bit (integration with system web server) is the
functionality that had security concerns raised by Ron and for which
there are patches in the debian package. In recent versions (> 6.4,
Apr '15), upstream has dropped support for generating scripts that
expect to be written to system cgi-bin folder. As demonstrated by
wookey[1], it is still possible to use the dynamic html option as a
non-root user by running a web server using higher numbered ports.

IIUC, Ron's patch removes the need for htags to write to system
cgi-bin (by providing an indirection mechanism for incoming http
requests).

Serving dynamic source index via system cgi-bin is, IMHO, a small part
of the functionality provided by the package and should be seen as
such while making any decision regarding the package.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=816924#65
Post by Didier 'OdyX' Raboud
--
Cheers,
OdyX
[0] http://meetbot.debian.net/debian-ctte/2016/debian-ctte.
2016-11-22-16.59.html
Ian Jackson
2016-11-30 16:56:21 UTC
Permalink
Post by Punit Agrawal
In offline discussions with Wookey, we came to the realisation that
reading the various bug reports (including this one) it is not very
clear how global (upstream) is structured, the functionality it
provides and bits that are holding back the debian update.
Thank you for your clear and excellent explanation.
Post by Punit Agrawal
Optionally, "htags" can generate a dynamic index (which reduces disk
space usage) but requires an http server setup to be able to serve the
pages. In this scenario, you will also need to be able to execute CGI
scripts as the symbol lookup is done when serving the http request (as
opposed to pre-processed when using static pages).
And this last bit (integration with system web server) is the
functionality that had security concerns raised by Ron [etc.]
So, to be clear, it is this functionality which is dropped in the
package that you and Wookey uploaded to experimental/delayed ?
Post by Punit Agrawal
In addition to the above mechanisms, upstream also provides "htags"
which can be used to generate an HTML version of the index. "htags"
uses the index created by "gtags" and the source tree as input and
generates html files which allow you to navigate the source code in
the browser. By default, htags generates static pages which means you
can browse the code in a local browser without requiring a web server.
So the impact for a user of the existing functionality the regression
is not that their entire workflow is disrupted.

Rather (unless the software is improved) they have these choices:

- Put up with pregenerated html indices. Is the only downside
of this that they use additional disk space, and presumably
cpu time to generate them ?

- Run the htags server on a high-numbered port somehow. (Is there
some kind of simple scriptery provided, to do this?)

- If the machine is not really multi-user, tear down the security
boundary defending the webserver from their user account, and give
their user account access to the webserver cgi directory (or plumb
it in with symlinks). (Are there any instructions or scripts for
this?) (Also this assumes that the source code is not super
secret.)

I don't know much about this, but several of these choices seem likely
to be plausible choices for many if not most current users of htags.


FAOD none of this changes my view about the proper resolution of this
TC petition. But answers might help clarify the discussion.

Thanks,
Ian.
--
Ian Jackson <***@chiark.greenend.org.uk> These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Sam Hartman
2016-11-30 19:11:43 UTC
Permalink
I'd really like to see the TC offer at least the following advice:

1) We believe that strong evidence is required to hold back integrating
new versions of software like global. The burden of proof is on those
who propose not to update, not on those who would like Debian to contain
current upstream features.

2) This burden has not been met with regard to htags and regressions
related to htags.

3) Delays in discussion of this issue over the year suggest that having
more people involved in maintaining the global package would help
address a perception that the maintainer is blocking forward progress.

I don't think I'd support giving global to someone else. I don't think
we even need to say "Ron you did something bad." I do think that Ron
contributed to a harmful perception that damages those who would use and
contribute to global in Debian.
I think taking steps like involning others in the process would be good
in fighting that perception and would be good for the package at a
technical level.
If we can find a path forward that gets a new global into Debian, I'd be
happy only offering advice. If we get stuck doing that, I think we need
to overrule something.
Didier 'OdyX' Raboud
2016-12-02 09:44:39 UTC
Permalink
Post by Sam Hartman
1) We believe that strong evidence is required to hold back integrating
new versions of software like global. The burden of proof is on those
who propose not to update, not on those who would like Debian to contain
current upstream features.
2) This burden has not been met with regard to htags and regressions
related to htags.
3) Delays in discussion of this issue over the year suggest that having
more people involved in maintaining the global package would help
address a perception that the maintainer is blocking forward progress.
Absolutely. This would be a the very minimal statement I'd like us to emit.
Post by Sam Hartman
I don't think I'd support giving global to someone else.
I would support handing global to new maintainers, really. We have 4 persons
who have contributed to the newly-available package in experimental:
https://tracker.debian.org/news/820174
Their total work is a magnitude more than what was given to the package by the
current maintainer in the last 6 years.
Post by Sam Hartman
I don't think we even need to say "Ron you did something bad." I do think
that Ron contributed to a harmful perception that damages those who would
use and contribute to global in Debian.
I wouldn't support a decision where we state that Ron did something bad. It
would be unneeded blaming (especially in a TC decision), and unnecessarily
agressive.

I'd support a decision handing the package to better hands though. For me, it
is now obvious that there exists a group of maintainers out there who would do
better service to the maintenance of global, than is currently done.
Post by Sam Hartman
If we can find a path forward that gets a new global into Debian, I'd be
happy only offering advice. If we get stuck doing that, I think we need
to overrule something.
Sure, absolutely. But its really also a question of timing, and allowing Ron
to tell the TC (in direct words, through further NAK'ing, or through inaction)
"fine, I've won another release with global v5 in, I'll let the package go
after the release of stretch", we will have rewarded stop-energy and inertia,
over service to our users.

Although we probably haven't reached consensus, I'd like to see this subject
move forward; what about the following ballot (with options to be refined, of
course):

A) Using §6.1.5, the TC offers advice (insert Sam's advice above) about the
maintenance of src:global.
B) Using §6.1.2, the TC decides that the src:global maintainer is now (insert
name)
C) Further discussion
--
Cheers,
OdyX
Sam Hartman
2016-12-02 13:50:52 UTC
Permalink
Like you I want to see global6 for stretch.
I'm not sure I want to see it bad enough to override someone.
I'd rank doing so above FD though but below a pure advice option.
Ian Jackson
2016-12-02 15:20:59 UTC
Permalink
Post by Sam Hartman
Like you I want to see global6 for stretch.
I'm not sure I want to see it bad enough to override someone.
I'd rank doing so above FD though but below a pure advice option.
Why are you prepared to override[0]

Andrew Bailey, Era Eriksson, Markus Grunwald, Thomas Viehweger,
Hongzheng Wang, Vincent Bernat, Pranith Kumar, Punit Agrawal, Volker
Mische, Johannes Stezenbach, Wei Liu, Alberto Luaces, Pierre-Elliott
Becue, Wookey

but not prepared to override

Ron Lee

?

Have you been reading debian-project ? From my point of view[1]
perhaps I should give up on trying to persuade TC members.

After all if the TC does not transfer maintainership in such an
extreme situation as this, my job of perusading DDs to vote to give
responsibility for these decisions to someone else is very easy.

Ian.

[0] I have listed all the people who explicitly requested a new
version (assuming, I think, that they are Debian users who want to use
global 6), and the people who have done work to prepare new versions,
in chronological order. I have excluded Shigio Yamaguchi because I
think their primary status is upstream.

[1] My point of view of trying to fix Debian's totally nonfunctional
processes for dealing with unwarranted blocking by maintainers, which
is a drum I have been banging for years and years now.
--
Ian Jackson <***@chiark.greenend.org.uk> These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Sam Hartman
2016-12-02 17:25:35 UTC
Permalink
Like you I want to see global6 for stretch. I'm not sure I want
to see it bad enough to override someone. I'd rank doing so
above FD though but below a pure advice option.
Ian> Why are you prepared to override[0]

I am not prepared to override those seeking global6 either.

Ian> but not prepared to override

Ian> Ron Lee

Because in this instance I believe giving advice is going to be
sufficient to get action.
If some reasonable version of global6 doesn't happen post haste, I could
change my mind in the order of a week or two.

Ian> Have you been reading debian-project ?
No.
Ian Jackson
2016-12-02 17:51:37 UTC
Permalink
Post by Sam Hartman
I am not prepared to override those seeking global6 either.
I don't know quite how to put this. I'm afraid that what I say is
going to sound quite aggressive and hurtful. But I think it is a very
important point that needs to be made:

By actively resisting the idea that the TC should swiftly make a
concrete and definitive decisioin, you are very directly reinforcing
the blockage which is preventing the prospective global (6)
maintainers from improving Debian.

So you _are_ overriding those seeking global (6). (TC members who are
saying nothing are contributing to the blockage too: with power comes
a responsibility to use it to right injustice, where one can.)


You've no doubt heard the famous quote by Paulo Freire:

Washing one's hands of the conflict between the powerful and the
powerless means to side with the powerful, not to be neutral.

But you are doing more than washing your hands. You are yourself in a
position of power, and you are actively using that position to
reinforce Ron's.

I know that you do not _set out_ reinforce Ron's position of power
over his victims. That is not your goal. You are trying to come to
an amicable settlement. You are trying to get everyone to be nice.

But when people are being oppressed, it is quite wrong to make the
feelings of the perpetrator a primary consideration. First help the
victims, by relieving them from the grasp of their oppressor.

This is all very dramatic language. Of course I know this is "only
Debian" and of course no-one is dying here. But the principles are
the same.

Ian.
--
Ian Jackson <***@chiark.greenend.org.uk> These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Sam Hartman
2016-12-02 20:01:24 UTC
Permalink
Ian> I know that you do not _set out_ reinforce Ron's position of
Ian> power over his victims. That is not your goal. You are trying
Ian> to come to an amicable settlement. You are trying to get
Ian> everyone to be nice.

Ian> But when people are being oppressed, it is quite wrong to make
Ian> the feelings of the perpetrator a primary consideration. First
Ian> help the victims, by relieving them from the grasp of their
Ian> oppressor.

I disagree with the idea that this situation is about an aggressor and
victims.
I agree such situations exist. I do not think this situation currently
is such a situation.

That is key to my position.

I do not think engaging furthure on this point will serve this
discussion.
Sam Hartman
2016-12-02 20:03:30 UTC
Permalink
So, does someone want to propose a resolution so we can move this
forward?
Punit Agrawal
2016-11-30 19:19:29 UTC
Permalink
On Wed, Nov 30, 2016 at 4:56 PM, Ian Jackson
Post by Ian Jackson
Post by Punit Agrawal
In offline discussions with Wookey, we came to the realisation that
reading the various bug reports (including this one) it is not very
clear how global (upstream) is structured, the functionality it
provides and bits that are holding back the debian update.
Thank you for your clear and excellent explanation.
Post by Punit Agrawal
Optionally, "htags" can generate a dynamic index (which reduces disk
space usage) but requires an http server setup to be able to serve the
pages. In this scenario, you will also need to be able to execute CGI
scripts as the symbol lookup is done when serving the http request (as
opposed to pre-processed when using static pages).
And this last bit (integration with system web server) is the
functionality that had security concerns raised by Ron [etc.]
So, to be clear, it is this functionality which is dropped in the
package that you and Wookey uploaded to experimental/delayed ?
The debian packaging added two tools not present in upstream -
htconfig and htmake. These tools and debian patched htags together
make the integration with system web server work to the extent that
Ron was comfortable shipping the package. Both htmake and htconfig are
authored by Ron.

So although the package uploaded by Wookey retain the tools they have
become non-functional due to upstream changes to htags. But as you
point out, you can still achieve a very similar end result using other
mechanisms.
Post by Ian Jackson
Post by Punit Agrawal
In addition to the above mechanisms, upstream also provides "htags"
which can be used to generate an HTML version of the index. "htags"
uses the index created by "gtags" and the source tree as input and
generates html files which allow you to navigate the source code in
the browser. By default, htags generates static pages which means you
can browse the code in a local browser without requiring a web server.
So the impact for a user of the existing functionality the regression
is not that their entire workflow is disrupted.
That is exactly what I was trying to get across in my previous email.
Post by Ian Jackson
- Put up with pregenerated html indices. Is the only downside
of this that they use additional disk space, and presumably
cpu time to generate them ?
There
Post by Ian Jackson
- Run the htags server on a high-numbered port somehow. (Is there
some kind of simple scriptery provided, to do this?)
It would be a web server - htags isn't a server in itself. Upstream's
suggestion of using

python -m CGIHTTPServer

in the generated HTML folder worked for the package Wookey has
uploaded. This command can be executed with user privileges and runs a
local instance of an http server.

IIUC, running the web server with user privileges, prevents it from
binding to external interfaces/IP addresses.
Post by Ian Jackson
- If the machine is not really multi-user, tear down the security
boundary defending the webserver from their user account, and give
their user account access to the webserver cgi directory (or plumb
it in with symlinks). (Are there any instructions or scripts for
this?) (Also this assumes that the source code is not super
secret.)
So I am not very familiar with cgi scripts (having never used it
myself) but I believe what you say is possible - somebody with a
little more knowledge than me should be easily able to come up with
the instructions.
Post by Ian Jackson
I don't know much about this, but several of these choices seem likely
to be plausible choices for many if not most current users of htags.
FAOD none of this changes my view about the proper resolution of this
TC petition. But answers might help clarify the discussion.
Thanks,
Ian.
--
a private address which bypasses my fierce spamfilter.
Punit Agrawal
2016-12-01 12:18:39 UTC
Permalink
I just noticed I missed out responding to one of your queries in my reply.
Post by Punit Agrawal
On Wed, Nov 30, 2016 at 4:56 PM, Ian Jackson
Post by Ian Jackson
Post by Punit Agrawal
In offline discussions with Wookey, we came to the realisation that
reading the various bug reports (including this one) it is not very
clear how global (upstream) is structured, the functionality it
provides and bits that are holding back the debian update.
Thank you for your clear and excellent explanation.
Post by Punit Agrawal
Optionally, "htags" can generate a dynamic index (which reduces disk
space usage) but requires an http server setup to be able to serve the
pages. In this scenario, you will also need to be able to execute CGI
scripts as the symbol lookup is done when serving the http request (as
opposed to pre-processed when using static pages).
And this last bit (integration with system web server) is the
functionality that had security concerns raised by Ron [etc.]
So, to be clear, it is this functionality which is dropped in the
package that you and Wookey uploaded to experimental/delayed ?
The debian packaging added two tools not present in upstream -
htconfig and htmake. These tools and debian patched htags together
make the integration with system web server work to the extent that
Ron was comfortable shipping the package. Both htmake and htconfig are
authored by Ron.
So although the package uploaded by Wookey retain the tools they have
become non-functional due to upstream changes to htags. But as you
point out, you can still achieve a very similar end result using other
mechanisms.
Post by Ian Jackson
Post by Punit Agrawal
In addition to the above mechanisms, upstream also provides "htags"
which can be used to generate an HTML version of the index. "htags"
uses the index created by "gtags" and the source tree as input and
generates html files which allow you to navigate the source code in
the browser. By default, htags generates static pages which means you
can browse the code in a local browser without requiring a web server.
So the impact for a user of the existing functionality the regression
is not that their entire workflow is disrupted.
That is exactly what I was trying to get across in my previous email.
Post by Ian Jackson
- Put up with pregenerated html indices. Is the only downside
of this that they use additional disk space, and presumably
cpu time to generate them ?
AFAICT, yes.
Post by Punit Agrawal
Post by Ian Jackson
- Run the htags server on a high-numbered port somehow. (Is there
some kind of simple scriptery provided, to do this?)
It would be a web server - htags isn't a server in itself. Upstream's
suggestion of using
python -m CGIHTTPServer
in the generated HTML folder worked for the package Wookey has
uploaded. This command can be executed with user privileges and runs a
local instance of an http server.
IIUC, running the web server with user privileges, prevents it from
binding to external interfaces/IP addresses.
Post by Ian Jackson
- If the machine is not really multi-user, tear down the security
boundary defending the webserver from their user account, and give
their user account access to the webserver cgi directory (or plumb
it in with symlinks). (Are there any instructions or scripts for
this?) (Also this assumes that the source code is not super
secret.)
So I am not very familiar with cgi scripts (having never used it
myself) but I believe what you say is possible - somebody with a
little more knowledge than me should be easily able to come up with
the instructions.
Post by Ian Jackson
I don't know much about this, but several of these choices seem likely
to be plausible choices for many if not most current users of htags.
FAOD none of this changes my view about the proper resolution of this
TC petition. But answers might help clarify the discussion.
Thanks,
Ian.
--
a private address which bypasses my fierce spamfilter.
Shigio YAMAGUCHI
2016-12-09 02:58:50 UTC
Permalink
Hello all,
Post by Ron
Which for those who don't speak it, is perl for "anyone can execute
arbitrary shell commands by typing them into a web browser", since
$pattern is an unsanitised, untrusted, input from the query string.
This code is for Windows; it is not used in UNIX.
Ron's quotation seems to be part of the following code:

------------------------------------------------------------------------------
[global.cgi.tmpl.in] (global-6.5.2)
------------------------------------------------------------------------------
if ($^O eq 'MSWin32') {
open(PIPE, '@globalpath@' . " --result=ctags-xid $flags $pattern
|");
} else {
open(PIPE, "-|") || exec '@globalpath@', '--result=ctags-xid',
$flags, $pattern;
if ($?) {
error_and_exit("Cannot execute global.");
}
}
------------------------------------------------------------------------------

Though I don't recognize it is a security hole on Windows, I don't know
whether
it is true in the future. So, it is commented out in the latest release now.

------------------------------------------------------------------------------
[global.cgi.in] (global-6.5.5)
------------------------------------------------------------------------------
if ($^O eq 'MSWin32') {
#
# This code was commented out, because it may have a security hole
in the
# future. To use this code, please uncomment in your own
responsibility.
#
#open(PIPE, "$global_command" . " --result=ctags-xid $flags
\"$pattern\" |");
error_and_exit("Feature not implemented.");
} else {
open(PIPE, "-|") || exec "$global_command", '--result=ctags-xid',
$flags, $pattern;
if ($?) {
error_and_exit("Cannot execute global.");
}
}
------------------------------------------------------------------------------

Please see the following thread, for the details.

[A CGI security hole on Windows?]
http://lists.gnu.org/archive/html/bug-global/2016-03/msg00002.html
Post by Ron
The .cgi scripts are generated from .in files which are correctly
unhelpfully ships pre-generated versions with the above arbitrary
local paths, but we kicked the build to force regeneration of these so
that all the scripts come out with correct debian paths. That was in
6.5.5-0.1 and is in 6.5.5-0.2 (with ctags path set correctly
too). Please file a bug if we missed any.
It's my mistake. I will fix it soon.

It is helpful if these bug reports are sent to bug-***@gnu.org.
Thank you in advance.

Regards,
Shigio
--
Shigio YAMAGUCHI <***@gnu.org>
PGP fingerprint: D1CB 0B89 B346 4AB6 5663 C4B6 3CA5 BBB3 57BE DDA3
A long mail is hell.
-- An anonymous philosopher in ancient Greece
Philip Hands
2016-12-09 09:26:29 UTC
Permalink
Hi Shigio,

Thanks for getting involved.
Post by Shigio YAMAGUCHI
Hello all,
Post by Ron
Which for those who don't speak it, is perl for "anyone can execute
arbitrary shell commands by typing them into a web browser", since
$pattern is an unsanitised, untrusted, input from the query string.
This code is for Windows; it is not used in UNIX.
------------------------------------------------------------------------------
[global.cgi.tmpl.in] (global-6.5.2)
------------------------------------------------------------------------------
if ($^O eq 'MSWin32') {
} else {
$flags, $pattern;
Is it not the case that this last line forks and execs global, passing
$pattern as a parameter to global's -e option, and that $pattern is
untrusted input?

Looking at global.c it seems that before it is passed on to popen, it is
run through quote_shell() which quotes any single-quotes in the string.

That seems to deal with Ron's assertion that it's exploitable, although
I have a slight feeling of impending doom about relying upon just this.

Would it not be wise to make the network-facing perl code runnable with
strict and taint turned on, if only to stop people reacting with horror
at first glance?

I presume patches would be welcome?

Cheers, Phil.
--
|)| Philip Hands [+44 (0)20 8530 9560] HANDS.COM Ltd.
|-| http://www.hands.com/ http://ftp.uk.debian.org/
|(| Hugo-Klemm-Strasse 34, 21075 Hamburg, GERMANY
Shigio YAMAGUCHI
2016-12-09 12:14:18 UTC
Permalink
Hi,
Post by Philip Hands
Post by Shigio YAMAGUCHI
$flags, $pattern;
Is it not the case that this last line forks and execs global, passing
$pattern as a parameter to global's -e option, and that $pattern is
untrusted input?
Yes. $patern is untrusted input.
Post by Philip Hands
Looking at global.c it seems that before it is passed on to popen, it is
run through quote_shell() which quotes any single-quotes in the string.
That seems to deal with Ron's assertion that it's exploitable, although
I have a slight feeling of impending doom about relying upon just this.
I agree.
I uses popen() in global.c to call idutils(1). I would like to rewrite it
in near future.
Post by Philip Hands
Would it not be wise to make the network-facing perl code runnable with
strict and taint turned on, if only to stop people reacting with horror
at first glance?
I presume patches would be welcome?
Of course! Thank you.

Regargs,
Shigio
--
Shigio YAMAGUCHI <***@gnu.org>
PGP fingerprint: D1CB 0B89 B346 4AB6 5663 C4B6 3CA5 BBB3 57BE DDA3
Wookey
2016-12-09 10:31:31 UTC
Permalink
Post by Shigio YAMAGUCHI
Hello all,
Post by Wookey
The .cgi scripts are generated from .in files which are correctly
unhelpfully ships pre-generated versions with the above arbitrary
local paths, but we kicked the build to force regeneration of these so
that all the scripts come out with correct debian paths. That was in
6.5.5-0.1 and is in 6.5.5-0.2 (with ctags path set correctly
too). Please file a bug if we missed any.
It's my mistake. I will fix it soon.
Thank you in advance.
Yes. Now that we have the package in some sort of reasonable shape in
debian we will forward the relevant bits upstream, in order to
minimise debian patches. There are quite a few things we want to
discuss :-)

Wookey
--
Principal hats: Linaro, Debian, Wookware, ARM
http://wookware.org/
Shigio YAMAGUCHI
2016-12-09 13:10:43 UTC
Permalink
Hi all,
Please forgive me one more comment.
... and it may well be that this has actually happened now with
upstream's decision to drop all support for providing a secure
system CGI of any form that people can use for this. The upstream
code is basically now back to what it was in the 90's, with the only
way to use this being to allow execution of a generated CGI in the
same tree as the html content. Which was already well known to be a
dangerous and ill advised idiom even back then ...
Apache + system CGI is somewhat overdone to use htags.
GLOBAL is just a source code tagging tool for developers;
it is not a system to publish something to the world.

My answer is htags-server(1), a private http server for htags.
You should invoke this command for each project like this:

$ gtags
$ htags --suggest2
$ htags-server
Please access at http://127.0.0.1:8000
Python2 http/cgi server
Serving HTTP on 127.0.0.1 port 8000 ...

You can see the output of htags through 'http://127.0.0.1:8000'.
It is easy to use, and is safer because it runs with user's
privilege without publishing to the network by default.
This command was added to GLOBAL-6.3 in 2014.

IMO, it is useless to continue supporting system CGI.
It is difficult to set up, and never makes something safer.

Regards,
Shigio
--
Shigio YAMAGUCHI <***@gnu.org>
PGP fingerprint: D1CB 0B89 B346 4AB6 5663 C4B6 3CA5 BBB3 57BE DDA3
I don't like FUD.
-- Anonymous
Loading...