From: Xho Jingleheimerschmidt on
Mart van de Wege wrote:
> Sherm Pendley <sherm(a)debian.shermpendley.com> writes:
>
>> "jl_post(a)hotmail.com" <jl_post(a)hotmail.com> writes:
>>
>>> Sadly, some people just don't think the perldocs are all that
>>> important.
>> Having perldocs on a production server is *not* important.
>
> Quite frankly, I disagree. When I am debugging a production problem, and
> part of the problem is perl scripts and/or modules, it's convenient to
> read the docs in-place, especially since I know the version of the docs
> should fit the deployed perl.

I run into this problem frequently. I'm trying to debug something on
production server, and discover that not only does it not have perldoc
or man, it also doesn't have top, ps, strace, ltrace, etc. The
resulting conversation goes something like this:

me: "Please install these on the production servers"

IT: "We don't put debugging tools on the production servers because you
are supposed to debug on dev, not production"

me: "The problem does not manifest on dev servers, only production, so
it has to be debugged on production."

IT: "That's impossible, the production servers are identical to the dev
servers."

me: "Obviously not. If they were indentical, then all the debug tools I
just requested would already be there. They aren't."

IT and me: Mutual head explosion.

Xho
From: Martijn Lievaart on
On Sat, 06 Mar 2010 10:39:25 -0800, Xho Jingleheimerschmidt wrote:

> I run into this problem frequently. I'm trying to debug something on
> production server, and discover that not only does it not have perldoc
> or man, it also doesn't have top, ps, strace, ltrace, etc. The
> resulting conversation goes something like this:

Oh yes, production servers without man. So I am supposed to "know" all
parameters of those seldom used commands on $SOME_ANCIENT_UNIX and will
never make a mistake, while the hacker that just entered the machine is
seriously hampered by the omission of man pages. Yeah sure.

I call that a production risk. Unfortunately, many risk "professionals"
don't agree with me.

M4

From: sln on
On Sat, 06 Mar 2010 10:39:25 -0800, Xho Jingleheimerschmidt <xhoster(a)gmail.com> wrote:

>me: "The problem does not manifest on dev servers, only production, so
>it has to be debugged on production."
>
>IT: "That's impossible, the production servers are identical to the dev
>servers."
>
>me: "Obviously not. If they were indentical, then all the debug tools I
>just requested would already be there. They aren't."
>

You lost me with these 3. The debug tools on dev are the only difference
between the two and it doesen't manifest on dev ??
There must be some serious background proceses hooking going on, on dev.

So, you want to put debug tools on production, making it identical to dev?
Thats not a step in the right direction.

-sln
From: Mart van de Wege on
Martijn Lievaart <m(a)rtij.nl.invlalid> writes:

> On Sat, 06 Mar 2010 10:39:25 -0800, Xho Jingleheimerschmidt wrote:
>
>> I run into this problem frequently. I'm trying to debug something on
>> production server, and discover that not only does it not have perldoc
>> or man, it also doesn't have top, ps, strace, ltrace, etc. The
>> resulting conversation goes something like this:
>
> Oh yes, production servers without man. So I am supposed to "know" all
> parameters of those seldom used commands on $SOME_ANCIENT_UNIX and will
> never make a mistake, while the hacker that just entered the machine is
> seriously hampered by the omission of man pages. Yeah sure.
>
> I call that a production risk. Unfortunately, many risk "professionals"
> don't agree with me.
>
Yes, well, my daily job *is* security.

And I run into a lot of 'consultants' who do security by the book, which
means that rules like 'no extraneous software on production servers' get
implemented in the most hare-brained way possible, and any protest that
you *need* certain tools is met with 'That's a security risk'.

You know, sometimes security is best served by giving an admin the tools
to reduce their response time to an incident. That gives them the time
to ascertain whether the incident is merely an innocent crash or an
actual exploit attempt.

And for that, I often need dev tools and documentation.

Mart

--
"We will need a longer wall when the revolution comes."
--- AJS, quoting an uncertain source.
From: Peter J. Holzer on
On 2010-03-06 22:08, sln(a)netherlands.com <sln(a)netherlands.com> wrote:
> On Sat, 06 Mar 2010 10:39:25 -0800, Xho Jingleheimerschmidt <xhoster(a)gmail.com> wrote:
>
>>me: "The problem does not manifest on dev servers, only production, so
>>it has to be debugged on production."
>>
>>IT: "That's impossible, the production servers are identical to the dev
>>servers."
>>
>>me: "Obviously not. If they were indentical, then all the debug tools I
>>just requested would already be there. They aren't."
>>
>
> You lost me with these 3. The debug tools on dev are the only difference
> between the two and it doesen't manifest on dev ??

No. He didn't say that they are the *only* difference. He said that this
is an *obvious* difference, so saying that prod and dev are identical is
bullshit. The IT department just thinks the differences are irrelevant.

There may be other differences which the IT department thinks are
irrelevant.

One difference between prod and dev which should be obvious to everybody
is:

Users.

The prod system has real users, the dev system doesn't. At best it has a
handful of carefully chosen test users which behave completely
differently than real users.

So you need all the monitoring and debugging aids on prod, because
that's where the problem occurs.

Please note that "monitoring and debugging" is not the same as "making
changes". You need to be free to see what's going on on the production
server, but any change you make must go through proper change
management.


> There must be some serious background proceses hooking going on, on dev.

Huh? I don't understand this sentence.

> So, you want to put debug tools on production, making it identical to dev?

I want the tools I need to find the problem on the server where the
problem occurs. They are useless where the problem doesn't occur. It's
like the old joke about the man who's looking for his keys under a
street lamp:
Passerby: What are you looking for?
Man: My house keys.
Passerby: Did you lose them here?
Man: Probably not. I think I lost them over there.
(points into a dark alley)
Passerby: Then why are you looking here?
Man: Because here I can see.

hp