Prev: XDrawImageString
Next: ISO: ideas for creating a more consistent user experience wth copy and paste in Tk
From: Alexandre Ferrieux on 22 Jun 2010 13:54
On Jun 22, 7:37 pm, davidlallen <d...(a)jendaveallen.com> wrote:
> I did not intend to step into a previous hot debate.  The reason my
> code switches in and out of nonblocking mode is because the nntp.tcl
> file has many other routines, not shown here.  There *seem* to be two
> different uses of "gets" from the channel; one is used many times to
> get a single line of reply which is expected to be short.  The second
> one which I showed is used to get a large multiline block of data,
> such as an entire usenet message or a range of multiple headers.  By
> volume, 99%+ of the data transfer uses the second one.  My hope was to
> make a more limited change and only switch to non-blocking mode for
> the second one.  Of course, if the shorter single line transaction is
> happening when the server dies, my application will still hang; but I
> hope that will become "much less likely".
>
> I have modified the code to check fblocked as shown in "man fblocked",
> but I have not had the opportunity to finish testing it.  I will
> update when I have a result.

To clarify, here is a drop-in replacement for [gets] taking just an
extra timeout argument:

oc gets_timeout {ch vline timeout} {
upvar $vline line
set id [after $timeout set ::_gt($ch) 1]
set blo [fconfigure $ch -blocking]
fconfigure $ch -blocking 0
fileevent $ch readable [list set ::_gt($ch) 2]
set err NONE
while {1} {
vwait ::_gt($ch)
if {$::_gt($ch)==1} {
set err TIMEOUT
break
}
set n [gets $ch line]
if {$n<0} {
if {[fblocked $ch]} continue
set err EOF
}
after cancel $id
break
}
fconfigure $ch -blocking $blo
switch $err {
NONE {return $n}
TIMEOUT {error TIMEOUT}
EOF {return -1}
}
}

-Alex
From: tom.rmadilo on 22 Jun 2010 14:20
On Jun 21, 11:43 pm, Alexandre Ferrieux <alexandre.ferri...(a)gmail.com>
wrote:
> On Jun 22, 2:10 am, "tom.rmadilo" <tom.rmad...(a)gmail.com> wrote:
>
>
>
>
>
> > On Jun 21, 9:21 am, davidlallen <d...(a)jendaveallen.com> wrote:
>
> > > Folks,
>
> > > I am using the nntp library in tcllib 1.12 for an application which
> > > fetches millions of headers from a news server.  It works; but
> > > occasionally the link fails and a transaction stalls.  Today this
> > > causes the application to simply hang forever, since the nntp library
> > > does not support any timeout.  You can see below the original code
> > > which simply uses "while not eof on socket".  Some other libraries,
> > > including the http library in tcl itself, provide a timeout.  So I
> > > have tried to modify the nntp code so that it will timeout, say after
> > > 60 seconds.
>
> > > My problem is that the new code seems to work correctly on small
> > > testcases but in the actual application it loses data, resulting in
> > > the downloaded binary file failing to unrar.  When there is no timeout
> > > situation, the application using the original nntp library does not
> > > have this problem.  So, somehow my fix "appears" correct but is not
> > > really.
>
> > > Can anybody suggest a different approach by inspection of the changes
> > > I have made?  I am certainly not an expert on vwait and fileevent.
>
> > > Original (see tcllib 1.12 nntp library):
>
> > >     proc ::nntp::fetch {name} {
> > >         upvar 0 ::nntp::${name}data data
> > >         set eol "\012"
> > >         if {![::nntp::okprint $name]} {
> > >             return ""
> > >         }
> > >         set sock $data(sock)
> > >         if {$data(binary)} {
> > >             set oldenc [fconfigure $sock -encoding]
> > >             fconfigure $sock -encoding binary
> > >         }
> > >         set result [list ]
> > >         while {![eof $sock]} {
> > >             gets $sock line
> > >             regsub -- {\015?\012$} $line $data(eol) line
> > >             if {[string match "." $line]} {
> > >                 break
> > >             }
> > >             if { [string match "..*" $line] } {
> > >                 lappend result [string range $line 1 end]
> > >             } else {
> > >                 lappend result $line
> > >             }
> > >         }
> > >         if {$data(binary)} {
> > >             fconfigure $sock -encoding $oldenc
> > >         }
> > >         return $result
> > >     }
>
> > > Modified:
>
> > >     proc ::nntp::fetch {name} {
> > >         upvar 0 ::nntp::${name}data data
> > >         if {![::nntp::okprint $name]} {
> > >             return ""
> > >         }
> > >         set sock $data(sock)
> > >         if {$data(binary)} {
> > >             set oldenc [fconfigure $sock -encoding]
> > >             fconfigure $sock -encoding binary
> > >         }
> > >         # davidlallen changes start here; see getline and timeout
> > > below
> > >         set data(after) [after 60000 [list ::nntp::timeout $name]]
> > >         set data(result) [list ]
> > >         fconfigure $sock -blocking off
> > >         fileevent $sock readable [list ::nntp::getline $name]
> > >         vwait ::nntp::${name}done
> > >         fileevent $sock readable {}
> > >         catch {after cancel $data(after)}
> > >         fconfigure $sock -blocking on
> > >         # davidlallen changes end here
> > >         if {$data(binary)} {
> > >             fconfigure $sock -encoding $oldenc
> > >         }
> > >         return $data(result)
> > >     }
>
> > >     proc ::nntp::getline {name} {
> > >         upvar 0 ::nntp::${name}data data
> > >         set eol "\012"
> > >         set sock $data(sock)
> > >         gets $sock line
> > >         regsub -- {\015?\012$} $line $data(eol) line
> > >         if {[string match "." $line]} {
> > >             set ::nntp::${name}done 1
> > >         }
> > >         if { [string match "..*" $line] } {
> > >             lappend data(result) [string range $line 1 end]
> > >         } else {
> > >             lappend data(result) $line
> > >         }
> > >     }
>
> > >     proc ::nntp::timeout {name} {
> > >         upvar 0 ::nntp::${name}data data
> > >         set data(result) {}
> > >         set ::nntp::${name}done 1
> > >     }
>
> > Problem is that Tcl's API isn't complete. You can have non-blocking
> > xyz, but you can't try to [read] some number of bytes and either
> > timeout or return a number of bytes less than requested. You should be
> > able to try to read X bytes and give up after some number of
> > milliseconds. If you try to read a line with [gets], you should be
> > able to give up after some number of milliseconds, regardless of the
> > number of reads which failed to find an EOL.
>
> > It is also possible to manipulate the Tcl [gets] API into adding
> > additional lines. Just send \r wait a while then send \n. The actual
> > byte stream has remained the same, but permissive applications might
> > add an additional line. This could affect when headers end and message
> > body starts.
>
> > Many current exploits are based upon this permissive "can't we all
> > just get along" programming model. Unfortunately you cannot avoid this
> > with Tcl's [gets] API. But you can use [read] one byte at a time.
>
> You did it again. Advising [read 1] because you just don't understand
> the nonblocking-gets/fblocked/chan-pending idiom. Please stop
> confusing people who are asking for help.

Right! You keep talking about [gets] and [fblocked], etc. when the
question is about a communication protocol. You have zero established
experience in interpretation of protocol messages, yet you continue to
somehow believe that [gets] solves all protocol details.

There are a few problems with the nntp code:

1: When using [gets] the "line" variable will not contain the eol
character(s), was it \n \r or \r\n? The only valid line ending in nntp
is \r\n, identical to email messages, unlike HTTP, which can accept
(stupidly) any line ending sequence. So the nntp code above which
looks for and tries to replace eol sequences are not needed.

2: Embedded \r or \n or NUL (and \t?) chars are illegal in nntp, if
they are present they should be replaced with a single space. If
[gets] is used with the default translation of {auto auto}, additional
lines may appear with invalid data.

3: The code doesn't account for line folding, so messages which use
line folding will at minimum confuse the client.

4: It isn't clear what effect switching channel encoding (from/to
binary) will have on processing while using [gets].

And [gets] also has issues. At least with nntp and email messages you
know that the only acceptable line ending sequence is <CR><LF>,
however, programmers could use the default translation {auto auto} and
interpret input incorrectly. Also, by design, or by random probability
a fileevent could trigger [gets] to examine a buffer which ends in \r
and assume a newline. The next chunk of data could start with \n,
which could also get interpreted as a newline. The result is an
additional (empty) line. In many protocols, this empty line signals
the end of headers and the start of the message body.

This is just a basic issue with [gets] which can't be solved with non-
blocking mode or [fblocked]. If <CR><LF> is the only allowed line
ending sequence, I think you can use [gets], but the -translation mode
probably needs to be {crlf crlf}.
From: Alexandre Ferrieux on 22 Jun 2010 18:02
On Jun 22, 8:20 pm, "tom.rmadilo" <tom.rmad...(a)gmail.com> wrote:
>
> > > Many current exploits are based upon this permissive "can't we all
> > > just get along" programming model. Unfortunately you cannot avoid this
> > > with Tcl's [gets] API. But you can use [read] one byte at a time.
>
> > You did it again. Advising [read 1] because you just don't understand
> > the nonblocking-gets/fblocked/chan-pending idiom. Please stop
> > confusing people who are asking for help.
>
> Right! You keep talking about [gets] and [fblocked], etc. when the
> question is about a communication protocol.

Nope. The question is precisely "How do I do a timeout-protected
[gets]".
I answered, first by explaining the role of [fblocked], then by
providing immediately usable code. Period.

> You have zero established experience in interpretation of protocol messages

Oh ? So you know me very well... Sigh.

> yet you continue to somehow believe that [gets] solves all protocol details.

Nope. I just keep explaining to you an idiom that definitely won't
stick to your over-experienced mind. It took me several hours to show
you that regexps could do the same as your handwritten [read 1]
automaton, I won't repeat the experience. Stay were you are, but by
all means stop giving counterproductive advice to newcomers.

>
> There are a few problems with the nntp code:
>

Wake up ! The OP didn't provide that code, he took it from tcllib. His
only interest, and a legitimate one at that, is a _modular_
replacement of a blocking gets by a timeout-protected one. "Fixing"
the surrounding code might be interesting, but is entirely off-topic.

-Alex

From: davidlallen on 26 Jun 2010 00:06
I added "fblocked" and it appears to be working. Thanks for your help!
First  |  Prev  | 
Pages: 1 2
Prev: XDrawImageString
Next: ISO: ideas for creating a more consistent user experience wth copy and paste in Tk