Prev: Analog to Digital conversion using 18F452
Next: flash memory write endurance
From: Tim Wescott on 27 Nov 2009 12:12
On Fri, 27 Nov 2009 09:28:59 -0600, Vladimir Vassilevsky wrote:

> dlopez wrote:
>> Hi,
>> I'm having trouble figuring out how to architect a robust system using
>> the FTDI chips in FIFO mode. I'm pretty sure some of you have been
>> through this before.
>> I'm using the FT2232H in synchronous FIFO, but my concerns are general.
>>
>> The system has PC <-> FT2232H <-> FPGA, and is used to both
>> stream/receive data between the PC and the FPGA.
>>
>> I'm wanting to stream continuous data at about 1MB/s each way, WHILE
>> also sending various control commands to the FPGA (low rate).
>>
>> Now the FPGA will have some kind of state-machine that reads data from
>> the FTDI. It needs to decide if this 'data' is for streaming, or if
>> it's a command to write to a register. I thought of building and
>> sending a fixed 'packet', that always contain say 32 kBytes of
>> streaming data, and 8kBytes of control, or some variant of this.
>>
>> Now my concern is that there is no synchronization between the FPGA
>> state machine and the FT2232H FIFO. If something goes wrong, the FPGA
>> could start mixing the data.
>>
>> I'm used to I2C, or SPI, where there is either a CLATCH event, or a
>> START condition, that marks the beginning of a new transfer. Here,
>> there is none of this.
>>
>> Any idea?
>
> There are three simple solutions:
>
> 0. Do it like Hayes modems. Data mode, Command mode, "+++" and such.
>
> 1. Byte stuffing.
> Reserve a byte (like 0x00, 0xFF or 0x55) as "Escape" character, so the
> next byte will be interpreted as a "command".
>
> 2. Packetization. Packet, length, checksum, timeout. The receiver
> automatically synchronizes to the packet structure.
>
>
> Vladimir Vassilevsky
> DSP and Mixed Signal Design Consultant http://www.abvolt.com

I generally prefer to do this with packetization, although the other
protocol types have their place.

Figure out the longest latency you can have between the PC internals
issuing a command and the FPGA actually getting it -- this will help to
set the largest possible packet. Then figure out the best packet
structure for overhead, ease of design, and state machine size in the
FPGA. Then implement and go.

Some folks like to have separate checksums for the header and the whole
packet -- it lets the receiver figure out early that it's receiving
gibberish; it's nice to know you don't have to find a home for those
30000 bytes of data, but it is particularly useful if there's a mechanism
for it to send a NACK on the back-channel to abort a long packet.

--
www.wescottdesign.com
From: dlopez on 27 Nov 2009 12:32
Hi,
When you say 'packetization', does it mean you have only and only ONE
packet structure? For example:

myPacket = [startByte, dataStream(32 kbytes), dataControl(512 bytes),
Checksum, StopByte]

startByte: 0x55
Stream: whatever random data to stream
dataControl: say a bunch of [address, data] to be written in the FPGA.
CRC: a few bytes here
stopByte: 0x99

Then you send this packet over and over, regardless if the PC is needing to
send control data or not?

Actually, after re-reading your email, I think you were saying to build
different packets, some for only streaming, some for control etc...The
streaming one are the longest, so they add latency before the next command
could be received.

Are there any hands-on litterature on this type of stuff? I don't want a
'communication systems' texbook:)

Thanks!
Diego


>I generally prefer to do this with packetization, although the other
>protocol types have their place.
>
>Figure out the longest latency you can have between the PC internals
>issuing a command and the FPGA actually getting it -- this will help to
>set the largest possible packet. Then figure out the best packet
>structure for overhead, ease of design, and state machine size in the
>FPGA. Then implement and go.
>
>Some folks like to have separate checksums for the header and the whole
>packet -- it lets the receiver figure out early that it's receiving
>gibberish; it's nice to know you don't have to find a home for those
>30000 bytes of data, but it is particularly useful if there's a mechanism

>for it to send a NACK on the back-channel to abort a long packet.
>
>--
>www.wescottdesign.com
>



---------------------------------------
This message was sent using the comp.arch.embedded web interface on
http://www.EmbeddedRelated.com
From: Mel on 27 Nov 2009 12:42
Tim Wescott wrote:

> If your data can be _truely_ random this opens up the possibility for the
> pathological case where you have a chuck of data that's rich in escape
> characters that suddenly bogs down your transmission. The longer I stay
> in this business the more the phrase "that's really unlikely" translates
> in my head to "that'll get missed by testing and caught by some important
> customer right before my big vacation".

Good point. Also when data aren't random enough. Therefore it's a good
idea NOT to use 0x00 or 0xFF for escape bytes. And avoid 0x55 (and 0xAA) if
you suspect people use them for dummy data.

My baptism of fire here (I think I've mentioned this before) was when
something in the channel/hardware/driver was injecting extra 0x00 bytes into
the data stream. So we escaped the zero data and stripped 0x00 out of the
stream. To avoid explosive escaping we xored all the data pre-protocol with
0x55 to make escaped zero values less common.

Mel.


From: Tim Wescott on 27 Nov 2009 12:58
On Fri, 27 Nov 2009 11:32:23 -0600, dlopez wrote:
(top posting fixed)
>>I generally prefer to do this with packetization, although the other
>>protocol types have their place.
>>
>>Figure out the longest latency you can have between the PC internals
>>issuing a command and the FPGA actually getting it -- this will help to
>>set the largest possible packet. Then figure out the best packet
>>structure for overhead, ease of design, and state machine size in the
>>FPGA. Then implement and go.
>>
>>Some folks like to have separate checksums for the header and the whole
>>packet -- it lets the receiver figure out early that it's receiving
>>gibberish; it's nice to know you don't have to find a home for those
>>30000 bytes of data, but it is particularly useful if there's a
>>mechanism
>
>>for it to send a NACK on the back-channel to abort a long packet.
>>
> Hi,
> When you say 'packetization', does it mean you have only and only ONE
> packet structure? For example:
>
> myPacket = [startByte, dataStream(32 kbytes), dataControl(512 bytes),
> Checksum, StopByte]
>
> startByte: 0x55
> Stream: whatever random data to stream dataControl: say a bunch of
> [address, data] to be written in the FPGA. CRC: a few bytes here
> stopByte: 0x99
>
> Then you send this packet over and over, regardless if the PC is needing
> to send control data or not?
>
> Actually, after re-reading your email, I think you were saying to build
> different packets, some for only streaming, some for control etc...The
> streaming one are the longest, so they add latency before the next
> command could be received.
>
> Are there any hands-on litterature on this type of stuff? I don't want a
> 'communication systems' texbook:)

I don't know if there's literature -- I learned this stuff on the job by
looking at existing protocols. If someone knows of something I'd like to
see it!

I was suggesting a variable-length packet structure -- so at the least
you have a length field, a data field (which may be zero bytes if you
want to allow that) and a checksum field. Normally you also add a data
type field, to tag the 'meaning' of the data, and often you add some sort
of a "start of packet" flag if you want the receiver to be able to sleep
until it sees a potential new packet.

The minimum that I'd suggest would be

<start of packet><length><type><data><CRC>

Use a CRC (cyclic redundancy check) instead of a checksum -- it'll be
much more likely to catch an error. I'd suggest two bytes each for
length, type, and CRC, and one byte for the start of packet -- but a one-
byte type field often works well (until you're trying to find a home for
the 257th message type -- then it's a disaster). When folks use a start-
of-packet byte it's often something obvious like 0x55 or 0xaa, etc.

On a really noisy link, or one that absolutely positively must
synchronize quickly, it's often useful to put a CRC on the header:

<SOP><length><type><CRC><data><CRC>

This adds two bytes to each packet as overhead, but it vastly simplifies
sorting through a stream of data for a packet start, as it lets you
discard a bad packet a lot quicker.

--
www.wescottdesign.com
From: dlopez on 27 Nov 2009 13:33
Tim,
Thanks a lot for the reply, it's really helpful.

>On a really noisy link, or one that absolutely positively must
>synchronize quickly, it's often useful to put a CRC on the header:
>
><SOP><length><type><CRC><data><CRC>
>
>This adds two bytes to each packet as overhead, but it vastly simplifies
>sorting through a stream of data for a packet start, as it lets you
>discard a bad packet a lot quicker.

Trying to confirm my understanding:

So let's say for some reason that things are not in-sync (maybe power
up...). The FPGA would be reading bytes from the FTDI. It wants to see a
<SOP> byte, so would keep reading until it gets it.
1) If it's a legitimate <SOP> we're in-sync again.

2) If it was just a random data byte with the same value, the FPGA would
start reading the <length> and <type>, which would be 'garbage' but there
is no way to know. Then it reads <CRC>, which would hopefully fail, and
start this process again. I guess there IS a slight chance that this CRC
passes, so we would keep reading the data and we might be doing some very
bad things with it.

Are you saying you should NOT execute any command until the last <CRC> is
read and passes? I guess this requires a buffer in the FPGA that is the
length of the longest packet. This could be quite big in the case of the
streaming packet...


Diego

---------------------------------------
This message was sent using the comp.arch.embedded web interface on
http://www.EmbeddedRelated.com
First  |  Prev  |  Next  |  Last
Pages: 1 2 3 4
Prev: Analog to Digital conversion using 18F452
Next: flash memory write endurance