scidoc1430 — Document

S-LINK NOAO email to Miquel

January 1, 2008

Hi Miquel, Hi Laia,

These results look very promising .. Can I asume that the forward link limit is principally limited by the 40MHz clock used for the tests ? This would still allow a 33.6 (or 67.2) MPixel flow at a confortable 85% utilization - which allows 10% for housekeeping (telemetry, etc.) and 5% margin.

I also have been doing a little hacking and this may be a very good time to bring you up to date. I'll make a clarification with regard to the reset capabilities first first. There is no mechanism available for the DHE hardware to remotely reset the systran PCI card. Unfortunately, when the DHE Systran is reset (by a power or boot event, or by setting PIO_1 and PIO_2 true for > 15ms), The systran emits a random burst of data onto the fiber. This has to be manually cleared by the application before synchronization can be established.

I fully agree with your intended use of the control signals. It makes good sense. Do you want to establish the codes relevent for MONSOON (i.e. data packet, control packet, etc.) ?

I'll let Nick answer your question with regard to data buffering at the LDC.

Okay, my progress:
1). I have the S-LINK physically mated to the master control board and powered up. To achieve this I had to make the following modifications:
a). Physically remove component F2.
b). Wire (on the bottom of the board) TP6 to TP26.
c). Remove standoffs from S-LINK card (they appear to be too long anyway).
d). Remove jumper JP2
e). Protect the exposed PCB lands and support the S-LINK board by applying double sided tape to the area close to the MCB fron panel.

2). I have begun to build two firmware modules for the MCB fpgas. I intend to use the LSC LR1 and LR3 return lines to control the following functions:
LR3 LR1 Action
0 0 Do nothing
0 1 Put LSC into test mode
1 0 Send synthetic data *
1 1 abort any action and reset the MCB.
I am planning to use a 15ms delay to after any return line change of state to assure that I interpret the correct code. Does this sound reasonable to you ?

Synthetic data:
I will build two generators into the logic. One to generate a high data rate and one a medium data rate.

The medium data rate will comprise of a realistic stream of data representing a 144 MiPixel focal plane read out at 250KPixel/channel/sec on 36 channels. This equates to a fiber load of 36 MByte/second average for approx. 17 second readout (this is the specification for DES).

The high data rate will generate synthetic data to represent a 64 MiPixel focal plane read out through 32 channels at 800KPix/channel/second. This equates to a fiber load of 102.4 MByte/second average for approx. 2.6 seconds readout (ODI / LSST specifications).

To accomodate these two simulations I will develop two MONSOON configuration file sets for your use. To switch between modes in the firmware I plan to use the state of pin 1 on J4 (also on J9) so that when high (default), high speed simulation is used and when low (pin 1 shorted to pin 2 on J4 or J9), medium speed data is generated. Do these plans seem usable to you ?

Once again, thank you for your news, I am sure the MONSOON system has been shipped and is on it's way to you so not too much time before we can begin to get our hands dirty for real :-)

Regards (from Hot and Sunny La Serena !!),
Peter.

On Mon, 06 Feb 2006 13:23:11 +0100
Miquel Barcelo <barcelo@ifae.es> wrote:
Hi all,

We are still waiting for a basic MONSOON system to work with but meanwhile e have been taking a look to the software and working with the return link.

- Return link tests

As promised we implemented a faster "serial" protocol for the return link, using the 4 return lines available. AFAIK, these lines are equivalent to the PIO lines of the systran board. The tests have been successful and we got an average transfer rate for the return link of 100KiBytes/sec (Ki = 1024) and a forward transfer rate of 151MiBytes/sec (Mi = 1024*1024) at the same time, during 17 hours and with no data transfer errors. Some tech details about the tests can be found at the end of the mail if you're interested.
Notice that for this speeds we need to use the four return lines of S-Link so in the MCB the pins 2 and 6 of J6/P6 connector must be wired to the FPGA.

- (remote) Link reset using PIO lines

This comes in response to some doubts from Nick that I'm not sure if I fully understand. Link can be reset from both sides (PC/FILAR and MCB/HOLA). We can send a reset signal from PC to the DHE using one of the return lines, but this would decrease the BW for the return link. Instead of this, the current protocol for the return link sends some extra bits for each 32 bit word which are now unused. These bits can be used to send a "low level" reset to the DHE FW. It looks like there's also needed to be able to reset the PAN interface from the MCB, is this right? The only way to do this is using the forward link, so some "reset" command should be implemented.

- Distinguishing data from command replies

I think that the S-Link equivalent for the SYNC signal is the CONTROL signal. As the name indicates this signal is used to indicate if the data sent is plain data or control words but it also it is used to mark the beginning / ending of a data block and to get information about the data errors detected during data transfer. And work using data blocks is not only a good practice but almost mandatory in S-Link. There are some bits that can be freely used in the control words and we can use them to indicate if the data corresponds to a command reply or pixel data or even a reset from the DHE to the PAN, if necessary. This way we can have "pixel data blocks" and "command reply blocks" that can be distinguished at PAN.

- Software integration

Many low level functions of the systran and the S-link PC drivers/lib look pretty similar but it looks like the way the data is read is different. I still don't have all the Systran documentation so maybe I'm wrong, but it looks like Systran keeps the received data in the boards buffers until a read command is issued from the user-level software.
In S-Link this works in a different way as the application provides some memory addresses to the FILAR in anticipation to the data actually being received. The low level driver and library automatically transfer the received data from the S-Link FILAR board to these memory addresses without waiting for the user-level application so the buffers at the board can be efficiently used. The application occasionally checks which addresses have been filled with data and provides new addresses to the S-Link library/driver tandem to store the upcoming data while the application processes the received data.
If the systran works the way it looks to me its behavior can be emulated by using some PC memory as a temporary data buffer before passing the data to the higher layers of the monsoon system. This of course implies extra memory operations that can reduce the available PC memory bandwidth for the application.

And I think that's all by now. Please send us any comments / corrections / questions about these points or any other thing.
Cheers,

Miquel

* Some details about the return link test:

To test the return link the FW uses two memory buffers. One of them is filled with the received data from the return lines decoder and, at the same time, the data in the other memory buffer is sent using the forward link to the PC. When the first buffer is filled they swap roles so the last "data block" received from the return lines is now sent to the PC. The data blocks sent using the return lines are generated using a simple pseudo-random algorithm, its seed depending on the clock of the PC. The first word of the data block is the seed and the next words are the pseudo-randomly generated data. When the data block comes back to the PC through the forward link is checked that the seed corresponds to the last data block sent and that each data word follows the sequence for that seed in addition to the standard S-Link error checks.
The FW blocks that decode the data from the return link and send it back using the forward link are written in VHDL. In order to test its portability the same test has been performed using an Altera's FLEX20k (Pulsar board) and a Cyclone (dev kit from Altera). Both tests lasted more than 17 hours and the average data transfer rates were 151MiBytes/sec for the forward link and 100KiB/sec for the return link, with no errors found.
Two threads are used in the software to send and receive data. Actually, the max. speed for the return link can be greater than 100KiB/s but then the extra PCI transfers can slowdown the forward link, so the thread controlling the return link is put to sleep some miliseconds after each data block.



Peter Moore - Detector Engineer - ETS.
Email: pmoore@noao.edu
Smail: National Optical Astronomy Observatory,
Casilla 603, La Serena, Chile.
Phone: +56 51 205233 or +56 51 205294
Faxes: +1 520 318 8303
Skype: pmoore159

status

About the Document

Id: scidoc1430
Release date: January 1, 2008, 5:21 pm

Document Formats

scidoc1430.txt
8.95 KB