Eight annotated EISCAT experiments and some discussion about experiment checking and howto generate tarlan files

Eight different experiments will here be showed with rather heavily annotate experiment source files. These experiments (cp1l, dlayer, arc1, arc_dlayer,  meteor, arc_slice, tau1 and stef0fe) are using many of the different DSP modes that can be found on the refurbished EISCAT mainland system and ESR. These experiments can be used as a good boiler plate for many types of special programs. In this document the different files that are needed to setup and run a particular experiment is shown with annotation to be simpler to understand, for the D-layer and arc1/arc_slice experiment the decoding algorithm is also discussed.

When writing a completely new elan file please use the tau1.elan as a templet, as this experiment follows the proper naming convention and other structural rules.

Also a small discussion about DSP programming and checking is included.

Some small python scripts for making tarlan file and the t_to_ps file which is used when initilising guisdap have been added. tau_gen.py is the main python script which the user can change for different IPP, baud lengths and code sets. Also the tlan_mod.py file is needed and should reside in the same directory as the main script, this file contains all special defined subroutines which most times need no editing. Also suplied is the mostly used randomised alternating code set tautest.ac, which also needs to be in the directory where the tau_gen.py script is started. To run these script you need python which is found at http://www.python.org

Note! Some of the elan files may not be exactly the same as these currently running on the system.

Main land system

• CP1L This is the well known and much used common program version which has been the base for many different special programs on the old EISCAT system. On the refurbished DSP this experiment have been as closed as possible been copied. The Tarlan code is almost the same, except some new commands for the new system and some old commands that have disappeared due to the changed hardware. For the Tromsø, radar these files have been included here: cp1l.elan this is the elan file which as in the old system hold all different system settings which is done just before the actually experiments is started like load channel board with proper filters and NCO values, loading radar controllers move the antenna and polarizers and so on. Next we have the tarlan file which contains state commands both for the transmitter and receiver system: cp1l-u.tlan this file need to compiled by the tarlan compiler some notes on the tarlan compiler The main file for the DSP system which is the configuration file for the lag_wrap program which is running on the crate computer and taking care of the real time tasks like reading data from the channel boards and do necessary calculations and integration’s before sending the data to the main server which is running the decodump program which make all final calculations before the data are dumped as MATLAB file, for the cp1 experiment it is this file: cp1l-u.fil this file contains definitions for each channel board how many samples to take and what type of calculations should be made. This file needs to be compiled by using this command filan -f $PWD/cp1l-u.fil or if you are on the crate computer lag_wrap -c -f $PWD if you are in the directory where your source file are. After this compilation a new file called cp1l-u.DECO is found, this is the set-up file for the decodump application. This file will show how many data points that will be saved in the output matlab file, also the data positions are found here. The cp1l experiment have a alternating code part which is received on channel board 3, to be able to decode this the transmitted code sequence need to be known by the DSP software, in this file the complete code sequence is found with each line as one code set ac.txt, to find out the number of data points in each lag profile for the alternating code part a small Matlab script can be found here. The NCO settings which are downloaded for each channel board is ch1_cp1l-u.nco, ch2_cp1l-u.nco, ch3_cp1l-u.nco, and ch4_cp1l-u.nco. For the remote sites the hardware setup files is shown, the elan file cp1l.elan is the same for all sites. The tarlan file cp1l-r.tlan for the remote sites can be found here. The DSP source file cp1l-r.fil contain the same basic building blocks as for the Tromsø¸ site. This file must also be compiled to get the cp1l-r.DECO file which is the set-up file for the decodump program running on the server computer. The alternating code is received and decoded, ac.txt. Also for the remote station four channel boards are used and the frequency list are as follows ch1_cp1l-r.nco, ch2_cp1l-r.nco, ch3_cp1l-r.nco and ch4_cp1l-r.nco.For the E and the D-region a new experiment has been developed, the basic type is for the VHF radar using the whole antenna with a range resolution of 600 m covering 60 km to 126.6 km in range. Other more special purpose version have also been made as split beam and UHF versions. Here will the basic VHF experiment be discussed. The elan file can be found here dlayer.elan this file loads the proper settings for the hardware including radar controllers and filters/samplings speeds for the channel boards. This experiment used the newly developed codes which are FIR decoded and pulse to pulse lags are calculated, these codes have special features where even lags inside the code sets behave very good regarding range side loobs. Some notes about the coding technique used here and in the arc experiment can be found in fir_decoder. Due to the FIR filter normalization and the applied window, the zero lag estimate at 80 us and 32 us needs a certain scaling factors.
  

  Zero lag estimate at (us)	Scale factor to the 1.748 ms lag	Scale factor to the 4*1.748 ms lag
  32	10.408	40.371
  80	9.551	37.048

  Note that these number is only true for this code set code.txt and this window function Wj=(cos(pi(j/k-(1+1/k)/2)))^(1/4) where k is the length of the filter. If the code set is changed or the window function, the scaling factors needs to be recalculated. This can be done with this Matlab script calc_const.m. An example of a excerpt from a Matlab script lag_prof_mat.m showing one possible way of scaling and making lag profile matrices from D-layer raw data files.The tarlan code dlayer-v.tlan is rather simple but long, the transmission is done on a single frequency and in each ipp 150 signal samples are taken. Also in the first ipp 15 calibration samples are taken in the end of the cycle and in the ipp following 15 samples with signal plus background are taken, this is repeated in every ipp. For this data set only a zero lag without any decoding is calculated, as the calibration and background samples contain signal in this case from the F-region it can not be used for calculating system temperature, only for gain determinations. This system gain determinations is also used for the arc1 experiment. The DSP set-up dlayer-v.fil inside the file more information about the set-up can be found. The resulting dlayer-v.DECO after compilation of the dlayer-v.fil. To calculate and decode the signal for different lags, also different decoding files are needed, for the 80 us zero lag estimate for the D-layer long lags this FIR file is needed code2.txt, for the long fully decoded D-layer lags FIR file code1.txt, the FIR file for the short E-layer lags is this one code3.txt. These FIR decoder files have individual window functions applied on them. The same frequency is used for all the three channel boards so only ch1_dlayer-v.nco is shown.

• The arc1 experiment is special designed to study auroral arcs with good time resolution. This experiment have both high time resolution (0.44 s) and a good range resolution (0.9 km) without range side loobs, first range gate at 96 km and last at 422 km. This experiment closely resembles the D-layer experiment with 128 different 64 bit codes used code.txt, the difference is that in this experiment is that no inter pulse lags are calculated, how the decoding is done can be found in fir_decoder.txt. The experiment can be run on the UHF radar and also remote code are available. The best time resolution for Tromsø is 0.44 s, it can be changed by setting the proper res_mult number in the arc1-u.fil if this done a recompilation by filan is need to get the proper arc1-u.DECO , note that if this done the real-time graph set-up file need to be change and also the guisdap init file need to be changed. The tarlan file arc1-u.tlan is as simple as the dlayer one. The elan file arc1.elan is looks very similar to the elan for the cp1l experiment, only the experiment setup variables are changed inside the file arc1.elan for the remote sites. The DSP setup file for the remote stations are found here arc1-r.fil and compiled arc1-r.DECO, for the remote stations no time slicing is performed. The tarlan file for the remotes arc1-r.tlan.

• Another D-layer experiment are available where the same code set as in the arc1 experiment is used. The interpulse lags is calculated and also decoded and coherently integrated which can be used for calculating realy longs lags ( > 0.2 s). In this version the height resolution is 300 m, lag resolution is 1350 us and a total of 127 lags are calculated. It is also possible to form longer lags at a resolution of 0.1728 s (= 128*1350 us), also three 32 us short lags suitable for the E-region is also available. Range coverage is 60 – 140.1 km. The same code set as in arc1 is used code.txt.  As this experiment is very similiar to the arc1 experiment above, we will only discuss the DSP setup file arc_dlayer.fil, and especially how the really long lags at 0.1728*n s can be formed. The 32 us lag can be used as a pseudo zero lag for the long1350 us pulse to pulse lags. The scale factor is 31.75/6 ~ 5.3 compared to the D-layer part.

• The meteor experiment is a good example how raw data dumping is done. This experiment is a Swedish SP to do high resolution measurements of meteor head echoes. We will not go into details how this experiment is designed, instead we will see how the raw data handling is handled. The Tromsø¸ tlan is found here meteor-u.tlan the Baud length is 2.4 us, and a 32 bit code is used. The setup of the channel boards can be found in meteor.elan. Channel 1 is running at a sample speed of 0.6 us and this is the channel that will be used for the raw data taking, also for this data real time decoding is done and can be used for quick look on the real time graph. A slow channel, sampling at 2.4 us is also used for real time graphing. The experiment will dump data every 5 s, which is 1183 complete STC/BUFLIP cycles, with a synchronization time gap of 2.898 ms, to see more about data dumping. The DSP is set up meteor-u.fil to do both raw data dumping from the fast sampling channel 1 and also do FIR matched filter decoding for this data for quick look data displays, also data from the slow channel 2 with a sampling speed of 2.4 us is also decoded with a FIR filter. As explained in the meteor-u.fil file the raw data will be dumped in a separate data vector inside the matlab data file, this vector is called d_raw and is a vector consisting of real/imaginary pairs which are 16+16 bits signed integers, to keep the output matlab files as compact as possible. Please note some caveats that are discussed in the meteor-u.fil file.

• The tau1 experiment is shown mainly for the unified elan file. Using an unified elan file for all sites of the mainland radars makes updating of the experiments much esiear and fewer errors will sneak in. The tau1 experiment can be run both on the VHF and the UHF with the remotes also online. The VHF part are in principle the same exp as the tau2 family only more lags are calculated. For UHF the range coverage have been extended compared to the tau3 experiment, also the transmitted pulse is longer. The used elan file can be seen here tau1.elan, and the used scanpatterns.elan which contains the scans for cp1 cp2 and cp3.  For reference the dsp and tlan files for each site is also available.
  VHF: tau1-v.tlan, tau1-v.fil, tau1-v_split.fil
  UHF: tau1-u.tlan, tau1-u.fil
  Remote: tau1-r.tlan, tau1-r.fil

ESR

For the ESR radar a new experiment called arc_slice is available which is high time and range resolution without range side loobs, which is in principle the same as arc1 for the mainland system described above. For ESR clutter subtraction is necessary and are also used.  Only the differences between arc1 will be discussed in more detailed. At ESR a transmitter/receiver language called racelan is in use, also a new version called tarlan_esr is available which supports the new DSP hardware installed at ESR. These compilers resembles tarlan used at mainland. For the arc_slice experiment, arc_slice.tlan is the used racelan file. The clutter subtraction for this experiment works in the same way as for the dlayer experiment for ESR, the whole 64×128 code set is transmitted twice and the height region where one expect clutter are subtracted from each other in sample domain. The DSP setup file is found here arc_slice.fil. The arc_slice.elan are in principle the same as for the mainland system. At ESR the transmitter frequencies can be set arbitrary between 498-502 MHz with a resolution of 14 mHz,  this is done with the loadexciter elan command. This is the frequency table for arc_slice.frq, more information about the frequency file can be found here esr frq. The nco file ch1_arc_slice.nco have the same layout as for the mainland, for the ion line DSP box a 15 MHz AD with the last IF centered at 11.25 MHz as on mainland, is used.  The first LO is at 430 MHz and the second LO at 81.25 MHz for the ion line channel. A block diagram on the current IF chain can be seen here.
At ESR two DSP systems are available, one is taking care of the plasma line capabilities of the 32m antenna, and the second is handling the ion line data from the 32 m and 42 m systems. A block diagram of the ESR system can be found here. The steffe experiment is a multipurpose E and F layer (fully decoded gates from 105-216 km and 259-965 km in range with a resolution of 2.4 km) experiment with plasma line capabilities, especially for analyse calibration purposes.  The tarlan file for the 32 m (steffe@32m.tlan) antenna are using some new command for controling the plasma line channelboards, also inside steffe.elan which have controlling the loading and starting of the experiment do also have some new commands to take care of the plasma line DSP. A more throughly discussion about the new syntax regarding the plasma line DSP box can be found here.

Some checks of the DSP code

How to see if the crate/server computer will cope with your DSP calculation scheme. First thing to do is to run the *.fil program in the check mode on any of the available crate computers (v/u5011, k5011, s5011 or the e5011) that is done like this. Log into the crate computer go to your directory where you have your *.fil and do lag_wrap -c -f $PWD/[filename].fil, if the lag_wrap dont find any errors you will have an output like this:

kstdev@k5011:/kst/exp/ei/meteor2>lag_wrap -c -f $PWD/meteor-r.fil
file is /kst/exp/ei/meteor2/meteor-r.fil
read_sem is 1
Init structs has taken 1.3e+05 us
DECO file /kst/exp/ei/meteor2/meteor-r.DECO
CALIBRATING
Speed is non threaded 0.00019, threaded 0.00065, reading 0.00025 s
NON THREADED RUN
SPEED IS 0.00019 66896 1e+03 Mflops
This experiment will run non threaded

From this you can see that the calculation on the crate computer will take ~0.19 ms and the estimated time to read all your data from the channel boards will take ~0.25 ms. Also one can see that the lag_wrap program will not do the calculation in a threaded mode, i.e. It will do the calculation in sequence from for each channel board. Note that these figures you get here are under different conditions from what you will have when you run the whole experiment, the main difference will be that at each integration start you will have the data dump going to the server computer and also the STC interrupt handling. The rest of the output from lag_wrap -c is for debugging purposes and will at a later stage be removed.

The next step is to run your experiment and check the output from the running lag_wrap program.

This is a typical view of running an experiment under EROS4.

The wanted correlator (lag_wrap) window is by default ionized. Double click the correlator window, and a detailed view will be seen as shown below.

This is the output from lag_wrap program when running an experiment. From this we can see that the dump of data from the crate computer to the server have taken first 261 ms and the second time seen in the window 264 ms, also a dump speed is calculated (13 MB/s) the dump time is also seen (Thu Oct 31 09:08:29 2002). Total read cycle after intr max=627 and min 245 us means that after the crate computer have recognized the STC interrupt it have taken at maximum 627 us to read all data from all the channel boards used and the minimum time have been 245 us for the same operation. The maximum read time have been during the very beginning of the integration period when the data dump over to the server computer have been running in parallel with the data reading and calculations. Total calc cycle after intr max=1371 us and min=646 us is the total time after STC interrupt until the calculation of data from all channel boards are ready including data reading. This is the most interesting number, because this is the number that can not be bigger than the time between STC, one should have around 10-15 % spare time before the next STC interrupt, to be on the safe side. A fatal error to the experiment is if  the message “interrupt error” is seen persistent in the correlator window. This means that the lag_wrap process has got to few interrupts (STC) during the integration period. This error can result from setting the wrong number of nr_stc, nr_rep in the *.fil file, or  the lag_wrap and/or decodump process can not cope with the data handling. In this case recheck the *.fil file and also check the calculation and data dump time if these are to big, and try to reduce the computational and/or data dumping load.

The read speed from the channel boards can be as high as 30 MB/s but this under the condition that you have long data vectors to read, it is quite expensive to set up the VME data bus for the fast DMA transfers that we are using here. The speed on the VME bus will also depend on the calculation load on the crate computer. The dump speed to the server can be up to 15 MB/s but this is also depending on the calculation load on the crate computer.