• Print
• .

Prepending WERA header to D-Tacq files

 --------Message --------
 Subject: 	quick fixes for removing extra byte
 Date: 	Sun, 1 Dec 2019 23:00:31 -1000
 From: 	Pierre Flament <pflament@hawaii.edu>
 To: 	radlab

 1. through shell script:

 sunrise:/home/sunset0/data/kna/realtime> cat cleanraw
 #!/bin/csh -f

 dd if=$1 bs=512 count=1 >! $2

 dd if=$1 bs=512 skip=513 iflag=skip_bytes >> $2

 sunrise:/home/sunset0/data/kna/realtime> cleanraw 20130932330_kna.RAW.old 20130932330_kna.RAW

 sunrise:/home/sunset0/data/kna/realtime> ls -l 20130932330_kna*RAW*
 -rw-rw-r-- 1 wera users 251658752 Dec  1 22:44 20130932330_kna.RAW
 -rw-rw-r-- 1 wera users 251658753 Apr  3  2013 20130932330_kna.RAW.old

 2. through matlab m-file

 sunrise:/home/sunset0/data/kna/realtime> cat cleanraw.m
 % cleanraw: remove extra header byte
 % define filein and filout before calling

 IQ=2;           % number of channels to make pair 
 NANT=16;        % the WERA number of antennas
 MT=1920;        % the number of WERA samples/chirp 
 NCHIRP=2048;    % the number of chirps 

 % read binary wera data in .RAW format of short int16

 f=fopen(filein,'r','ieee-le')

 header=uint8(zeros(512,1));
 skip=uint8(zeros(1,1));
 raw=int16(zeros(IQ*MT*NANT*NCHIRP,1));

 header=fread(f,512,'uint8=>uint8');
 skip=fread(f,1,'uint8=>uint8');
 raw=fread(f,IQ*MT*NANT*NCHIRP,'int16=>int16');

 fclose(f);

 % output the wera .RAW file

 f=fopen(fileout,'w','ieee-le');

 fwrite(f,header,'uint8');
 fwrite(f,raw,'int16');

 fclose(f);

 -------- Forwarded Message --------
 Subject: warning: invalid Kaena "kna" radial data 2013 to 11/30/2019
 Date: Sun, 1 Dec 2019 17:59:57 -1000
 From: Pierre Flament <pflament@hawaii.edu>
 To:   hfrnet administrators <hfrnet.administrators@sio.ucsd.edu>

1. Synopsis

2. Actions

3. Q&A __

1. Synopsis:

• erroneous insertion of a spurious byte between the ascii meta-data header and the raw antenna time series was discovered
• extra byte was probably inserted when configuring the site, by using vi editor instead of bvi editor on the header file, automatically appending a new-line "\n" upon write
• each integer*16 time series sample consists then of 8 of the least significant bits from the previous sample, the sign bit and 7 of the most significant bit of the current sample, with the new sign bit being random, and the old sign bit falling in the middle
• spurious clipping and random sign changes resulted in corrupt Fourier transforms and corrupt phases, preventing beamforming (see attached maps before and after repair, which was performed on 20193341500).
• all Kaena radial data provided are invalid, including derivative data partly based on them (vector currents, assimilating model output)

What was done:

 char*19  char*478  int16                          int16                         int16

 "time"   "header"  ("\n" sample1.byte1)           (sample1.byte2 sample2.byte1) (sample2.byte2 sample3.byte1) ...

What should have been done:

 char*19  char*478  int16                          int16                         int16

 "time"   "header"  (sample1.byte1 sample1.byte2)  (sample2.byte1 sample2.byte2)  (sample3.byte1 sample3.byte2) ...

If the good data was:

 int16			int16			int16

 (sabcdefg hijklnop)	(sabcdefg hijklnop)	(sabcdefg hijklnop) ....

It was erroneously written as:

 int16                  int16                   int16

 ("\n" sabcdefg)        (hijklnop sabcdefg)	(hijklnop sabcdefg)   ....

(where letters denote 0's or 1's with a the most significant and where s is the sign bit)

2. Actions:

• all Kaena data and derivative until 11/30/2019 should be expunged immediately from all servers
• all users registered to have downloaded the data should be warned not to use them

3. Q&A:

• is the current real-time data for Kaena good? YES, the bug was fixed at 15:00 UTC on 11/30/2019.
• can the archived data be repaired? NO; because we use truncated Fourier transforms, we cannot reconstitute the original time series and byte-shift them
• are other Hawaii radars affected? NO; it has been verified that only Kaena is affected and of course combined Kaena-Kapolei vector currents
• is there a mean of recovering valid data? may be YES; until September 2016, some if not most of the raw time series were archived at 6 kHz on external disks, which can be slowly reloaded and reprocessed with the correct header; since October 2016, the raw time series were decimated in matlab to 750 Hz, and archived as compressed .mat files on external disks, easier and faster to reload and reprocess
• what is the schedule for recovering/reprocessing the available radial data? data since October 2016 will be reloaded and reprocessed by the end of April 2020 at the latest; data prior to September 2016 will only be reloaded and reprocessed depending on staff availability
• what is the schedule for regenerating combined vector currents? radlab does not generate vector currents, nor has software to do so; once the radial data is reprocessed and validated, it will be made available to the HFRnet server and to the PacIOOS server, which have their own policy/procedures for reprocessing older data
• how could such problems be avoided or detected earlier? they are currently unavoidable because we lack reasonable support staff for real time operations; we currently only have 1 technical staff for maintenance of the radars, but no staff for programming, archiving and quality control, thereby left to chance