• Print
• .

Compression

Windowing, filter tails, decimate quirks

UPDATE

24bit v. 32 bit data

At LPWR the data files are pretty big, but the data actually fits into 24 bits; so if we could save the data in 24 bits, we could keep full resolution and shrink the data a bit.

The first way I looked at doing this takes advantage of the fact that matlab has save and load functions that work with 32bit data but not 24bit data. I converted the final wera variable into individual bytes, cut out the extra byte, then restrung those bytes together into a new, shorter series of 32bit numbers. Then you can save it just like any other 32bit dataset, though it needs to be changed back before use. I have already tested this, and it works perfectly. There is no difference between the data that has been converted to 24bits (and back to 32bit) and the 32bit data I started with.

The problem is that the file saved is the same size once this is done. I've checked over all the variables to make sure that I wasn't saving anything unnecessary, etc. The space actually set aside for the two different wera variables (bytes when you use whos) is as expected, and saving the files as '-v6' gives file sizes as expected. The only thing I can think of is that the compression that matlab applies to mat files takes advantage of the unused bits in the 32bit data to compress it further, while it can't do so in the 24bit case. The difference is actually quite large considering the small file sizes.

The other thing I checked into was taking advantage of the fact that fwrite can write files in 24bits. So I had matlab write out the wera variable into 24bits as a separate file, and also with 32bits. The 32bit version of wera took 16 Mb, while the 24bit version took up 12 Mb. When compressed, the 32bit version took 2.8 Mb, while the 24bit version took 2.4 Mb. The rest of the Matlab file takes up 674 Kb.

The numbers for LPWR are listed in the table, except for the size of the Matlab file without the variable timechirp (which is equivalent to wera), which is 208 Kb.

If using 24bits is really what we want to do, then the second method is the only one that saves space. Reading the data from the file is very simple as well, so it's not a hassle.

All methods used 32x compression - note that I also had to change two things in the radar_header.m file of LPWR: I changed OVER from 2 to 4 and COMP_FAC from 16 to 8; I read in the documentation for decimate that it works best when the factor is less than 13. As a result of these changes, I also had to change at the very end the dds chirp length to * 2 instead of * OVER.

Changes in SHIFT

Pierre said that the SHIFT parameter is vastly different with the newer Dtacqs

If you compare int32 versions of a chirp from KAL and LPWR, you find that there is something funny going on.

The bytes do not line up between the two data sets. What happens if we look at all 8 bytes that went into the original dataset?

There aren't really any systematic differences noticeable here.

Now let's look at what the different SHIFT parameters do to both datasets (5, 10, 15):

SHIFT moves things to different bytes. LPWR uses SHIFT=15, while KAL uses SHIFT=8.