Any MATLAB experts?

[MindBlast]

If so, I need some help.

I'm reading in a binary file (an XTF file actually), and I'm looking to store the data blocks in a structure array. The problem is that the structure array is "growing" every time the loop executes, and we all know how bad it is to grow arrays. I need to pre-initialize this thing, but how? Here's what I have and what I want:

The file is written like this, in binary:

fileheader with channelInfo as a substructure.
pingHeader. (this is the sonar data, it comes in pings or traces, there are 34801 of these in the particular file I'm looking at, but other files could have any number)
pingchannelHeader1 (it's a 2-channel system, there's one of these for each channel)
pingNdata1 (first channel data for the n-th ping)
pingchannelHeader2
pingNdata2 (second channel data for the n-th ping)



I've managed to extract all the header information, and then I loop going through to extract the ping header info from each ping, plus the data. I'm storing the data in a structure array called DATA.

The DATA structure array is eventually supposed to have 34801 (or however many there are) pings in it PER channel, so 2*34801. Each ping is 500 bytes long. I store the ping data in a 500 x 2 matrix (500 rows, 2 columns).

So the DATA structure looks like:

Ping1, Ping2, Ping3, ...., PingN

where each Ping is [500 x 2].

I can reference or call any ping by going DATA{k}, where k>=0.


My problem is that I need to initialize this structure array so the loop doesn't put Ping1 in, then Ping 2, and so on and grow the array each time.

I've tried using various things with the struct command, but it's no hope.

Assume I know how many pings (traces) there are, because I do. It's in the file header so I can call that variable.

Any help is appreciated.

 

[Corny]

hmm i don't know matlab but why is this a problem? you just need an array that stores an array.

oh in most languages a growing array is somehow possible as an arraylist or vector.

i know matlab is well .. pretty mathematical but it should have tools for that .. or not?


to draw it, it would look like this

array pings( 
			0	->	array ping(
								row 0	-> data
								row 1	-> data
								row ..	-> data
								row 500 -> data
							)

			1	-> array ping(
								row 0	-> data
								row 1	-> data
								row ..	-> data
								row 500 -> data
							)
							
			..	-> array ping(
								row 0	-> data
								row 1	-> data
								row ..	-> data
								row 500 -> data
							)
			34801 -> array ping(
								row 0	-> data
								row 1	-> data
								row ..	-> data
								row 500 -> data
							)							
			)

 

[MindBlast]

hmm i don't know matlab but why is this a problem? you just need an array that stores an array.

oh in most languages a growing array is somehow possible as an arraylist or vector.

i know matlab is well .. pretty mathematical but it should have tools for that .. or not?


to draw it, it would look like this

A structure field in MATLAB is an array that stores arrays. They can be any type or mix of types (float, double, unsigned integers, signed or unsigned characters, you name it).

Growing the array is possible. It's what I'm doing right now. But it's causing the computer to crash when the ping number gets too high, because the growing array is using too much resources.

What I wanted to do was initialize the array by filling it with zeros, and then replacing the zeros with the actual data. I just don't know the syntax how to do it. I drew a picture of what I want to have in the long run when it's filled. Imagine each ping is filled with zeros. That's what I want to do. I want to create the DATA structure and fill each ping with 2 columns of zeros. That's why I needed a matlab specific instruction set to help me initialize the array.

 

[thermodynamics]

I think you actually need to "flip" your structure. Rather than having individual arrays (ping1, ..., pingN), I think you should have DATA(pingnumber).{500,2}, or something like that. Typically when you have some number of identical things, the best way is to use DATA(somenumber).something.

When you add a record to DATA (let's say another ping), it's just another record in DATA, and this does NOT require a continuous block of memory. Records of structs can be scattered in memory.

I'm sorry I can't give you more help. It's been 6 months since I've touched MATLAB. But I did use a lot of structs and they worked very well for me.

 

[Corny]

in some languages you can define the size of an array somehow like that

int dataarray[][] = new int[200][200];

or similar (example is java).

matlab is a bit too different, i can't tell you any specifics here.

if it is using too much resources maybe you can consolidate the data somehow. the individual pings might be expressed just a one number.

25 37 -> 2537

this will spare the extra array structures and reduce the resource overhead.

 

[sid6p8]

Might not be thought through, but couldn't you calculate the number of pings in your file from the total file size?

 

[MindBlast]

Might not be thought through, but couldn't you calculate the number of pings in your file from the total file size?

Yup, but I already know how many there are (it's in the header info at the start of the file) . And whether the array is transposed or not doesn't really make a difference.




@ Thermodynamics

That's close to what I want. Thanks, I'll try something like that. It's just really getting the syntax correct.
The part that's slow is the fact that each time my loop goes through and reads a new data chunk (ie: a ping), it's creating a new 500 x 2 array and that's choking the computer. I need to have all 34801 arrays pre-made and filled with zeros before I start putting the data in them, rather than create them on the fly.

 

[sid6p8]

If you know how many ping there are, what's the problem? You can easily create empty arrays with the zeros command.

 

[winterknight]

DATA=cell(numberofpings,1);
for n = 1:numberofpings
    DATA{n}=zeros(500,2);
end

and then when you have the pings read in, you can just say
DATA{n}=pingdata;

remember you need to use curlies to subscript a cell array, otherwise you just get a smaller cell array back.

 

[sid6p8]

... or DATA=zeros(numberofpings, 500, 2)

 

[winterknight]

You could, but personally I'd rather stick with the cell array --
It reinforces the fact that it's "numberofpings" separate 500x2 records, rather than one big 3D matrix. Your way would work and might be slightly faster, but it's a lot less maintainable when you have to remember what it does six months down the line.

 

[Corny]

^ that saves resources.

and i am thoroughly confused that creating a new 500 x 2 array is choking the computer. that's basically nothing ..

 

[winterknight]

^ that saves resources.

Unfortunately, it's also not something that our previous ML guru seemed to grasp. And guess who got stuck with the job of maintaining his code?

 

[Corny]

hm maybe he was good with math then, but not with programming.

or imagination.

you just need to draw the filled 3 dimensional array to immediately see what's wrong there ..

 

[MindBlast]

DATA=cell(numberofpings,1);
for n = 1:numberofpings
    DATA{n}=zeros(500,2);
end

and then when you have the pings read in, you can just say
DATA{n}=pingdata;

remember you need to use curlies to subscript a cell array, otherwise you just get a smaller cell array back.

Thanks so much, this worked!

 

[MindBlast]

^ that saves resources.

and i am thoroughly confused that creating a new 500 x 2 array is choking the computer. that's basically nothing ..

It's not just a single 500 x 2 array. It's tens of thousands of them. In a loop. I just needed a logical place to store them all without losing their index.
The part where it says "Reading the Data" is where they're created, near the bottom of this.

And yes, I'm good at math and suck at programming. This is my original part of the code before I fixed it. SideScanData is where it's being stored.

EDIT: Some variables you see here aren't defined in the code below, that's because they're defined earlier in the code. This is just a chunk of my code.

%%%%%%%%%%%%%%%%%%%%%%%% START DATA PARSING %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
refendhdr=ftell(fidXTF); %remember where we were at end of header
fseek(fidXTF,0,'eof');
refendfilebytes=ftell(fidXTF); %finds out how many bytes the file is
fseek(fidXTF,refendhdr,'bof'); %sets us back to position at end of header

%*NOT DONE!* remember to add preallocation for data structure here! *NOT DONE!*
k=1;
while (ftell(fidXTF))<refendfilebytes
    %this stuff is exclusive per trace (common to both channels in that trace though)
    DATA.xtfpingheader{k}.MagicNumber=fread(fidXTF,1,'uint16');
    DATA.xtfpingheader{k}.HeaderType=fread(fidXTF,1,'uint8');
    DATA.xtfpingheader{k}.SubChannelNumber=fread(fidXTF,1,'uint8');
    DATA.xtfpingheader{k}.NumChansToFollow=fread(fidXTF,1,'uint16');
    DATA.xtfpingheader{k}.Reserved1=fread(fidXTF,2,'uint16');
    DATA.xtfpingheader{k}.NumBytesThisRecord(fidXTF,1,'uint32');
    DATA.xtfpingheader{k}.Year=fread(fidXTF,1,'uint16');
    DATA.xtfpingheader{k}.Month=fread(fidXTF,1,'uint8');
    DATA.xtfpingheader{k}.Day=fread(fidXTF,1,'uint8');
    DATA.xtfpingheader{k}.Hour=fread(fidXTF,1,'uint8');
    DATA.xtfpingheader{k}.Minute=fread(fidXTF,1,'uint8');
    DATA.xtfpingheader{k}.Second=fread(fidXTF,1,'uint8');
    DATA.xtfpingheader{k}.HSeconds=fread(fidXTF,1,'uint8'); %1/100ths of a second
    DATA.xtfpingheader{k}.JulianDay=fread(fidXTF,1,'uint16');
    DATA.xtfpingheader{k}.EventNumber=fread(fidXTF,1,'uint32');
    DATA.xtfpingheader{k}.PingNumber=fread(fidXTF,1,'uint32'); %!TRACE NUMBER!!%
    DATA.xtfpingheader{k}.SoundVelocity=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.OceanTide=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.Reserved2=fread(fidXTF,1,'uint32');
    DATA.xtfpingheader{k}.ConductivityFreq=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.TemperatureFreq=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.PressureFreq=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.PressureTemp=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.Conductivity=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.WaterTemperature=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.Pressure=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.ComputedSoundVelocity=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.MagX=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.MagY=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.MagZ=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.AuxVal1=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.AuxVal2=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.AuxVal3=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.AuxVal4=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.AuxVal5=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.AuxVal6=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.SpeedLog=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.Turbidity=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.ShipSpeed=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.ShipGyro=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.ShipYcoordinate=fread(fidXTF,1,'double');
    DATA.xtfpingheader{k}.ShipXcoordinate=fread(fidXTF,1,'double');
    DATA.xtfpingheader{k}.ShipAltitude=fread(fidXTF,1,'uint16');
    DATA.xtfpingheader{k}.ShipDepth=fread(fidXTF,1,'uint16');
    DATA.xtfpingheader{k}.FixTimeHour=fread(fidXTF,1,'uint8');
    DATA.xtfpingheader{k}.FixTimeMinute=fread(fidXTF,1,'uint8');
    DATA.xtfpingheader{k}.FixTimeSecond=fread(fidXTF,1,'uint8');
    DATA.xtfpingheader{k}.FixTimeHSecond=fread(fidXTF,1,'uint8');
    DATA.xtfpingheader{k}.SensorSpeed=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.KP=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.SensorYcoordinate=fread(fidXTF,1,'double');
    DATA.xtfpingheader{k}.SensorXcoordinate=fread(fidXTF,1,'double');
    DATA.xtfpingheader{k}.SonarStatus=fread(fidXTF,1,'uint16');
    DATA.xtfpingheader{k}.RangeToFish=fread(fidXTF,1,'uint16');
    DATA.xtfpingheader{k}.BearingToFish=fread(fidXTF,1,'uint16');
    DATA.xtfpingheader{k}.CableOut=fread(fidXTF,1,'uint16');
    DATA.xtfpingheader{k}.Layback=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.CableTension=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.SensorDepth=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.SensorPrimaryAltitude=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.SensorAuxAltitude=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.SensorPitch=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.SensorRoll=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.SensorHeading=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.Heave=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.Yaw=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.AttitudeTimeTag=fread(fidXTF,1,'uint32');
    DATA.xtfpingheader{k}.DOT=fread(fidXTF,1,'float32');
    DATA.xtfpingheader{k}.NavFixMilliseconds=fread(fidXTF,1,'uint32');
    DATA.xtfpingheader{k}.ComputerClockHour=fread(fidXTF,1,'uint8');
    DATA.xtfpingheader{k}.ComputerClockMinute=fread(fidXTF,1,'uint8');
    DATA.xtfpingheader{k}.ComputerClockSecond=fread(fidXTF,1,'uint8');
    DATA.xtfpingheader{k}.ComputerClockHSec=fread(fidXTF,1,'uint8');
    DATA.xtfpingheader{k}.FishPositionDeltaX=fread(fidXTF,1,'int16');
    DATA.xtfpingheader{k}.FishPositionDeltaY=fread(fidXTF,1,'int16');
    DATA.xtfpingheader{k}.FishPositionErrorCode=deblank(char(fread(fidXTF,1,'1*uchar=>uchar'))');
    DATA.xtfpingheader{k}.ReservedSpace2=fread(fidXTF,11,'uint8');
    for m=1:1:header.NumberOfSonarChannels
        
       %%Reading the data
               current=ftell(fidXTF);
               if DATA.xtfpingheader{k}.xtfpingCHANheader{m}.NumSamples~=0
                      size2read=DATA.xtfpingheader{k}.xtfpingCHANheader{m}.NumSamples;
                  SideScanData{m}(:,k)=fread(fidXTF,size2read,'uint8');
        End
    end
    k=k+1;
end

 

[winterknight]

hm maybe he was good with math then, but not with programming.

or imagination.

you just need to draw the filled 3 dimensional array to immediately see what's wrong there ..

Quite.

I doubt that it's the 500x2 array that's the chokehold, BTW. I'm guessing that the parent cell array has grown beyond the space originally allocated for it, and the interpreter is having to find a new (34k * sizeof whatever-the-internal-data-structure-is, and I doubt it's just a simple pointer) block of contiguous memory to hold it.

 

[MindBlast]

Quite.

I doubt that it's the 500x2 array that's the chokehold, BTW. I'm guessing that the parent cell array has grown beyond the space originally allocated for it, and the interpreter is having to find a new (34k * sizeof whatever-the-internal-data-structure-is, and I doubt it's just a simple pointer) block of contiguous memory to hold it.

The problem was that there was *no* space originally allocated for it, because I didn't know the syntax, lol.

 

[winterknight]

The problem was that there was *no* space originally allocated for it, because I didn't know the syntax, lol.

When you autovivify (yes, it's a word) a matrix or array, Matlab makes what it thinks is a educated guess at how big it might become, and automatically allocates that amount of memory. You can grow it up to that size without trouble, but any bigger and it has to move the whole thing to a new block of memory.

If you are growing a cell array up to 34000 entries one at a time, this is going to happen a lot. I dunno if you've been on any "official" MathWorks training, but pre-allocation is one of the things they really hammer in to you.

 

[MindBlast]

When you autovivify (yes, it's a word) a matrix or array, Matlab makes what it thinks is a educated guess at how big it might become, and automatically allocates that amount of memory. You can grow it up to that size without trouble, but any bigger and it has to move the whole thing to a new block of memory.

If you are growing a cell array up to 34000 entries one at a time, this is going to happen a lot. I dunno if you've been on any "official" MathWorks training, but pre-allocation is one of the things they really hammer in to you.

No, I've never been to their training, but I already try and preallocate stuff where possible. My problem was that I didn't know the syntax for preallocating structures, or 3d arrays.