Resolved How to access an element from a non-sequential 2D array

[SankaUD]

Hi All,

This may sound stupid to you, as I'm a casual programmer. I have a two-dimensional array of values. Where rows are angles in decimal degrees, increments can be by 1 degree, 5 degrees even 10. Columns are also organized similar manner. I know how to access a generic two-dimensional array, but not sure how to go about this one, say if I want to access a value referenced at 5 degrees (row) and 100 degrees (col). Do I need to store the row and column values in two different matrixes and use a for loop to iterate through data? Any suggestions?

Ex: (x) represents a decimal number

5 0 10 20 30 40
10 x x x x x
15 x x x x x
20 x x x x x
25 x x x x x
30 x x x x x

 

[Skydiver]

What is your objective? Fast access or low memory usage?

 

[SankaUD]

What is your objective? Fast access or low memory usage?

Thanks for responding to my query @Skydiver , it's fast access. FYI, the angles of the measurements taken for both vertical and horizontal planes are written to the text file with the data itself. I am capable of extracting the data, but I need to know what's the fastest way to reference locations as, sometimes, calculations need to be done thousands of times.

 

[Skydiver]

If you are looking for the fastest access possible, just create a 31x41 array. It's going to be a big chunk of memory where about 90% of the array will never be used, but indexing into arrays is very very very fast, where the only thing throttling it is the bounds checks that C# always imposes (unless you start using unsafe code and pointers) So: 31 rows x 41 columns x 8 bytes + array overhead = ~ 10,168 bytes . That's approximately 10KB. Considering that modern machines start at 4GB of RAM, you should have the head room.

If you want to try to conserve on the memory, you could try for 2 dictionaries plus a 5x5 array. The first dictionary maps the row value to a row index into the 5x5 array. The second dictionary maps the column value into a column index of the 5x5 array. So your memory cost would be:
5 rows x 5 columns x 8 bytes + array overhead = ~200 bytes
2 dictionaries * 5 entries * (~16 bytes entry overhead + 4 byte key + 4 byte value) = ~240 bytes
To give you about only 540 bytes. Just a little over half a kilobyte. Over an order of magnitude less in terms of memory usage.
But for every lookup you have to do, you'll be doing two dictionary lookups before indexing into the array. The dictionary look ups are not quite as fast as direct array index look ups. In my guesstimation, it's at least an order of magnitude more CPU operations.

Also consider that simply starting up any .NET program takes up several megabytes of RAM, so it's very possible that an extra 10KB may not make that much difference.

 

[SankaUD]

If you are looking for the fastest access possible, just create a 31x41 array. It's going to be a big chunk of memory where about 90% of the array will never be used, but indexing into arrays is very very very fast, where the only thing throttling it is the bounds checks that C# always imposes (unless you start using unsafe code and pointers) So: 31 rows x 41 columns x 8 bytes + array overhead = ~ 10,168 bytes . That's approximately 10KB. Considering that modern machines start at 4GB of RAM, you should have the head room.

If you want to try to conserve on the memory, you could try for 2 dictionaries plus a 5x5 array. The first dictionary maps the row value to a row index into the 5x5 array. The second dictionary maps the column value into a column index of the 5x5 array. So your memory cost would be:
5 rows x 5 columns x 8 bytes + array overhead = ~200 bytes
2 dictionaries * 5 entries * (~16 bytes entry overhead + 4 byte key + 4 byte value) = ~240 bytes
To give you about only 540 bytes. Just a little over half a kilobyte. Over an order of magnitude less in terms of memory usage.
But for every lookup you have to do, you'll be doing two dictionary lookups before indexing into the array. The dictionary look ups are not quite as fast as direct array index look ups. In my guesstimation, it's at least an order of magnitude more CPU operations.

Also consider that simply starting up any .NET program takes up several megabytes of RAM, so it's very possible that an extra 10KB may not make that much difference.

Your first option is not clear to me. Say, I want to access vertical angle (row) 50 and horizontal angle (col) 25. How would you go about a dataset similar to the one below? I just need clarification on the method, not the code.

 

[cjard]

Are you asking how to read the data out of excel, or how to access it once you've read it into a 2D array?

 

[Skydiver]

Ah, so you have more than just those angles from 5 to 30 for vertical and 0 to 40 for horizontal. That may dramatically increase memory usage. (Up to over half a megabyte.) But anyway, you would have something like this pseudo-code:

var table = new double[181, 361];

// Read in values from source file
Open source text file with values
Read horizontal angle header values (0, 5, 10, 15, 25, ... 360) into list
while not end of file:
    Read vertical angle row header value
    for each horizontal angle header value:
        read matrix entry
        table[vertical, horizontal] = entry

// to look up a value
var foundValue = table[50, 25];    // should contain 0.98.

 

[cjard]

I think, by that point, I'd be indexing a dictionary with a tuple, but worth perf testing

Also, just because the values run 5,10,15,20..360 doesnt mean we need an array 360 wide. 360/5 will suffice (If you want the 50 value, it's at index (50/5)). Same for y indexes. If they're regular a bit of math can save a lot of allocation

 

[SankaUD]

Are you asking how to read the data out of excel, or how to access it once you've read it into a 2D array?

No, the data is saved tab-delimited text file. Excel image to illustrate how the data should be arranged. I know how to import the measured angles for rows, columns and data into individual arrays/lists. It's about how finding a faster/easier way to search for the data referenced by row and col value. The only way I know how to achieve this is by iterating through row and column matrix to find indexes and then iterating the main data set using the indexes to find the value. For ex. If I want the value referenced by row 50 and col 25, I need to find the index of row 50, and then find the index of col 25, then using those indexes, I can find the values I want from the main data table.

 

[SankaUD]

Ah, so you have more than just those angles from 5 to 30 for vertical and 0 to 40 for horizontal. That may dramatically increase memory usage. (Up to over half a megabyte.) But anyway, you would have something like this pseudo-code:

var table = new double[181, 361];

// Read in values from source file
Open source text file with values
Read horizontal angle header values (0, 5, 10, 15, 25, ... 360) into list
while not end of file:
    Read vertical angle row header value
    for each horizontal angle header value:
        read matrix entry
        table[vertical, horizontal] = entry

// to look up a value
var foundValue = table[50, 25];    // should contain 0.98.

Thanks @Skydiver , I will try this. I need to check the performance of this method with the Dictionary method as well to see if the two methods will have significant performance differences.

 

[SankaUD]

I think, by that point, I'd be indexing a dictionary with a tuple, but worth perf testing

Also, just because the values run 5,10,15,20..360 doesnt mean we need an array 360 wide. 360/5 will suffice (If you want the 50 value, it's at index (50/5)). Same for y indexes. If they're regular a bit of math can save a lot of allocation

Agree, I will try the method suggested by @Skydiver and use an expression to calculate angle intervals to size the array.

 

[Skydiver]

Assuming rows increment by 10 and columns increment by 5: 180/10 = 18 rows, 360/5 = 72 columns: ~11,096 bytes .

But then on your opening post, you had indicated that the angles may increment by 1, 5, or 10 degrees, and hence my suggested solution of just creating an array with a lot of wasted space.

 

[SankaUD]

Assuming rows increment by 10 and columns increment by 5: 180/10 = 18 rows, 360/5 = 72 columns: ~11,096 bytes .

But then on your opening post, you had indicated that the angles may increment by 1, 5, or 10 degrees, and hence my suggested solution of just creating an array with a lot of wasted space.

Yes, testing angles can be vary on the testing laboratory.

 

[cjard]

No, the data is saved tab-delimited text file. Excel image to illustrate how the data should be arranged.

Show us the full exact file contents please, not an Excel mock-up of how you think the data should be arranged in memory

Also, please use [CODE] ... [/CODE] tags around the file contents

 

[SankaUD]

Show us the full exact file contents please, not an Excel mock-up of how you think the data should be arranged in memory

Also, please use

 ...

tags around the file contents

Below is an extract from the actual dataset:

0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.0 45.0 50.0 55.0 60.0 65.0 70.0 75.0 80.0 85.0 90.0 95.0 100.0 105.0
110.0 115.0 120.0 125.0 130.0 135.0 140.0 145.0 150.0 155.0 160.0 165.0 170.0 175.0 180.0
0.0 90.0 180.0 270.0 360.0
1400.734 1398.456 1374.237 1333.317 1276.078 1202.066 1110.461 1006.302 892.741 774.296 654.736 537.447 425.874 321.486
225.278 139.505 69.087 21.752 1.288 0.671 1.035 1.758 2.496 3.384 4.353 5.237 6.225 7.106
8.049 8.949 9.726 10.572 11.275 12.057 12.680 13.094 13.268
1400.734 1384.893 1362.734 1324.096 1268.993 1197.711 1108.523 1005.510 892.906 775.262 656.559 540.323 429.048 324.650
228.363 142.541 70.829 21.450 0.517 0.000 0.000 0.296 1.220 2.301 3.231 4.334 5.278 6.294
7.270 8.144 9.073 9.831 10.733 11.464 12.147 12.679 13.268
1400.734 1396.554 1370.509 1327.727 1268.035 1192.095 1098.487 991.733 876.026 756.559 636.745 520.386 409.607 305.437
209.379 124.615 56.134 14.073 0.000 0.000 0.000 0.775 1.854 2.841 3.960 4.973 6.082 7.088
8.049 9.008 9.823 10.702 11.487 12.199 12.844 13.267 13.268
1400.734 1379.348 1351.499 1307.695 1247.560 1170.390 1076.484 970.428 856.312 738.308 620.713 506.334 397.666 296.162
203.108 121.258 54.462 15.163 1.017 0.993 1.431 1.991 2.797 3.618 4.481 5.444 6.283 7.229
8.037 8.838 9.577 10.310 11.102 11.743 12.414 12.900 13.268
1400.734 1398.456 1374.237 1333.317 1276.078 1202.066 1110.461 1006.302 892.741 774.296 654.736 537.447 425.874 321.486
225.278 139.505 69.087 21.752 1.288 0.671 1.035 1.758 2.496 3.384 4.353 5.237 6.225 7.106
8.049 8.949 9.726 10.572 11.275 12.057 12.680 13.094 13.268

As you can note, the first two lines have vertical angle measurements taken every 5 degrees, and the horizontal plane, the third line, is only done every 90 degrees. following four datasets, terminated with newline characters, represent measurements taken at each vertical angle for four horizontal planes.