Recursive Struct Operations

[0xfbad]

Currently dealing with a file format where I want to look recursively at structs and create custom logic based off that. For example, say num_sections is 3 and I want to create SectionData structs at data offset for each SecondaryStruct, how would I go about doing it? Looking at the documentation I did see for loops, but they didn't seem to work (no errors but it just didnt create any new struct objects), any suggestions?

TL:DR: I want to essentially do SectionData @ header.sections[i].data_offset with the provided snippet

struct SecondaryStruct {
  // ...
  type::Hex<u32> data_offset;
}

struct MainStruct {
 // ... 
 u32 num_sections;
 SecondaryStruct sections[num_sections];
}

struct SectionData {
  // ...
}

MainStruct header @ $;

[paxcut]

Loops cannot be used inside of structs. Instead you want to create an array of structs which will function as the loop. The function std::core::array_index() tends to be very useful for this kind of setup because it gives you access inside the struct you are making an array of to the current index being processed so you can use it to index placed arrays that maybe located at the parent struct using the parent keyword. If still unsure I can try to post some code if you can give a minimal example description of what you are trying to do.

[0xfbad]

That makes sense, though I'm not seeing much documentation on the array loops, only seeing the Functions - Loop Control bit. Are there any other examples of arrays being used recursively?

I guess what I want to do is

import std.array;

struct SecondaryStruct {
  // ...
  type::Hex<u32> data_offset;
}

struct MainStruct {
 // ... 
 u32 num_sections;
 std::Array<SecondaryStruct, num_sections> sections;
}

struct SectionData {
  // ...
}

MainStruct header @ $;

// then create an array? of SectionData structs for the offsets we have. i.e.:
// SectionData sectionA @ header.sections[0].data_offset;
// SectionData sectionB @ header.sections[1].data_offset;
// SectionData sectionN @ header.sections[N].data_offset;

There may be additional things I want to do with those sections in terms of using them in functions to extract data. So it might be worthwhile making that also an array. Hopefully that gives a bit more context of what my goal is here, let me know if not.

[paxcut]

In a minimal example you remove all unnecessary details and focus on the problem at hand. One part you don't specify is how the data is stored at those addresses so I just used a generic name for a type (DataStruct) . The following pseudo code will show how one may approach these type of formats using the concepts described above.

import std.core;

struct SecondaryStruct {
  // ...
  u32 data_offset;
}


struct SectionData {
  DataStruct data @ parent.sections[std::core:array_index()].data_offset;
}

struct MainStruct {
 // ... 
 u32 num_sections;
 SecondaryStruct sections[num_sections];
 SectionData sectionData[num_sections];
}

MainStruct header @ 0;

// then create an array? of SectionData structs for the offsets we have. i.e.:
// SectionData sectionA @ header.sections[0].data_offset;
// SectionData sectionB @ header.sections[1].data_offset;
// SectionData sectionN @ header.sections[N].data_offset;

[0xfbad]

I meant that SectionData is the literal struct I want to place at each data_offset, I don't have another struct. My SectionData is just a random u32 for marking purposes within the file, nothing of significance.

I'd want it to be also outside of the header scope, so not a descendant of MainStruct.

Maybe closer to

struct SecondaryStruct {
  // ...
  type::Hex<u32> data_offset;
}

struct MainStruct {
 // ... 
 u32 num_sections;
 SecondaryStruct sections[num_sections];
}

struct SectionData {
  u32 testl
}

MainStruct header @ $;
// SectionData data_sections[header.num_sections] @ header.sections[ .. some way of creating SectionData at each offset ..].data_offset;

[paxcut]

My aim is not to write the exact code you want to end up using. My only intent is to show how to read an array of structs that have their addresses specified by a previously read array of offsets. That is only the part that you are writing fake code for presumably because you are unsure how ImHex can do that operation. My idea is that once that is understood, the other details for your particular pattern should no be that complicated.

Here is the problem. Youwant to perform this operation

SectionData data_sections[i] @ header.sections[ i].data_offset;

for all values of i between 0 and header.num_sections as if it was being done in a loop. As explained, loops cannot be used inside structs and you need to create an array of structs to accomplish the equivalent work done in the loop the ImHex way.

When I say array of structs I don't mean an array of SectionData structs. If you make an array of them you will only be able to place the first one and the rest will simply follow each other.
To achieve the placing of each array element at the corresponding offset found in the array of offsets a totally new structure, call it whatever you want but I'll use DataStruct, must be created. Inside it you place one of the section array elements using the function shown above to get the corresponding offset. As the array is being created, the array_index() function returns the value of the current array index and since the two indices in the loop were the same hare you just use it directly. This way we place one structure at one address. How and where you get the offsets is not important for the explanation. Using parent is cleaner and preferable but when starting to write patterns there is a tendency to assign data to global variables because it makes things easier. Easier means less powerful too. Again this is not important for the current discussion. So this is what i would think is the explanation you seek.

struct DataStruct {
    SectionData data_sections @ header.sections[ std::core::array_index()].data_offset;
};
This is not optional. You can't place the entire array at sets of addresses directly. You have to do it one at a time. You wrap it up by creating an array of Datastructs. The array obviously has to be the same size as the array of offsets. 
```cpp
// Inside some struct that you need to place
DataStruct ds[header.num_sections];
...

You don't have to necessarily see the DataStruct variable in your pattern. There are a number of ways like [[inline]] that can let you see exactly what you want in the pattern data window.

To be honest, this idea of using offsets to place structs is almost never needed. All input files can be read from the beginning to the end once you learn whats located where. Also it is preferable to end up with 1 pattern that is placed at zero and reads the entire file. This makes modifying the pattern a lot easier and makes the code look cleaner. If you browse through the pattern library provided by imhex you will see a majority of patterns follow those principles as close as possible. At the very least that was my experience when I did that.

[0xfbad]

Okay thanks that made a lot more sense. I just wasn't completely sure on the syntax. I just wanted to avoid using a struct to call another struct at a location. I know inline exists but it in experimenting I didn't like how it represented the start address within the pattern view.

[paxcut]

The thing is that it can't be avoided for this use case. I don't know what difficulties you are having using inline. There are several ways to apply it effectively and each is designed to have the best side effects that users have asked for. You can use it to suppress nodes, you can also use it to eliminate array indices. inline doesn't modify any address in the pattern data view. I haven't run into a situation that couldn't be handled with combinations of attributes. It sounds like you didn't experiment extensively. Here is an example of what I was able to do with the following code

import std.core;

struct SecondaryStruct {
  // ...
  u8 data_offset;
}[[inline]];

struct SectionData {
u8 t[3];
}[[sealed]];


struct DataStruct {
  SectionData sectionData @ parent.sections[std::core::array_index()].data_offset;
}[[inline]];

struct MainStruct {
 // ... 
 u8 num_sections;
 SecondaryStruct sections[num_sections];
 DataStruct data[num_sections][[inline]];
};

MainStruct header @ 0;

this is the pattern data view

there are no problems with addresses and you can't even see arrays or the structs you want to hide. show only what you want.

[0xfbad]

Ah, I see. I think the difference was I was adding the inline tags to the objects, not the structs themselves. That seemed to fix all of my issues with the addresses.

[paxcut]

Attributes can be applied to the pattern or to the members. The effect they have on each mode varies with attribute, but it generally does what you would expect. If you are interested in learning, I really recommend looking at and running patterns in the library to see a wider range of what Pattern Language can do. Reading the docs only gives you a very basic look.