The rsrc_static Attribute

Appliance

This page explaines the rsrc_static attribute, which is used in multiple macros:

• Hubs
• Links
• Non-Historized Links
• Record Tracking Satellites

Purpose

The rsrc_static attribute is only used, when an entity is loaded by multiple sources. The main purpose of the rsrc_static attribute is, to optimize performace of incremental loads.

Typically you only want to look at those records of the stage, that are newer than the maximum load date inside the target entity. This behaviour epecially improves performance, if the underlying staging areas hold a large amount of historical loads, since the number of rows scanned is drastically reduced.

If an entity is loaded from multiple sources, each source should be filtered down to records, that are newer than the maximum load date of this exact source, not the overall maximum load date inside the target entity. To allow that behaviour, the maximum load date for each different source needs to be calculated.

Sometimes the record source column inside one staging area is not always the same, but has some dynamic parts in it. Sometimes the rsrc column includes the ldts of each load and could look something like this: ‘SALESFORCE/Partners/2022-01-01T07:00:00’.

Obviously the timestamp part inside that rsrc would change from load to load, and we now need to identify parts of it that will be static over all loads. In the shown example it would be ‘SALESFORCE/Partners’. That expression now needs to be enriched with wildcard expressions to catch all occurances of the static part of the record source. In BigQuery the wildcard would be ‘*’ and therefor the rsrc_static would be ‘*/SALESFORCE/Partners/*’.

Special Cases