The Tables Reconciliation is a feature that enables analysis of changes made to the connected source data platform tables upon which your project’s tables are based. It provides a valuable solution for two primary scenarios, throughout the project lifecycle:
Note: The Tables Reconciliation process is relevant only for those tables that were built based on source data platforms, using the DB Explorer, since introducing this feature - Fabric 8.0. While an LU's table is built, the platform saves additional data - the source data, schema and table it was created upon - as part of the table's metadata in Fabric; this enables the Reconciliation process to find any existing mismatches.
The first step of the Reconciliation feature is to find the mismatches. This can be done according to 1 of the following 3 options: Find only added/removed tables' columns (added to the source or removed from it), Find unused columns in the project tables (comparing with the source), Find all mismatches between the project and the source tables (added/removed/unused).
You can trigger the mismatches finder either in schema level or per table:
Activating the mismatches finder in the schema - the main tables design surface - provides a mismatches overview of the LU (aka Data Product) schema's tables.
You can activate the mismatches finder on a specific table, rather than on an LU schema:
In the schema, choose a table and click to open its Properties panel.
In the Project tree, choose the table and open it; then click to open its Properties panel.
Note: The mismatches finding feature looks either directly at the source data platforms or at the Catalog's Discovery outcome (when the latter is being used).
Upon completion of the mismatches finding process:
When the mismatches finding process is activated, the Reconciliation action icon turns orange, hinting to the user that the Reconciliation process is now active. Being in this state:
Upon opening a table that was marked by the Reconciliation Mismatches Finder, the Table Editor opens in a Reconciliation Mode.
The green and red colored table's entries, are those that are identified with changes in the data source. Gray colored entries are those that exist in the source data platform tables, but not in the corresponding LU table.
The above example illustrates a case where there are various changes (added, removed, not-in-use columns). Usually, within a specific table, fewer changes will be made.
Tables' columns are represented in the Table Editor as entries/lines; the corresponding terminology used in this article is columns or entries.
According to the Reconciliation Mismatches Finder results, as shown in the Table Editor screen, you can choose - for each one of the mismatched columns - to either take action or postpone such decision and do it later, as follows:
Added column(s) to the source table (green-colored entries):
Decide Later whether to add it or not. For example, you may prefer to postpone your decision for later, as you wish to consult with your team first as to whether this column shall be used in your project.
Add the column also to the project's LU table.
Do not Add the column to the project's LU table.
According to this decision, Studio will not alert you later about this column's existence in the source. However, you will still be able to find it when searching for unused columns.
Removed columns from source (light red-colored entries):
Decide Later whether to remove it or not. For example, you may prefer to postpone your decision for later, as you wish to consult with your team first as to whether this column shall be removed or maybe it is still needed.
Preserve the column, although it was removed from the source. This is probably useful in cases where this column is in use within the implementation and therefore needs to remain. Note that it is your responsibility - from now on - to populate its data as the data will not be retrieved from source.
According to this decision, Studio will not alert you later about the removal of this column from at the source.
Not In Use at the LU table, while existing at the source data table:
Once done with handling a table - either if you have made a decision regarding the table's entries and have taken action or not - you can close it by 1 of the following 2 options:
The Table Properties Panel's second tab - Data Source Origin - provides an overview and indicates upon which source data platform tables, this LU table is based and built.
The tab's top header shows when was the last time that the table was updated during the Reconciliation process, and it also enables you to activate the Reconciliation Mismatches Finder on this specific table, as explained above.
The tab's body lists the source data platform columns, organized by Interface, the schema and the table in the source data platform.
In case you deal with a large schema, which contains many tables, you can easily find these tables that were found during the Reconciliation Mismatches Finder process, by using the Highlighting option:
At the top schema toolbar look for the Highlighting action icon.
Click on the arrow next to this icon and choose the Mismatches vs Source option.
As a result, the relevant tables are highlighted while other are grayed-out.
In case the Reconciliation process is not activated in the schema - this option is disabled.
The Tables Reconciliation is a feature that enables analysis of changes made to the connected source data platform tables upon which your project’s tables are based. It provides a valuable solution for two primary scenarios, throughout the project lifecycle:
Note: The Tables Reconciliation process is relevant only for those tables that were built based on source data platforms, using the DB Explorer, since introducing this feature - Fabric 8.0. While an LU's table is built, the platform saves additional data - the source data, schema and table it was created upon - as part of the table's metadata in Fabric; this enables the Reconciliation process to find any existing mismatches.
The first step of the Reconciliation feature is to find the mismatches. This can be done according to 1 of the following 3 options: Find only added/removed tables' columns (added to the source or removed from it), Find unused columns in the project tables (comparing with the source), Find all mismatches between the project and the source tables (added/removed/unused).
You can trigger the mismatches finder either in schema level or per table:
Activating the mismatches finder in the schema - the main tables design surface - provides a mismatches overview of the LU (aka Data Product) schema's tables.
You can activate the mismatches finder on a specific table, rather than on an LU schema:
In the schema, choose a table and click to open its Properties panel.
In the Project tree, choose the table and open it; then click to open its Properties panel.
Note: The mismatches finding feature looks either directly at the source data platforms or at the Catalog's Discovery outcome (when the latter is being used).
Upon completion of the mismatches finding process:
When the mismatches finding process is activated, the Reconciliation action icon turns orange, hinting to the user that the Reconciliation process is now active. Being in this state:
Upon opening a table that was marked by the Reconciliation Mismatches Finder, the Table Editor opens in a Reconciliation Mode.
The green and red colored table's entries, are those that are identified with changes in the data source. Gray colored entries are those that exist in the source data platform tables, but not in the corresponding LU table.
The above example illustrates a case where there are various changes (added, removed, not-in-use columns). Usually, within a specific table, fewer changes will be made.
Tables' columns are represented in the Table Editor as entries/lines; the corresponding terminology used in this article is columns or entries.
According to the Reconciliation Mismatches Finder results, as shown in the Table Editor screen, you can choose - for each one of the mismatched columns - to either take action or postpone such decision and do it later, as follows:
Added column(s) to the source table (green-colored entries):
Decide Later whether to add it or not. For example, you may prefer to postpone your decision for later, as you wish to consult with your team first as to whether this column shall be used in your project.
Add the column also to the project's LU table.
Do not Add the column to the project's LU table.
According to this decision, Studio will not alert you later about this column's existence in the source. However, you will still be able to find it when searching for unused columns.
Removed columns from source (light red-colored entries):
Decide Later whether to remove it or not. For example, you may prefer to postpone your decision for later, as you wish to consult with your team first as to whether this column shall be removed or maybe it is still needed.
Preserve the column, although it was removed from the source. This is probably useful in cases where this column is in use within the implementation and therefore needs to remain. Note that it is your responsibility - from now on - to populate its data as the data will not be retrieved from source.
According to this decision, Studio will not alert you later about the removal of this column from at the source.
Not In Use at the LU table, while existing at the source data table:
Once done with handling a table - either if you have made a decision regarding the table's entries and have taken action or not - you can close it by 1 of the following 2 options:
The Table Properties Panel's second tab - Data Source Origin - provides an overview and indicates upon which source data platform tables, this LU table is based and built.
The tab's top header shows when was the last time that the table was updated during the Reconciliation process, and it also enables you to activate the Reconciliation Mismatches Finder on this specific table, as explained above.
The tab's body lists the source data platform columns, organized by Interface, the schema and the table in the source data platform.
In case you deal with a large schema, which contains many tables, you can easily find these tables that were found during the Reconciliation Mismatches Finder process, by using the Highlighting option:
At the top schema toolbar look for the Highlighting action icon.
Click on the arrow next to this icon and choose the Mismatches vs Source option.
As a result, the relevant tables are highlighted while other are grayed-out.
In case the Reconciliation process is not activated in the schema - this option is disabled.