Page tree
Skip to end of metadata
Go to start of metadata

Description:

This property specifies the source for fetching the metadata details of data having a dynamic schema where the structure is different between the file and the HCatalog definition.

Values and behavior:

  • Table - Metadata will come from the HCatalog metastore.

  • File - Metadata will come from the Parquet file.

Default value:

Table

Scope:

Registered File: If the property is set on a for a particular registered file, then the value will be applicable for a particular file, and will override the value applied (if any) at the connection.

Comes into effect:

The value of the property can be changed at any time. If the schema changes, then the registered file must be saved again with the updated field lists to use them further.

Dependencies and related properties:

The registered file must be built on a dynamic schema from a Parquet file from the HCatalog table.

Recommendation:

Generally, the metadata should come from HCatalog as HCatalog can handle dynamic schema. This property should only be changed when the file has a proper structure, and HCatalog does not.

  • No labels