This property specifies how to handle invalid rows when reading data during a dataset or cube build job. The rows that have any a mismatch in the data type or in the number of columns (specifically in case of files created on HCatalog) are considered invalid rows.
Values and behavior:
- True: Kyvos will skip the invalid data rows while reading data during the build process. The build will fail only if the number of skipped records exceeds the threshold defined in the kyvos.records.skipbadrows.threshold property. In this case, the processed row count as shown in the build summary may be lesser than the total number of rows in the source data.
- False: Kyvos will not skip any invalid row and immediately fail the build as soon as an invalid record is encountered.
- Connection: If the property is set at the connection then the property value is applicable for all dataset build, cube build, or data profile jobs.
- Cube: If the property is set on a cube, then the value will override the connection level value for that cube’s build job.
- Dataset: If the property is set on a dataset, then the value will override cube level value for that cube’s dataset build job.
NOTE: If the property is set on a dataset and a dataset is built, then the value will override the connection level value for that dataset build job.
Comes into effect:
The value of the property can be changed at any time (not recommended) and comes into effect in the next build.
Dependencies and Related Properties:
If data is likely to have invalid rows that can be skipped, then set this property to true with an appropriate threshold. We do not recommend to change the value of this property in between builds as a change may result in inconsistent aggregated values.