This property specifies the number of invalid rows that can be skipped with the kyvos.records.skipbadrows property. Once this threshold is reached, the dataset or cube build will fail.
Values and behavior:
Any positive number. The build will continue till the time when skipped record count is lesser than or equal to the specified value, else would fail.
When set to 0, there is no limit to the number of rows that may be skipped.
100 - Once 100 invalid rows are found, the build will fail.
- Connection: If the property is set at the connection, then the property value is applicable for all dataset build, cube build, or data profile jobs launched using Spark.
- Cube: If the property is set on a cube, then the value will override the connection level value for that cube’s build job.
- Dataset: If the property is set on a dataset, then the value will override cube level value for that cube’s dataset build job.
NOTE: If the property is set on a dataset and a dataset is built, then the value will override the connection level value for that dataset build job.
Comes into effect:
This property comes into effect only when kyvos.records.skipbadrows is set to TRUE. The value of the property can be changed at any time and will be respected in the next build.
Dependencies and related properties:
The threshold is applied per MapReduce Mapper or Spark Map task. For a build that has a threshold of 10,000 and 10 Mapper tasks, the build will not fail until a single task encounters 10,000 invalid rows. Each of the tasks can have 9,000 invalid rows, which means 90,000 invalid rows in total; and the build will succeed.