This property specifies the container memory for the map tasks in the MapReduce jobs launched during the cube build.
Values and behavior:
Any positive number, representing the number of megabytes of container memory.
If this property is not applied at any of the levels, it would fall back to the default YARN job configuration.
- Connection: If the property is set at the connection then the property value is applicable for all dataset build, cube build, or data profile jobs launched using Spark.
- Cube: If the property is set on a cube, then the value will override the connection level value for that cube’s build job.
- Dataset: If the property is set on a dataset then the value will override cube level value for that cube’s dataset build job.
NOTE: If the property is set on a dataset and a dataset is built, then the value will override the connection level value for that dataset build job.
Comes into effect:
This property comes into effect in all the cube build jobs. The value of the property can be changed at any time and will be respected in the next build.
Dependencies and related properties:
- The value of the mapreduce.map.memory.opts property should be 55-65% of the memory specified in this property.
- The property comes into effect only when kyvos.build.execution.engine is set as MapReduce
The default value of the property should work fine for most scenarios. If the cube build jobs fail due to memory-related failures, then the job memory can be increased using this property. Recommended range: 5120 to 15360