batch_cluster_config now available, Resolved Plan Hooks issue for Windows environments, Adds .tectonignore to ignore paths and files
batch_cluster_config now available for Feature Table declarations
You can now include batch_cluster_config
configuration in Feature Table declarations. By specifying an EMRClusterConfig
or DatabricksClusterConfig
configuration or referencing an ExistingClusterConfig
, the ingest materialization jobs will run on a cluster of workers with the specified configuration options like instance type, instance size, and spark configurations.
FeatureTable(
name="user_page_click_feature_table",
entities=[content],
schema=schema,
online=True,
offline=True,
owner="example@tecton.ai",
batch_cluster_config=DatabricksClusterConfig(
instance_type="m5.2xlarge",
number_of_workers=2,
spark_config={
"spark.executor.memory": "7000m",
},
),
)
Resolved Plan Hooks issue for Windows environments
Previously including a plan.py
would cause problems for users in a Windows environment. This issue is resolved in SDK version 0.0.54
.
Adds .tectonignore to ignore paths and files
You can now add .tectonignore
to your feature repository to ignore specified paths and files, similar to .gitignore
. See the usage guide for an example.