Forum Discussion
Post ProdigyDo pipelines replicate data or just metadata about the source dataset?
The intent is you only use the prod workspace for consumption...dev is for development and test is for testing. All downstream dataflows, datasets and reports should always be consuming from the prod workspace. Only promote to prod what has been tested.
There no additional cost associated with storing data in all three environments--depending on your data source there may be other costs associated with extraction, but using pipelines enables you to test without overwriting prod.
An option to limit data is to create a parameter is in your dataset (like "DevelopmentMode", and set to "dev", "test", or "prod"). Have your queries check this parameter and when "dev" only import a small numer of rows (I use 10), if "test" then import a medium-high amount (I use 10,000), and if "prod" or blank, don't impose a filter at all.
You can then assign deployment rules to set the parameter in each stage's Workspace in the pipeline settings.
Solution SageCorrect, data is not copied. Each Workspace only has the data contained within it--be it dataflows or datasets. The idea is to just work with assets within one environment: dev, test or prod.
When promoting to Test existing data may be retained if there are no structural changes; you'll have to refresh the dataflows and datasets in Test to populate them.
Post ProdigyThanks for the reply, jeffshieldsdev. Is there a recommended method, then, of working with a single dataset in a central workspace rather than having each workspace contain data in a dataset? I thought parameterization might be a viable option, but it sounds like the pipeline would actually "undo" that by requiring each workspace to contain its own dataset.
Just trying to think about long-term performance and not consuming data storage unnecessarily.
Oh, and to confirm I'm understanding, when you said the following:
jeffshieldsdev wrote:Correct, data is not copied. Each Workspace only has the data contained within it--be it dataflows or datasets. The idea is to just work with assets within one environment: dev, test or prod.
you were saying "the idea is to ensure all assets are contained within a single workspace" as opposed to saying "the idea is to store data in one workspace and have other stages reference data in that workspace?"
- jeffshieldsdev
The intent is you only use the prod workspace for consumption...dev is for development and test is for testing. All downstream dataflows, datasets and reports should always be consuming from the prod workspace. Only promote to prod what has been tested.
There no additional cost associated with storing data in all three environments--depending on your data source there may be other costs associated with extraction, but using pipelines enables you to test without overwriting prod.
An option to limit data is to create a parameter is in your dataset (like "DevelopmentMode", and set to "dev", "test", or "prod"). Have your queries check this parameter and when "dev" only import a small numer of rows (I use 10), if "test" then import a medium-high amount (I use 10,000), and if "prod" or blank, don't impose a filter at all.
You can then assign deployment rules to set the parameter in each stage's Workspace in the pipeline settings.