Forum Discussion
Kudo CollectorHow to save a "List" column for later use?
Perhaps what you're looking for is to save the full response from the API as a binary, save it to the lakehouse as a binary and then consume the binaries through a different dataflow.
If this is in line with your expectation, I'd suggest raising a new idea using the link below:You could also post an idea for the lakehouse team to increase the maximum string size. As mentioned before, this is not a limitation of the Dataflow engine, but rather a limitation of the destination which cannot handle more than X number of characters for a string and how many characters your string has.
All of the previously shared solutions are the ways that you can handle things in Dataflows Gen2 today:- Expand the complex values as those data types are not supported in the destination that you're mentioning
- Transform the complex values to a text string and save it as a string.
- Furthermore, you could encode the string so it shortens things and save it that way. It could also make the number of character expand tremendously, so it might not be a one-size-fits-all solution
So this a limitation on the Lakehouse/Warehouse end.
I've opened an idea so they increase the Text field size.
https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=fd452033-4bb2-ee11-92bd-000d3a0d7518
Please vote
Kudo CollectorThanks, but I do not want to expand the columns at this stage.
Just to get the whole JSON Text from the collapsed List and save it as such - Text.
I just want to deal withh all the expands lated. This is to separate the "data gathering" stage from "data transformation".
This separation is critical:
1. At the "Gathering Satge" I just need a simple API call that gets me everything as a "Raw" JSON List column. I don't want any transformation (apart of expracting the whole JSON as a text). I want as little transformations because the API is very very very sensitive to Rate Limit.
So I just want to "dump" everything AS-IS to Lakehouse.
2. Later, I'll connect to this Raw data and transform it as i please.
I think my scenario is very common, actually.
I tried to do it with Fabric Pipeline API action, but it is sooooo far away to be as friendly as Dataflow and also it doesn't suppot Gateways (which is required for my API).
So either the Dataflow should allow saving JSON files to Lakehouse, and not just Tables.
Or tables in Lakehouse should be able to store Text columns without characters' limit....
Community AdminPerhaps what you're looking for is to save the full response from the API as a binary, save it to the lakehouse as a binary and then consume the binaries through a different dataflow.
If this is in line with your expectation, I'd suggest raising a new idea using the link below:
You could also post an idea for the lakehouse team to increase the maximum string size. As mentioned before, this is not a limitation of the Dataflow engine, but rather a limitation of the destination which cannot handle more than X number of characters for a string and how many characters your string has.
All of the previously shared solutions are the ways that you can handle things in Dataflows Gen2 today:
- Expand the complex values as those data types are not supported in the destination that you're mentioning
- Transform the complex values to a text string and save it as a string.
- Furthermore, you could encode the string so it shortens things and save it that way. It could also make the number of character expand tremendously, so it might not be a one-size-fits-all solution
- iBusinessBI
So this a limitation on the Lakehouse/Warehouse end.
I've opened an idea so they increase the Text field size.
https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=fd452033-4bb2-ee11-92bd-000d3a0d7518
Please vote