Can anything in Microsoft Fabric do a Geospatial join

[mlane]

Power BI doesn't have an obvious solution that allows to you to do a geospatial join. 

So my question is does anything in Microsoft Fabric do a Geospatial join at this time?  So I can solve the migration plan problem to Power BI (aka do what is circled in red)

[miguel]

We don't have anything out of the box to handle polygon files, so while it might be technically possible to make something happen with Dataflow Gen2, it would require quite a code-first approach. I'm moving this to the Synapse forum to see if perhaps notebooks would be a better option.

[mlane]

Also by notebooks do you mean this?:  https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-development-using-noteb... 
Are they automated in any fashion? 

[v-nikhilan-msft · Accepted solution]

Hi @mlane 
The notebooks here refer to https://learn.microsoft.com/en-us/fabric/data-engineering/how-to-use-notebook

You can automate the execution of notebooks within pipelines. 

• Fabric Notebooks with Data Factory: Microsoft Data Factory allows you to orchestrate data-driven workflows. You can include a special "Notebook activity" within your pipeline that triggers the execution of a specific Fabric notebook https://learn.microsoft.com/en-us/azure/data-factory/transform-data-using-databricks-notebook.

  This way, your notebook becomes a step within a larger automated data processing workflow.
  
  You can also schedule the run of your notebook in this way:
  
  
  

  

  Hope this helps. Please let me know if you have any further questions.

[v-nikhilan-msft]

Hi @mlane 
We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks

[mlane]

@v-nikhilan-msft thanks. I marked your answer as the solution, but I did have a follow up question.
Right now I am trying to I figure out how much data-factory I need, so I can know order for this request to move up the chain of command.

My questions are:
1) What kind of the 3 different runtimes, which would be best for my organization given the information about our datasource?

https://learn.microsoft.com/en-us/azure/data-factory/choose-the-right-integration-runtime-configurat... 
2) If my organization already has 64 CU per year Gb of Fabric,  do we need to purchase additional datafactory? I just wanted to double check. 

Details:

• My organizaton primarily Azure SQL Servers and Azure PosgresSQL servers.  We have flat files in a Business OneDrive (sharepoint).   As a backup, we have an old EMC shared drives that keeps a updated copy of the flat files. 
• My organization has from my understanding either Fabric or Power BI Gateway.   I am already put a request in to find out if there are at least 8 to 16 FCU and what the status of the components are like the OneLake. 
• The number of rows in the Azure SQL datasource estimated to be 800,000 datapoints (by 2030) and the number of polygons is 23 in the map file
• The total number of activites is 4 (load data, look up changes in parameters, transform data, write data to sql).  We expect the process to take a maximum of 30 minutes (or else something will time out on the network probably) and its per day.

[v-nikhilan-msft]

Hi @mlane 
1) In fabric, there is no integration runtime as ADF. Instead, there are connections that can be on cloud or on gateways. We support connections to be on cloud and on Prem Gateway to be used in pipelines, VNET Gateway based connections are in progress.
Integrating On-Premises Data into Microsoft Fabric Using Data Pipelines in Data Factory | Microsoft ...

2) No you need not purchase anything else additionally.

Hope this helps. Please let me know if you have any further questions.

[miguel]

I'll let the team from this forum try to answer the question around the automation as they might be more familiar with the product that I am, but here's a quick article on how to leverage Notebooks inside of Fabric:

How to use notebooks - Microsoft Fabric | Microsoft Learn

The product that I'm most familiar with is Dataflow Gen2 and Data Factory in Microsoft Fabric. Feel free to post any questions about those in the appropriate forum if you ever have any questions around any those and I'll try my best to answer them.

Best!

[mlane]

So Geosptial automation it has been a long standing request in the power bi community.  Infact,  @miguel  ArcGIS themselves in their Arcgis Online plaform don't automate, but instead provide python code for automation:

• https://community.esri.com/t5/arcgis-online-questions/conduct-a-spatial-join-with-data-amp-esri-map-... (no reply)
• https://automating-gis-processes.github.io/CSC18/lessons/L4/spatial-join.html 
• https://community.esri.com/t5/arcgis-online-questions/power-automate-to-create-a-record-in-agol/m-p/... (uses power automate to automate the desktop I think)
• https://community.esri.com/t5/arcgis-online-questions/automated-upload-to-arcgis-online/m-p/134556 (automate the connection an push to Arggis)

The links above made us hopeful that someone out there has automated using the entire Fabric platform (Microsoft Fabric, Synapse, Dataflow, Power Automate, Power BI).... this core issue of geospatial automation.

So please if anyone has figured out how to automate mapping and geospatial joins using the Azure platform please let me know.

References to Power BI community:

• https://community.fabric.microsoft.com/t5/Desktop/spatial-join/m-p/422942 
• https://community.fabric.microsoft.com/t5/Desktop/Spatial-analysis-in-Power-BI-using-shapefile/td-p/...
• https://community.fabric.microsoft.com/t5/Desktop/Combining-Shape-Map-amp-Geospatial-Coordinates-in-... 
• https://community.fabric.microsoft.com/t5/Desktop/USING-OUR-OWN-SHAPEFILE-MAP-CONNECTING-WITH-A-REST...
• https://community.fabric.microsoft.com/t5/Desktop/Spatial-Data-from-SQL-Table-to-Power-BI-Shape-Cust... 
• Evidence PowerBI does not support posgres: https://community.fabric.microsoft.com/t5/Desktop/postgres-spatial-datatypes/m-p/163823  
• Remote Access Define: https://community.fabric.microsoft.com/t5/Desktop/Postgresql-Remote-Access-Define-connexion/m-p/2509... 

In fact people suggest Argis 365 (and we have a license as we produce maps the world use).  However, argis 365 is not automated as you can see from there documentation:

• https://doc.arcgis.com/en/microsoft-365/latest/power-bi/join-layers.htm