topic Re: Identify a shortcut in pyspark notebook in Data Engineering

Identify a shortcut in pyspark notebook

DennesTorres — Sat, 14 Oct 2023 09:50:30 GMT

Hi,

Shortcuts are very powerful and they are a great feature to enable a data mesh architecture.

However, thinking on corporation level it may be hard to manage who links with who. How could we identify a shortcut in a notebook?

I already tried:

metadata

os.listdir

mssparkutils.fs.ls

show tblproperties

spark.catalog.listtables

Nothing worked, they follow the shortcut and give no cluet that's not a local table.

Fabric has the information, of course, it changes the icon for shortcuts and shows in the lineage. But how could we access this information in a notebook to improve our shortcut management?

Kind Regards,

Denne

Re: Identify a shortcut in pyspark notebook

Anonymous — Sat, 14 Oct 2023 13:41:49 GMT

Hi @DennesTorres - Thanks for using Fabric Community,

I have reached the internal team for help on this. I will update you once I hear from them.

Appreciate your patience.

Re: Identify a shortcut in pyspark notebook

Anonymous — Mon, 23 Oct 2023 11:32:39 GMT

Hi @DennesTorres ,

Apologies for the delay in reply from our side.

The fabric release plan states that this feature will be available during Q4 2023 (create and manage shortcuts via REST API): Link

Appreciate your patience

Re: Identify a shortcut in pyspark notebook

Anonymous — Wed, 25 Oct 2023 17:07:32 GMT

Hi @DennesTorres ,

We haven’t heard from you on the last response and was just checking back to see if your query got answered? Otherwise, will respond back with the more details and we will try to help.

Re: Identify a shortcut in pyspark notebook

Anonymous — Mon, 30 Oct 2023 06:12:53 GMT

Re: Identify a shortcut in pyspark notebook

Syliano — Fri, 24 Jan 2025 10:23:18 GMT

I resolve this by using sempy :

import sempy.fabric as fabric import json import requests import fnmatch import base64 -- Get metadata from pyspark.sql import SparkSession from pyspark.sql.types import StructType, StructField, StringType # Récupérer les données depuis Fabric ws = fabric.get_workspace_id() items = fabric.list_items(workspace=ws) # Construire une liste de dictionnaires à partir des données items_data = [dict(row) for row in items.to_dict("records")] # print(items_data) # Définir le schéma (adapté selon vos colonnes) schema = StructType([ StructField("Id", StringType(), True), StructField("Display Name", StringType(), True), StructField("Description", StringType(), True), StructField("Type", StringType(), True), StructField("Workspace Id", StringType(), True), ]) # Créer le DataFrame Spark items_df = spark.createDataFrame(items_data, schema) # Créer une vue SparkSQL items_df.createOrReplaceTempView("items_view") -- get current lakehouse ws = fabric.get_workspace_id() items = fabric.list_items(workspace= ws) item = items[(items['Display Name'] == "Bronze")& (items['Type']=="Lakehouse")].iloc[0]["Id"] -- call api client = fabric.FabricRestClient() r = client.request(method="get", path_or_url=f"/v1/workspaces/{ws}/items/{item}/shortcuts") -- update or créate shortcup to an other workspace

This code di you help you ?