topic Custom Library not found in Data Engineering

Custom Library not found

sun-sboyanapall — Fri, 17 Jan 2025 18:04:26 GMT

Hello,

I have compiled a whl file and placed it unde rthe custom library section and published it and when i try to import it under notebook using the environment which has this library it says "ModuleNotFoundError: No module named 'sun_five9_runreport'"

when I do %pip list I can see the library under the list. can't wrap my head on why this this is happening.

here is how i constructed setup.py

from setuptools import setup, find_packages

setup(
name="sun_five9_runreport",
version="1.0.0",
packages=find_packages(),
description="A library to fetch reports from Five9 and process them into Spark DataFrames.",
author="xxx",
company="xxx",
)

please help!

Re: Custom Library not found

govindarajan_d — Sat, 18 Jan 2025 17:00:44 GMT

Hi @sun-sboyanapall,

Can you try republishing once? I have previously used a custom wheel in spark and faced no issues.

Also, can you try installing in a virtual environment in your local machine to check if the module gets correctly installed and imports without issue?

Re: Custom Library not found

v-hashadapu — Tue, 21 Jan 2025 07:35:33 GMT

Hi @sun-sboyanapall , thank you for reaching out to the Microsoft Fabric Community Forum.

Please try the following:

Double-check the name of the package and the import statement in your notebook. Make sure they match exactly, as Python is case-sensitive.
Sometimes the library might be installed in a different environment or path. Check if the notebook is using the same Python environment where the package is installed. You can verify this by running:
import sys

print(sys.executable)

Sometimes, reinstalling the package can resolve issues. Try uninstalling and reinstalling the package.
Ensure the module is being imported correctly in your notebook. Your import statement should look something like this:
from sun_five9_runreport import YourModuleName
Make sure your package structure is correct and includes an __init__.py file in each directory. Your file structure should look something like this:

sun_five9_runreport/

__init__.py

...

If this helps, please consider marking it 'Accept as Solution' so others with similar queries may find it more easily. If not, please share the details.
Thank you.

Re: Custom Library not found

sun-sboyanapall — Wed, 22 Jan 2025 19:29:01 GMT

I have rebulished it with a different name it worked this time.

Here is an example library for refreshing SQL Analytics End Point

import os import shutil import subprocess # Recreate the directory structure from scratch source_dir = "/mnt/data/lakehouse_sync_new" dist_dir = os.path.join(source_dir, "dist") if os.path.exists(source_dir): shutil.rmtree(source_dir) os.makedirs(os.path.join(source_dir, "lakehouse_sync_new")) # Create an empty __init__.py file with open(os.path.join(source_dir, "lakehouse_sync_new", "__init__.py"), "w") as f: f.write("") # Write the updated script without logging script_content_no_logging = """ import json import time from sempy.fabric import FabricRestClient from sempy.fabric.exceptions import FabricHTTPException, WorkspaceNotFoundException class LakehouseSync: def __init__(self, tenant_id, workspace_id, lakehouse_id): self.tenant_id = tenant_id self.workspace_id = workspace_id self.lakehouse_id = lakehouse_id self.client = FabricRestClient() self.max_retries = 60 # Timeout after 60 seconds self.sleep_duration = 1 # Sleep 1 second between retries def _make_get_request(self, uri): \"\"\"Helper to make a GET request and return JSON data.\"\"\" response = self.client.get(uri) response.raise_for_status() return response.json() def _make_post_request(self, uri, payload): \"\"\"Helper to make a POST request and return JSON data.\"\"\" response = self.client.post(uri, json=payload) response.raise_for_status() return response.json() def get_sql_endpoint(self): \"\"\"Retrieve the SQL endpoint ID for the lakehouse.\"\"\" uri = f"/v1/workspaces/{self.workspace_id}/lakehouses/{self.lakehouse_id}" data = self._make_get_request(uri) return data['properties']['sqlEndpointProperties']['id'] def sync_metadata(self, sql_endpoint_id): \"\"\"Initiate metadata synchronization and monitor progress.\"\"\" sync_uri = f"/v1.0/myorg/lhdatamarts/{sql_endpoint_id}" payload = {"commands": [{"$type": "MetadataRefreshExternalCommand"}]} # Start the sync response = self._make_post_request(sync_uri, payload) batch_id = response["batchId"] progress_state = response["progressState"] # Monitor sync progress status_uri = f"/v1.0/myorg/lhdatamarts/{sql_endpoint_id}/batches/{batch_id}" retries = 0 while progress_state == 'inProgress' and retries < self.max_retries: time.sleep(self.sleep_duration) retries += 1 status_response = self._make_get_request(status_uri) progress_state = status_response["progressState"] if retries >= self.max_retries: raise TimeoutError("Sync process timed out.") if progress_state == 'failure': raise Exception(f"Sync failed. Details: {status_response}") def run(self): \"\"\"Main method to orchestrate the synchronization process.\"\"\" sql_endpoint_id = self.get_sql_endpoint() self.sync_metadata(sql_endpoint_id) """ with open(os.path.join(source_dir, "lakehouse_sync_new", "lakehouse_sync_new.py"), "w") as f: f.write(script_content_no_logging) # Create a minimal setup.py file setup_content_minimal = """ from setuptools import setup, find_packages setup( name="lakehouse_sync_new", version="1.0.0", packages=find_packages(), author="Sai Boyanapalli", ) """ with open(os.path.join(source_dir, "setup.py"), "w") as f: f.write(setup_content_minimal) # Rebuild the .whl file subprocess.run(["python3", "setup.py", "bdist_wheel"], cwd=source_dir, capture_output=True, text=True)

Re: Custom Library not found

govindarajan_d — Wed, 22 Jan 2025 21:06:53 GMT

Wow. This is useful!