Forum Discussion

sun-sboyanapall's avatar
sun-sboyanapall
Advocate I
1 year ago
Solved

Custom Library not found

Hello, 

 

I have compiled a whl file and placed it unde rthe custom library section and published it and when i try to import it under notebook using the environment which has this library it says "ModuleNotFoundError: No module named 'sun_five9_runreport'"

when I do %pip list I can see the library under the list. can't wrap my head on why this this is happening.

here is how i constructed setup.py

from setuptools import setup, find_packages

setup(
name="sun_five9_runreport",
version="1.0.0",
packages=find_packages(),
description="A library to fetch reports from Five9 and process them into Spark DataFrames.",
author="xxx",
company="xxx",
)

please help! 



  • I have rebulished it with a different name it worked this time. 

    Here is an example library for refreshing SQL Analytics End Point

    import os
    import shutil
    import subprocess
    
    # Recreate the directory structure from scratch
    source_dir = "/mnt/data/lakehouse_sync_new"
    dist_dir = os.path.join(source_dir, "dist")
    
    if os.path.exists(source_dir):
        shutil.rmtree(source_dir)
    os.makedirs(os.path.join(source_dir, "lakehouse_sync_new"))
    
    # Create an empty __init__.py file
    with open(os.path.join(source_dir, "lakehouse_sync_new", "__init__.py"), "w") as f:
        f.write("")
    
    # Write the updated script without logging
    script_content_no_logging = """
    import json
    import time
    from sempy.fabric import FabricRestClient
    from sempy.fabric.exceptions import FabricHTTPException, WorkspaceNotFoundException
    
    class LakehouseSync:
        def __init__(self, tenant_id, workspace_id, lakehouse_id):
            self.tenant_id = tenant_id
            self.workspace_id = workspace_id
            self.lakehouse_id = lakehouse_id
            self.client = FabricRestClient()
            self.max_retries = 60  # Timeout after 60 seconds
            self.sleep_duration = 1  # Sleep 1 second between retries
    
        def _make_get_request(self, uri):
            \"\"\"Helper to make a GET request and return JSON data.\"\"\"
            response = self.client.get(uri)
            response.raise_for_status()
            return response.json()
    
        def _make_post_request(self, uri, payload):
            \"\"\"Helper to make a POST request and return JSON data.\"\"\"
            response = self.client.post(uri, json=payload)
            response.raise_for_status()
            return response.json()
    
        def get_sql_endpoint(self):
            \"\"\"Retrieve the SQL endpoint ID for the lakehouse.\"\"\"
            uri = f"/v1/workspaces/{self.workspace_id}/lakehouses/{self.lakehouse_id}"
            data = self._make_get_request(uri)
            return data['properties']['sqlEndpointProperties']['id']
    
        def sync_metadata(self, sql_endpoint_id):
            \"\"\"Initiate metadata synchronization and monitor progress.\"\"\"
            sync_uri = f"/v1.0/myorg/lhdatamarts/{sql_endpoint_id}"
            payload = {"commands": [{"$type": "MetadataRefreshExternalCommand"}]}
    
            # Start the sync
            response = self._make_post_request(sync_uri, payload)
            batch_id = response["batchId"]
            progress_state = response["progressState"]
    
            # Monitor sync progress
            status_uri = f"/v1.0/myorg/lhdatamarts/{sql_endpoint_id}/batches/{batch_id}"
            retries = 0
    
            while progress_state == 'inProgress' and retries < self.max_retries:
                time.sleep(self.sleep_duration)
                retries += 1
                status_response = self._make_get_request(status_uri)
                progress_state = status_response["progressState"]
    
            if retries >= self.max_retries:
                raise TimeoutError("Sync process timed out.")
    
            if progress_state == 'failure':
                raise Exception(f"Sync failed. Details: {status_response}")
    
        def run(self):
            \"\"\"Main method to orchestrate the synchronization process.\"\"\"
            sql_endpoint_id = self.get_sql_endpoint()
            self.sync_metadata(sql_endpoint_id)
    """
    with open(os.path.join(source_dir, "lakehouse_sync_new", "lakehouse_sync_new.py"), "w") as f:
        f.write(script_content_no_logging)
    
    # Create a minimal setup.py file
    setup_content_minimal = """
    from setuptools import setup, find_packages
    
    setup(
        name="lakehouse_sync_new",
        version="1.0.0",
        packages=find_packages(),
        author="Sai Boyanapalli",
    )
    """
    with open(os.path.join(source_dir, "setup.py"), "w") as f:
        f.write(setup_content_minimal)
    
    # Rebuild the .whl file
    subprocess.run(["python3", "setup.py", "bdist_wheel"], cwd=source_dir, capture_output=True, text=True)
    

4 Replies

  • I have rebulished it with a different name it worked this time. 

    Here is an example library for refreshing SQL Analytics End Point

    import os
    import shutil
    import subprocess
    
    # Recreate the directory structure from scratch
    source_dir = "/mnt/data/lakehouse_sync_new"
    dist_dir = os.path.join(source_dir, "dist")
    
    if os.path.exists(source_dir):
        shutil.rmtree(source_dir)
    os.makedirs(os.path.join(source_dir, "lakehouse_sync_new"))
    
    # Create an empty __init__.py file
    with open(os.path.join(source_dir, "lakehouse_sync_new", "__init__.py"), "w") as f:
        f.write("")
    
    # Write the updated script without logging
    script_content_no_logging = """
    import json
    import time
    from sempy.fabric import FabricRestClient
    from sempy.fabric.exceptions import FabricHTTPException, WorkspaceNotFoundException
    
    class LakehouseSync:
        def __init__(self, tenant_id, workspace_id, lakehouse_id):
            self.tenant_id = tenant_id
            self.workspace_id = workspace_id
            self.lakehouse_id = lakehouse_id
            self.client = FabricRestClient()
            self.max_retries = 60  # Timeout after 60 seconds
            self.sleep_duration = 1  # Sleep 1 second between retries
    
        def _make_get_request(self, uri):
            \"\"\"Helper to make a GET request and return JSON data.\"\"\"
            response = self.client.get(uri)
            response.raise_for_status()
            return response.json()
    
        def _make_post_request(self, uri, payload):
            \"\"\"Helper to make a POST request and return JSON data.\"\"\"
            response = self.client.post(uri, json=payload)
            response.raise_for_status()
            return response.json()
    
        def get_sql_endpoint(self):
            \"\"\"Retrieve the SQL endpoint ID for the lakehouse.\"\"\"
            uri = f"/v1/workspaces/{self.workspace_id}/lakehouses/{self.lakehouse_id}"
            data = self._make_get_request(uri)
            return data['properties']['sqlEndpointProperties']['id']
    
        def sync_metadata(self, sql_endpoint_id):
            \"\"\"Initiate metadata synchronization and monitor progress.\"\"\"
            sync_uri = f"/v1.0/myorg/lhdatamarts/{sql_endpoint_id}"
            payload = {"commands": [{"$type": "MetadataRefreshExternalCommand"}]}
    
            # Start the sync
            response = self._make_post_request(sync_uri, payload)
            batch_id = response["batchId"]
            progress_state = response["progressState"]
    
            # Monitor sync progress
            status_uri = f"/v1.0/myorg/lhdatamarts/{sql_endpoint_id}/batches/{batch_id}"
            retries = 0
    
            while progress_state == 'inProgress' and retries < self.max_retries:
                time.sleep(self.sleep_duration)
                retries += 1
                status_response = self._make_get_request(status_uri)
                progress_state = status_response["progressState"]
    
            if retries >= self.max_retries:
                raise TimeoutError("Sync process timed out.")
    
            if progress_state == 'failure':
                raise Exception(f"Sync failed. Details: {status_response}")
    
        def run(self):
            \"\"\"Main method to orchestrate the synchronization process.\"\"\"
            sql_endpoint_id = self.get_sql_endpoint()
            self.sync_metadata(sql_endpoint_id)
    """
    with open(os.path.join(source_dir, "lakehouse_sync_new", "lakehouse_sync_new.py"), "w") as f:
        f.write(script_content_no_logging)
    
    # Create a minimal setup.py file
    setup_content_minimal = """
    from setuptools import setup, find_packages
    
    setup(
        name="lakehouse_sync_new",
        version="1.0.0",
        packages=find_packages(),
        author="Sai Boyanapalli",
    )
    """
    with open(os.path.join(source_dir, "setup.py"), "w") as f:
        f.write(setup_content_minimal)
    
    # Rebuild the .whl file
    subprocess.run(["python3", "setup.py", "bdist_wheel"], cwd=source_dir, capture_output=True, text=True)
    
  • Hi sun-sboyanapall,

     

    Can you try republishing once? I have previously used a custom wheel in spark and faced no issues. 

     

    Also, can you try installing in a virtual environment in your local machine to check if the module gets correctly installed and imports without issue?

  • v-hashadapu's avatar
    v-hashadapu
    Community Support

    Hi sun-sboyanapall , thank you for reaching out to the Microsoft Fabric Community Forum.

    Please try the following:

    1. Double-check the name of the package and the import statement in your notebook. Make sure they match exactly, as Python is case-sensitive.
    2. Sometimes the library might be installed in a different environment or path. Check if the notebook is using the same Python environment where the package is installed. You can verify this by running:
      import sys

    print(sys.executable)

    1. Sometimes, reinstalling the package can resolve issues. Try uninstalling and reinstalling the package.
    2. Ensure the module is being imported correctly in your notebook. Your import statement should look something like this:
      from sun_five9_runreport import YourModuleName
    3. Make sure your package structure is correct and includes an __init__.py file in each directory. Your file structure should look something like this:

    sun_five9_runreport/

        __init__.py

        ...

     

    If this helps, please consider marking it 'Accept as Solution' so others with similar queries may find it more easily. If not, please share the details.
    Thank you.