How to run a pyspark code directly from a Github Repo?

Hi lchinelli,
Thank you for the follow-up.

I have practically reproduced your scenario using modular, object-oriented PySpark code structured across multiple folders like models/ and utils/.

where modular, object-oriented PySpark code is executed across multiple folders (like models/ and utils/) using a main driver script.

What I Did:

• Created a GitHub repository with the following folder structure:

/main.py

/models/

    └── cleaner.py

    └── validator.py

/utils/

    └── formatter.py

• Each helper module (cleaner.py, validator.py, formatter.py) contains a class or function for a part of the logic (e.g., data cleaning, validation, formatting).
• In the Notebook, I used requests.get().text to dynamically fetch each .py file from GitHub raw URLs, then ran each with exec() to load the functions/classes into memory.
• Then, I fetched and ran main.py, which uses the imported modules to:

• Create a sample Spark DataFrame

• Apply cleaning and validation
• Format and return the final result
• The final output was printed using df.show() inside main.py.
  
  

Attached the screenshot below showing the successful execution and expected output:

If you have any further questions, please don't hesitate to contact us through the community. We are happy to assist you.

Best Regards,
Ganesh singamshetty.