This project addresses the performance and scalability challenges of working with data encrypted by legacy systems. By modernizing the data pipeline using Microsoft Fabric, we transitioned from resource-intensive batch jobs to a streamlined, translytical flow. Data is ingested into Delta Tables, compared efficiently using delta logic, and decrypted on-the-fly via Python UDFs integrated into SQL—enabling direct use in Power BI without burdening the database. This approach improves system responsiveness, reduces storage overhead, and enhances security by supporting encryption at rest and in transit.
The decryption algorithm presented below was chosen at random; any decryption algorithm could replace the one displayed.
@udf(returnType=StringType())
def app_decrypt(cipher_text):
if not cipher_text:
return ''
try:
# This is not our actual encryption algorithm; for display only!
import base64
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import padding
from cryptography.fernet import Fernet
fernet = Fernet(FERNET_KEY)
decrypted = fernet.decrypt(cipher_text.encode())
return decrypted.decode('utf-8')
return decrypted.decode('utf-8')
except Exception:
return "{Decrypt Error}"
