data ops
3 TopicsIntelliventra – Smart Event Management Application
Intelliventra lets participants easily browse all sessions and add them to their personal agenda. Attendees can view session details such as time, room, track, and speaker. The page also allows attendees to add,remove sessions. It is designed to be simple, clear, and works for any event, from small workshops to large conferences.20KViews14likes0CommentsDecrypting data on the fly
This project addresses the performance and scalability challenges of working with data encrypted by legacy systems. By modernizing the data pipeline using Microsoft Fabric, we transitioned from resource-intensive batch jobs to a streamlined, translytical flow. Data is ingested into Delta Tables, compared efficiently using delta logic, and decrypted on-the-fly via Python UDFs integrated into SQL—enabling direct use in Power BI without burdening the database. This approach improves system responsiveness, reduces storage overhead, and enhances security by supporting encryption at rest and in transit. The decryption algorithm presented below was chosen at random; any decryption algorithm could replace the one displayed. @udf(returnType=StringType()) def app_decrypt(cipher_text): if not cipher_text: return '' try: # This is not our actual encryption algorithm; for display only! import base64 from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes from cryptography.hazmat.backends import default_backend from cryptography.hazmat.primitives import padding from cryptography.fernet import Fernet fernet = Fernet(FERNET_KEY) decrypted = fernet.decrypt(cipher_text.encode()) return decrypted.decode('utf-8') return decrypted.decode('utf-8') except Exception: return "{Decrypt Error}"1.6KViews1like0CommentsAugment data on the fly
You can programmatically fetch additional data related to the data within your report. Example, you can use this report to not only track and update the status of various collaborations, but you can also fetch additional data via API, such as the latest contact information for a registered partner company. Here is the Python code for the user data function that powers this scenario: import fabric.functions as fn import logging import requests udf = fn.UserDataFunctions() @udf.connection(argName="sqlDB",alias="Translytical") @udf.function() def GetContactInfo(sqlDB: fn.FabricSqlConnection, company: str, status: str, date:str, comment: str) -> str: logging.info('Python UDF trigger function processed a request.') # Error handling for no status selected or no company input if(status=="" or len(status) < 1): raise fn.UserThrownError("The status isn't valid.", {"status:": status}) if(company=="" or len(company) < 1): raise fn.UserThrownError("The company isn't valid.", {"company:": company}) #Call External API to get contact info within registered partner company url = "https://dummy-json.mock.beeceptor.com/users" response = requests.get(url) if response.status_code == 200: Jsondata = response.json() for i in Jsondata: #Check if company contact is registered in the external system if i['company'] == company: # Combine timestamp with comment comment_value = date + " - " + comment # Get customer contact and put in format based on if updating an record or adding new record Existing_Partner_data = (i['name'], i['username'], i['email'], i['address'], i['zip'], i['state'], i['country'], i['phone'], status, comment_value, company) New_Partner_data = (i['name'], company, i['username'], i['email'], i['address'], i['zip'], i['state'], i['country'], i['phone'], status, comment_value) # Establish a connection to the SQL database connection = sqlDB.connect() cursor = connection.cursor() #Check if there is a status record for the company SQL_read_Command = "SELECT * FROM [dbo].[CompanyStatus] WHERE Company = ?" cursor.execute(SQL_read_Command, company) if cursor.fetchone(): #if there is a status record for the company, update the status and contact info SQL_update_command = "UPDATE [dbo].[CompanyStatus] SET [Name] = ?, [Username] = ?, [Email] = ?, [Address] = ?, [Zip] = ?, [State] = ?, [Country] = ?, [Phone] = ?, [Status] = ?, [Comment] = ? WHERE [Company] = ?;" cursor.execute(SQL_update_command, Existing_Partner_data) else: #if there is not a status record for the company, add new record SQL_insert_command = "INSERT INTO [dbo].[CompanyStatus](Name, Company, Username, Email, Address, Zip, State, Country, Phone, Status, Comment) VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);" cursor.execute(SQL_insert_command, New_Partner_data) # Commit the transaction connection.commit() # Close the connection cursor.close() connection.close() return "Collab with " + company + " is now " + status.lower() + "." #if reached here in the code, then the company contact is NOT registered in the external system, throw error raise fn.UserThrownError("The company is not a registered partner.", {"company:": company}) Feel free to use this code as inspiration for your own data ops + write back scenario!5.8KViews0likes0Comments