Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
5 years ago
Solved

Adding new Python source refreshes all other Python source tables

Whenever I add a new data source via Python to PowerBi it refreshes all other tables with a Python source. Is there any way to stop this from happening?

 

here's an example of one code:

import pandas as pd

# Set location source of CSV's
data_location = 'C:/Data Sources/'

# Data source CSV names
base = f'{ data_location }BASE_DATA.csv'
latest = f'{ data_location }LATEST_DATA.csv'
customer_data = f'{ data_location }customer_data.csv'

# Load CSV's in to Dataframes
df_base = pd.read_csv(base , dtype=str, encoding='latin')
df_latest = pd.read_csv(latest , dtype=str, encoding='latin')
df_customer = pd.read_csv(customer_data , dtype=str, encoding='latin')

# Select only required columns and rename so all match
df_base = df_base[['CIDN_','CUSTOMER_NAME_']]
df_latest = df_latest[['cidn','Company Name']]
df_latest = df_latest.rename(columns={"cidn": "CIDN_", "Company Name": "CUSTOMER_NAME_"})
df_customer = df_maxim[['CIDN_','CUSTOMER_NAME_']]

# Merge and remove duplicates
TBL_CUSTOMER_DATA = df_latest.append(df_base)
TBL_CUSTOMER_DATA = TBL_CUSTOMER_DATA.append(df_customer)
TBL_CUSTOMER_DATA.drop_duplicates('CIDN_', inplace=True)

# Final output
TBL_CUSTOMER_DATA = TBL_CUSTOMER_DATA.reset_index(drop=True)

Adding this code as a data source then refreshes every other Python data source.

 

This is really frustrating as some of the other data sources are very large and take a long time (some can take hours) to refresh, which is exacerbated by the fact I need to connect to a VPN on the other side of the world to access them.

1 Reply