Python script does not give data. However in Visual studio code, it does

[sp_mike]

I have a Python script to get data from a website. This works perfect in Visual Studio Code. Now I run the same in Power BI. It does not give me an error bust I do not get data at all.

This is my piece of code to create the dataset:

table = driver.find_element(By.CLASS_NAME, 'ContentTable')
    rows = table.find_elements_by_tag_name('tr')

    row_num = 0    

    l = []
    for row in rows:
        if row_num > 0:
            td = row.find_elements(By.TAG_NAME, "td")
            row2 = [row.text for row in td]
            l.append(row2)
        row_num = row_num + 1
    df = pd.DataFrame(l, columns=["A", "B", "C", "D", "E"],dtype=float)  
    print(df)

In Visual studio, I get 21 rows in "df" PowerBI does not create the dataset.

What can be the difference?

[sp_mike · Accepted solution]

Got it 

My script uses functions, I had to pass df to my initial function and call this one like this:

output = main()

[vanessafvg]

well done!

[sp_mike]

When I do a "Raise error(df)" In my script is returns my data. But somehow it is not transferred to a powerBI table. My Python Map remains empty.

[vanessafvg]

is that your whole script?  stupid question cos i am not hugely proficient in all the ins out out on python, but do you import all the libraries you require?

[sp_mike]

This is my whole script

import time
import pandas as pd

import locale

from selenium import webdriver
from selenium.common.exceptions import WebDriverException
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from datetime import datetime
from decimal import *

locale.setlocale(locale.LC_ALL, 'nl_NL.UTF-8')

from selenium.webdriver.edge.options import Options
options = Options()
options.headless = True

MAX_WAIT = 10

def deal_with_cookies(driver):
    element = WebDriverWait(driver, MAX_WAIT).until(EC.presence_of_element_located((By.XPATH, '//*[@id="gdpr-consent-notice"]')))
    driver.switch_to.frame(element)
    button = WebDriverWait(driver, MAX_WAIT).until(EC.element_to_be_clickable((By.XPATH, '//button[@id="save"]')))
    button.send_keys(Keys.ENTER)
    driver.switch_to.default_content()

def click_volgende(driver):
    button = driver.find_element(By.ID, 'ctl00_ctl00_ctl00_ContentPlaceHolder1_LeftContent_Content_AdviesList_cmdNext')
    button.click()

def get_table_data(driver):
    start_time = time.time()
    try:
        table = driver.find_element(By.CLASS_NAME, 'ContentTable')
        rows = table.find_elements_by_tag_name('tr')

        row_num = 0
        l = []
        for row in rows:
            if row_num > 0:
                td = row.find_elements(By.TAG_NAME, "td")
                row2 = [row.text for row in td]
                l.append(row2)
            row_num = row_num + 1
        df = pd.DataFrame(l, columns=["A", "B", "C", "D", "E"],dtype=float)  
        print(df)
     
    except (AssertionError, WebDriverException) as e:
        if time.time() - start_time > MAX_WAIT:
            print("Unable to find element, ending script")
            raise e
        time.sleep(0.5)


def main():  
    driver = webdriver.Edge(options=options)
    driver.get('https://www.guruwatch.nl/aandeel/330011556/Boeing-Company/Adviezen.aspx')
    deal_with_cookies(driver)
    for page in range(1,2):
        print(f"Scraping page {page}.")
        get_table_data(driver)
    #    click_volgende(driver)
        time.sleep(5)
    driver.quit()

if __name__ == "__main__":
    main()

I just copy and paste it from VSC, where it is working perfectly

[vanessafvg]

how did you set up power bi to work with python?

are you getting a table retuning but not data, can you provide some screenshots  or an explanation of how you have set this up in power bi

[sp_mike]

Python itself is working.

This demo script from Microsoft works fine:

import pandas as pd
data = [['Alex',10],['Bob',12],['Clarke',13]]
df = pd.DataFrame(data,columns=['Name','Age'],dtype=float)
print (df)

It retuns a Pyhon map with the df table in it.

My script returns the python map but no table inside.

What I do is just copy and paste my VSC script in the power BI editor Get data > Python