mssparkutils file operations

Hi Anonymous and Anonymous 

I found the issue I was having.

In Fabric the top level folders of a workspace are condired "managed one lake folders" .

In the structure of a workspace these folders "Files" and "Tables" /MyLakehouse.lakehouse/Files and /MyLakehouse.lakehouse/Tables

At this level one cannot create or delete folders using the msspartutils only from the "second" level inside those folders.

For example: the code below will fail

import pandas as pd

data = {'name':['John']}
df = pd.DataFrame(data)

path = 'ManagedLevel'

try:
    if not mssparkutils.fs.exists(path):
        print(f'path does not exit, creating path: {path}')
        mssparkutils.fs.mkdirs(path)
    else:
        print(f'path: {path} already exists')

    print('trying to save pandas df in a just created folder')
    df.to_csv(f'/lakehouse/default/{path}/able_to_save.csv')
    print('save csv successfully')
except Exception as e:
    print(e)

Because I am trying to create a filder at the Managed Level or same level as Files and Tables.

The error is a bad request code 400 (not authorized code). 

 Operation failed: "Bad Request", 400, HEAD,

 However if I try and create a folder below the managed level then the code works fine to create the folder but fails to save the pandas dataframe.

import pandas as pd

data = {'name':['John']}
df = pd.DataFrame(data)

path = 'Files/second_level'

try:
    if not mssparkutils.fs.exists(path):
        print(f'path does not exit, creating path: {path}')
        mssparkutils.fs.mkdirs(path)
    else:
        print(f'path: {path} already exists')

    print('trying to save pandas df in a just created folder')
    df.to_csv(f'/lakehouse/default/{path}/able_to_save.csv')
    print('save csv successfully')
except Exception as e:
    print(e)

path does not exit, creating path: Files/second_level
trying to save pandas df in a just created folder
[Errno 2] No such file or directory: '/lakehouse/default/Files/second_level/able_to_save.csv'

I think because the condition that You can perform CRUD (Create, Read, Update and Delete) operations on any folder or file created within these managed folders, and perform read-only operations on workspace and item folders.

Finally at a folder within the second_level it is fine and it all works.

import pandas as pd

data = {'name':['John']}
df = pd.DataFrame(data)

path = 'Files/second_level/third_level'

try:
    if not mssparkutils.fs.exists(path):
        print(f'path does not exit, creating path: {path}')
        mssparkutils.fs.mkdirs(path)
    else:
        print(f'path: {path} already exists')

    print('trying to save pandas df in a just created folder')
    df.to_csv(f'/lakehouse/default/{path}/able_to_save.csv')
    print('save csv successfully')
except Exception as e:
    print(e)

path does not exit, creating path: Files/second_level/third_level
trying to save pandas df in a just created folder
save csv successfully