Need more hiarchy for delta tables.

[FredrikK]

The pyspark way of creating a delta table is with the package DeltaTable (Welcome to Delta Lake’s Python documentation page — delta-spark 2.1.1 documentation) and for example the class createIfNotExists. There you can specify the table path and table name along with the schema, partitions and other table properties. There is an important difference between the name and the path of the table. The name is the identifier to access the table using SQL, while the path is where the data resides, and thus can be specified with pyspark's spark.read.load. However, Fabric is not intended for specifying paths when creating tables with the DeltaTable package. If you create a table with a name "table1" and path "Files/test/table1" then the table will show up as table1 under Tables and its data will reside in the test subfolder under Files. This may look swell in the lakehouse, but when trying to build a semantic model on that lakehouse, or when trying to use the table as a destination in a copy data activity in a data pipeline, table1 will not show up. To give you more of an understanding of how Fabric looks for tables, let's see what happens if you write a table named table1 to the path "Tables/table2". Then 2 new tables are added to the Tables folder, but only table2 will show up when creating the semantic model or connecting from data pipeline. Okay, so why won't we just write the tables to the table folder as "Tables/test/table1" then? We cant name the table test/table1 due to a syntax error from the "/", and when setting the path as "Tables/test/table1" there are mulitple consequences. First of all, let's start with the good thing. A delta table is created and it is possible to use in a semantic model, just it will be under the schema test rather than the standard dbo. The table will on the other hand not be recognised in the lakehouse. It will show up as a folder Unidentified/test/table1 which is not accesible by SQL. The test folder will show up in the data pipeline activity as a destination, but not the subfolder table1. Writing to the folder test will tjen just write to the parent folder of table1. Moreover, when using createIfNotExists a second time to retrieve the delta table object, the operation will fail because no table name was specified with the following error: The schema `delta` cannot be found. You could of course assign a table name, but then you end up with the extra table in Tables which Fabric cant find except for when inside the lakehouse data explorer. So what do we want here? Either for delta tables in the Tables folder to be able to use subfolders and that those tables are recognised by the lakehouse, as data pipeline destinations and when creating a semantic model, or that likewise external tables in the Files folder with a table name in the Tables folder are also recognised and discoverable in the same places.