Forum Discussion
Dynamically combine SQL tables from multiple servers and databases
- 8 months ago
Putting this here for future reference...
I got some really cool code from GPT 4.1 which read my list of servers and databases from a SQL table, looped through them to combine all the tables. Also got a version which would let me use custom SQL to pull from each one instead of just a table. It all worked great in dataflow design view, but the dataflow would not save. It gave me the following error: "One or more tables references a dynamic data source." On researching this I realized there's probably no way around it, or at least no easy way, because... every time you add a new connection to a dataflow you also have to add the connection credentials (in my case a gateway connection). Even if the code would dynamically connect to a new server/database, the refresh would fail for lack of having the credentials.
For posterity's sake I am going to post the two versions of the code from GPT. (I don't have the actual working versions any more because I saved them in a comment block before the "let" statement of my old code and when I went back to retrieve it I discovered the DF dropped all the code. Argh!)
Here's the GPT code for dynamically aggregating a table across servers. I scrubbed the code of real server, database, table column names so hopefully I didn't jack it up too bad 😂
let DBConnectionQuery = " select [ServerName], [Database Name] from [ServersAndDatbases] ", DBConnections = Sql.Database( "<SQL server name>", "<SQL database name>", [Query = DBConnectionQuery] ), CustomSQL = " Select * from tblYourTable Where ... ); ", ConnectionRows = Table.ToRecords(DBConnections), GetTable = (r as record) => let server = r[ServerName], database = r[DatabaseName], result = Sql.Database(server, database, [Query = CustomSQL]) in result, YourTables = List.Transform(ConnectionRows, GetTable), Combined = Table.Combine(YourTables) in CombinedHere's the version for aggregating a custom SQL statement across servers:
let // 1. Get the list of server/database pairs dynamically DBConnectionQuery = " select [ServerName], [Database Name] from [ServersAndDatbases] ", Connections = Sql.Database( "<SQL server name>", "<SQL database name>", [Query = DBConnectionQuery] ), // 2. Set the table name and schema TableName = "tblYourTable", SchemaName = "dbo", // 3. Convert to records for easier iteration ConnectionRows = Table.ToRecords(Connections), // 4. Function to get the required table for one connection GetTable = (r as record) => let server = r[Server Name], database = r[Database Name], allTables = Sql.Database(server, database), tableRow = Table.SelectRows( allTables, each [Schema] = SchemaName and [Item] = TableName ), table = if Table.RowCount(tableRow) > 0 then tableRow{0}[Data] else null // handle missing tables gracefully in table, // 5. Get all tables (some could be null if not present) AllTables = List.Transform(ConnectionRows, GetTable), // 6. Remove any nulls in case some tables do not exist PresentTables = List.RemoveNulls(AllTables), // 7. Combine into one table Combined = Table.Combine(PresentTables) in Combined
Hi Believer ,
If you get a chance, please review the responses shared by lbendlin and AnkitKukreja , both community members have explained the possible solutions well.
If you need any additional information, please let us know.
lbendlin , Thanks for your continuous valuable inputs.
I left this question open in the hopes that someone else would have a more complete answer. Basically I heard "this is hard, don't do it if you don't have to" and "yes, this is hard but here's what you can do in theory..." 🙂 At this point I suppose I need to ask Google instead... <wink>