Skip to main content
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Did you hear? There's a new SQL AI Developer certification (DP-800). Start preparing now and be one of the first to get certified. Register now

trolson
‎04-11-2024 11:01 AM

Shortcuts to Google Cloud Storage, now available in Public Preview

We are excited to announce that you can now create OneLake shortcuts to your Google Cloud Storage (GCS) buckets. With the addition of GCS, you can now utilize cross-cloud shortcuts to analyze your data across all three major cloud platforms. Shortcuts in OneLake allow you to connect to your existing data through a single unified name space without having to copy or move data. Just open a Lakehouse in Fabric, create a shortcut to GCS, AWS S3 or ADLS Gen2 and immediately start analyzing your data through Spark, SQL and Power BI.

Shortcuts_to_Google_Cloud_Storage_now_available_in_Public_PreviewShortcuts_to_Google_Cloud_Storage_now_available_in_Public_Preview

Description automatically generated">

How to create GCS shortcuts

Open a Lakehouse in Fabric and create a new shortcut. If your target data is already in the Delta/Parquet format, create the shortcut in the “Tables” section of the Lakehouse. This will allow metadata synchronization across Spark, SQL and Power BI. If you target data is in another format, create the shortcut in the “Files” section of the Lakehouse.

Screenshot_of_right_click_context_menu_showing_where_to_select_New_shortcut_fromScreenshot_of_right_click_context_menu_showing_where_to_select_New_shortcut_from

Under External Sources, select Google Cloud Storage.

Screenshot_of_the_New_shortcut_window_showing_the_available_shortcut_types._TheScreenshot_of_the_New_shortcut_window_showing_the_available_shortcut_types._The

Configure your connection settings.

Screenshot_of_the_New_shortcut_window_showing_the_Connection_settings_and_ConnecScreenshot_of_the_New_shortcut_window_showing_the_Connection_settings_and_Connec

In the connection URL, you can specify either the global endpoint for the GCS storage service or a bucket specific endpoint.

GCS shortcuts utilize HMAC keys for authorization with Google. HMAC keys can be created for both user accounts and service accounts. Follow the guide here to create an HMAC key: Manage HMAC keys for service accounts.

The account associated with the HMAC key must have permission to access the data within GCS. If you provided a bucket specific URL, the account needs the following permissions:

  • storage.objects.get
  • stoage.objects.list
If you provided the global endpoint in the URL, the account would need the following additional permissions:
  • storage.buckets.list
Once your connection is configured, you can browse your storage account in GCS.

Screenshot_of_the_storage_browse_window_with_multiple_folders_selectedScreenshot_of_the_storage_browse_window_with_multiple_folders_selected

In this view, you can select multiple target locations to create shortcuts to.

The review page allows you to see all of the selections you made.

Screenshot_of_shortcut_review_page_with_options_to_rename_and_delete_shortcutsScreenshot_of_shortcut_review_page_with_options_to_rename_and_delete_shortcuts

Here you can use the pencil button to rename any of your selections. The trash button can be used to remove any unwanted selections. Click create and your shortcuts will appear in your lakehouse.

Screenshot_showing_a_Lake_view_list_of_tables_that_display_the_shortcut_symbolScreenshot_showing_a_Lake_view_list_of_tables_that_display_the_shortcut_symbol

Now start analyzing your data! Open the SQL endpoint and start writing new queries. Create a new Notebook and analyze your data with Spark. Open the semantic model and create a new Power BI report.

Caching

Just like AWS S3 shortcuts, GCS shortcuts also support caching. With caching enabled, egress costs are greatly reduced. As files are read through the GCS shortcut, the files are stored in a cache for the Fabric workspace. Subsequent read requests are served from cache rather than GCS directly.

Cache enablement is configured at a workspace level. To enable shortcut caching, workspace admins can start by opening a Fabric workspace and selecting “Workspace settings”

Shortcuts_to_Google_Cloud_Storage_now_available_in_Public_PreviewShortcuts_to_Google_Cloud_Storage_now_available_in_Public_Preview

In the workspace settings panel, select the “OneLake” tab. Switch the toggle for “Enable Cache for Shortcuts” to “On”. Then click the “Save” button.

Screenshot_of_workspace_settings_panel_with_OneLake_tab_selectedScreenshot_of_workspace_settings_panel_with_OneLake_tab_selected

Caching is now enabled for all GCS, S3 and S3 compatible shortcuts in that workspace. To learn more about shortcut caching see our blog here: Reduce egress costs with S3 shortcuts in OneLake | Microsoft Fabric Blog | Microsoft Fabric.

References

0