Data Ingestion - Connecting to data source (HTTP)

Data Ingestion - Connecting to data source (HTTP)

Here we are connecting HTTP as data source, following the below steps

-->First create a folder in our workspace, so that all the corresponding works regarding these will store in that folder-->open folder

-->Create a new lake house- dataengineering homepage-new item-search lakehouse-create a new Lakehouse with a name of your choice-create

-->On the Lake view tab in the pane on the left, in the … menu for the Files node, select New subfolder and create a subfolder named new_data.

-->create a pipeline-copydata-datasource-HTTP-Connect data source-give required url(from which you want to ingest),connection-create new connection,connection name-as per your requirement,data gateway-none,authentikind-anonymous

-->next-select as per requirement-for example(Relative URL: Leave blank, Request method: GET, Additional headers: Leave blank, Binary copy: Unselected, Request timeout: Leave blank, Max concurrent connections: Leave blank)

-->next,and wait for the data to be sampled and then ensure that the following settings are selected

(File format: Delimited Text, Column delimiter: Comma (,), Row delimiter: Line feed (\n), First row as header: Selected, Compression type: None)

-->Select Preview data to see a sample of the data that will be ingested. Then close the data preview and select Next.

-->Set the following data destination options, and then select Next:, Root folder: Files, Folder path name: new_data, File name: xxxxx.csv, Copy behavior: None,-->next

-->Set the following file format options and then select Next:, File format: DelimitedText, Column delimiter: Comma (,), Row delimiter: Line feed (\n), Add header to file: Selected, Compression type: None

-->On the Copy summary page, review the details of your copy operation and then select Save + Run. A new pipeline containing a Copy Data activity is created. -->Now pipeline will start with status as in queued/progress.

-->After successful run, check the data in corresponding data folder according to your selection