Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Data ingestion is the process of loading data from one or more sources into a table in Azure Data Explorer. Once ingested, the data is available for query. In this article, you learn how to get data from a local file into either a new or existing table.
For general information on data ingestion, see Azure Data Explorer data ingestion overview.
Prerequisites
- A Microsoft account or a Microsoft Entra user identity. An Azure subscription isn't required.
- Sign-in to the Azure Data Explorer web UI.
- An Azure Data Explorer cluster and database. Create a cluster and database.
Get data
From the left navigation pane, select Query.
Right-click on the database where you want to ingest the data. Select Get data.
Select data source
In the Get data window, the Source tab is selected.
Select the data source from the available list. In this example, you're ingesting data from a Local file.
Note
Ingestion supports a maximum file size of 6 GB. The recommendation is to ingest files between 100 MB and 1 GB.
Configure data ingestion
Select a target database and table. To ingest data into a new table, select + New table and enter a table name.
Note
Table names can be up to 1,024 characters, including spaces, alphanumeric characters, hyphens, and underscores. Special characters aren't supported.
Drag files into the window, or select Browse for files.
Note
You can add up to 1,000 files. Each file can be a maximum of 1 GB uncompressed.
Select Next.
Inspect
The Inspect tab opens with a preview of the data.
To complete the ingestion process, select Finish.
Optionally:
- Change the automatically inferred data format by selecting the desired format from the dropdown. See Data formats supported by Azure Data Explorer for ingestion.
- Edit columns.
- Explore Advanced options based on data type.
Edit columns
Note
- For tabular formats (CSV, TSV, PSV), you can't map a column twice. To map to an existing column, first delete the new column.
- You can't change an existing column type. If you try to map to a column having a different format, you may end up with empty columns.
The changes you can make in a table depend on the following parameters:
- Table type is new or existing
- Mapping type is new or existing
| Table type | Mapping type | Available adjustments |
|---|---|---|
| New table | New mapping | Rename column, change data type, change data source, mapping transformation, add column, delete column |
| Existing table | New mapping | Add column (on which you can then change data type, rename, and update) |
| Existing table | Existing mapping | none |
Mapping transformations
Some data format mappings (Parquet, JSON, and Avro) support simple ingest-time transformations. To apply mapping transformations, create or update a column in the Edit columns window.
Mapping transformations can be performed on a column of type string or datetime, with the source having data type int or long. Supported mapping transformations are:
- DateTimeFromUnixSeconds
- DateTimeFromUnixMilliseconds
- DateTimeFromUnixMicroseconds
- DateTimeFromUnixNanoseconds
Advanced options based on data type
Tabular (CSV, TSV, PSV):
If you're ingesting tabular formats in an existing table, you can select the table mapping drop-down and select Use exixting mapping. Tabular data doesn't necessarily include the column names that are used to map source data to the existing columns. When this option is checked, mapping is done by-order, and the table schema remains the same.
Otherwise, create a new mapping.
To use the first row as column names, select First row header.
JSON:
- To determine column division of JSON data, select Nested levels, from 1 to 100.
Summary
In the Data preparation window, all three steps show green check marks when data ingestion finishes successfully. You can view the commands that each step used, or select a card to query, visualize, or drop the ingested data.
