02 - Design and Develop Data Processing

Create an Azure Stream Analytics job to process data on the edge. Connect the new Azure Stream Analytics job with other IoT Edge modules. Deploy the Azure Stream Analytics job to an IoT Edge device from the Azure portal.

When you create an Azure Stream Analytics job to run on an IoT Edge device, it needs to be stored in a way that can be called from the device. You can use an existing Azure Storage account, or create a new one now.

Use the Append Variable activity to add a value to an existing array variable defined in a Data Factory or Synapse Analytics pipeline.

Tumbling window triggers are a type of trigger that fires at a periodic time interval from a specified start time, while retaining state. Tumbling windows are a series of fixed-sized, non-overlapping, and contiguous time intervals. A tumbling window trigger has a one-to-one relationship with a pipeline and can only reference a singular pipeline.

Data integration scenarios often require customers to trigger pipelines based on events happening in storage account, such as the arrival or deletion of a file in Azure Blob Storage account. Data Factory and Synapse pipelines natively integrate with Azure Event Grid, which lets you trigger pipelines on such events.

Azure Blob storage, Azure Data Lake Storage Gen2, and Azure SQL Database are currently supported as input sources for reference data.

Event Hubs, IoT Hub, Azure Data Lake Storage Gen2 and Blob storage are supported as data stream input sources.

A data stream is an unbounded sequence of events over time. Stream Analytics jobs must include at least one data stream input.

Reference data is either completely static or changes slowly. It is typically used to perform correlation and lookups.

Built-in temporal operators, such as windowed aggregates, temporal joins, and temporal analytic functions. Native Azure input and output adapters Support for slow changing reference data (also known as a lookup tables), including joining with geospatial reference data for geofencing. Integrated solutions, such as Anomaly Detection Multiple time windows in the same query Ability to compose multiple temporal operators in arbitrary sequences.

Stream Analytics Query Language (SAQL)

The conditional split transformation routes data rows to different streams based on matching conditions. The conditional split transformation is similar to a CASE decision structure in a programming language.

disjoint is false because the data goes to the first matching condition.

Azure Blob Storage (JSON, Avro, Text, Parquet) Azure Data Lake Storage Gen1 (JSON, Avro, Text, Parquet) Azure Data Lake Storage Gen2 (JSON, Avro, Text, Parquet) Azure Synapse Analytics Azure SQL Database Azure CosmosDB

Use the flatten transformation to take array values inside hierarchical structures such as JSON and unroll them into individual rows. This process is known as denormalization.