Skip to main content

Data streaming

Data streaming is the continuous transfer of data at a steady, high-speed rate.
Although the concept of data streaming is not new, its practical applications are a relatively recent development. This is because in the early years of the world wide web, internet connectivity was not always reliable and bandwidth limitations often prevented streaming data to arrive at its destination in an unbroken sequence. Developers created buffers to allow data streams to catch up, but the resulting jitter caused the user experience to be so poor that most consumers preferred to download content rather than stream it.

Today, with the advent of broadband internet, cloud computing and the internet of things (IoT), there is an increased interest in analyzing the data from streaming sources to make data-driven decisions in real time. To facilitate the need for real-time information from disparate data sources, many companies have replaced traditional batch processing with streaming data architectures that can accommodate batch processing.

In batch processing, newly arriving data elements are collected in a group and the entire group is processed at some future time. In contrast, a streaming data architecture processes data in motion and an ETL batch is treated as just one more event in a continuous stream of events.

To benefit from data streaming, businesses supported by streaming architectures require powerful analytics tools for ingesting and processing information. Popular tools for working with data streams include: 

Amazon Kinesis Firehose - an Amazon Web Service (AWS) for processing big data in real time. Kinesis is capable of processing hundreds of terabytes per hour from high volumes of streaming data from sources such as operating logs, financial transactions and social media feeds.

Apache Flink - a distributed data processing platform for use in big data applications, primarily involving analysis of data stored in Hadoop clusters. Flink handles both batch and stream processing jobs, with data streaming the default implementation and batch jobs running as special-case versions of streaming applications.

Comments

Post a Comment

Popular posts from this blog

Ghosting

Ghosting is to cease communications without notification. The use of the word "ghost" as a verb originated in social media in reference to dating, but the term is now used by employers to describe employees and potential employees who suddenly disappear. Typically, ghosting is used to describe: Job candidates who suddenly stop responding to messages. New hires who fail to show up for their first day of work. Employees who do not show up for a shift. Employees who leave work in the middle of the day and never come back. Some analysts blame ghosting on millennial entitlement. The reasoning is that members of the millennial generation have been brought up to feel they are special -- so special, in fact, that they do not need to follow conventional rules of behavior. Other analysts, however, maintain that ghosting behavior stems from changes in the job market and the phenomenon is simply a reflection of the laws of supply and demand in a healthy jo...
Post a Comment
Read more

Data deduplication

Data deduplication -- often called intelligent compression or single-instance storage -- is a process that eliminates redundant copies of data and reduces storage overhead. Data deduplication techniques ensure that only one unique instance of data is retained on storage media, such as disk, flash or tape. Redundant data blocks are replaced with a pointer to the unique data copy. In that way, data deduplication closely aligns with incremental backup, which copies only the data that has changed since the previous backup. For example, a typical email system might contain 100 instances of the same 1 megabyte (MB) file attachment. If the email platform is backed up or archived, all 100 instances are saved, requiring 100 MB of storage space. With data deduplication, only one instance of the attachment is stored; each subsequent instance is referenced back to the one saved copy. In this example, a 100 MB storage demand drops to 1 MB. Target vs. source deduplication Data deduplica...
Post a Comment
Read more

A Graphics Processing Unit (GPU)

A graphics processing unit (GPU) is a computer chip that performs rapid mathematical calculations, primarily for the purpose of rendering images. A GPU may be found integrated with a central processing unit (CPU) on the same circuit, on a graphics card or in the motherboard of a personal computer or server. In the early days of computing, the CPU performed these calculations. As more graphics-intensive applications such as AutoCAD were developed; however, their demands put strain on the CPU and degraded performance. GPUs came about as a way to offload those tasks from CPUs, freeing up their processing power. NVIDIA, AMD, Intel and ARM are some of the major players in the GPU market. GPU vs. CPU A graphics processing unit is able to render images more quickly than a central processing unit because of its parallel processing architecture, which allows it to perform multiple calculations at the same time. A single CPU does not have this capability, although multi...
Post a Comment
Read more