Hello !
Trace: • Authors • Installing Eclipse for Linux • IoT Data Analysis • Second Generation Applications • STM_IoT_2: Connecting to the WiFi Access Point and presenting IP • Sensor module • RGB LED example • Thermistor • Authors • Lidar

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
en:iot-reloaded:iot_data_analysis [2024/07/19 16:05] – agrisniken:iot-reloaded:iot_data_analysis [2025/05/17 08:56] (current) – agrisnik
Line 1: Line 1:
 ====== IoT Data Analysis ====== ====== IoT Data Analysis ======
-{{:en:iot-open:czapka_p.png?50|General audience classification icon}}{{:en:iot-open:czapka_b.png?50|General audience classification icon}}{{:en:iot-open:czapka_e.png?50|General audience classification icon}}\\ +IoT systems are built to provide better insights into different processes and systems to make better decisions. The insights are provided by measuring the statuses of the systems or process elements represented by data. Unfortunately, the bits and bytes become useless without adequately interpreting the data content. Therefore, providing a means for understanding data is an essential property of a modern IoT system. 
- +
-IoT systems, in their essence, are built to act as a tool for getting better insights into different processes and systems in order to make better decisions. The insights are provided by measuring the statuses of the systems or process elements represented by data. Unfortunately, without properly interpreting the data content, they turn into useless bits and bytes. Therefore, providing a means for understanding data is an essential property of a modern IoT system. +
 Today, IoT systems produce a vast amount of data, which is very hard to use manually. Thanks to modern hardware and software developments, it is possible to develop fully or semi-automated systems for data analysis and interpretation, which may go further into decision-making and acting according to the decisions.  Today, IoT systems produce a vast amount of data, which is very hard to use manually. Thanks to modern hardware and software developments, it is possible to develop fully or semi-automated systems for data analysis and interpretation, which may go further into decision-making and acting according to the decisions. 
  
-As various resources have stated, IoT in most cases, complies with the so-called big 5Vs of Big Data, where just one correspondence is needed to solve a Big Data problem. As has been explained by Jain et al. ((Jain, A., Mittal, S., Bhagat, A., Sharma, D.K. (2023). Big Data Analytics and Security Over the Cloud: Characteristics, Analytics, Integration and Security. In: Srivastava, G., Ghosh, U., Lin, J.CW. (eds) Security and Risk Analysis for Intelligent Edge Computing. Advances in Information Security, vol 103. Springer, Cham. https://doi.org/10.1007/978-3-031-28150-1_2)) Big Data might be of different forms, volumes and structures, and in general, the 5Vs i.e. Volume, Variety, Veracity, Velocity and Value might be interpreted as follows:+As various resources have stated, IoT, in most cases, complies with the so-called big 5Vs of Big Data, where just one correspondence is needed to solve a Big Data problem. As has been explained by Jain et al. ((Jain, A., Mittal, S., Bhagat, A., Sharma, D.K. (2023). Big Data Analytics and Security Over the Cloud: Characteristics, Analytics, Integration and Security. In: Srivastava, G., Ghosh, U., Lin, J.CW. (eds) Security and Risk Analysis for Intelligent Edge Computing. Advances in Information Security, vol 103. Springer, Cham. https://doi.org/10.1007/978-3-031-28150-1_2)) Big Data might be of different forms, volumes and structures, and in general, the 5Vs, e.g. Volume, Variety, Veracity, Velocity and Value might be interpreted as follows:
  
 === Volume === === Volume ===
Line 13: Line 11:
 === Variety === === Variety ===
  
-As Jain explained, big data is highly heterogeneous in terms of source, kind, and nature. Having different systems, processes, sensors, and other data sources, variety is usually a distinctive feature of practical IoT systems. For instance, a system of intelligent office buildings would need data from a building management system, appliances and independent sensors, and external sources like weather stations or forecasts from appropriate external weather forecast APIs (Application programming interfaces). Additionally, the given system might require historical data from other sources, like XML documents, CSV files or other sources, diversifying the sources even more. +Jain explained that Big Data is highly heterogeneous regarding source, kind, and nature. Having different systems, processes, sensors, and other data sources, variety is usually a distinctive feature of practical IoT systems. For instance, a system of intelligent office buildings would need data from a building management system, appliances and independent sensors, and external sources like weather stations or forecasts from appropriate external weather forecast APIs (Application programming interfaces). Additionally, the given system might require historical data from other sources, like XML documents, CSV files or other sources, diversifying the sources even more. 
  
 === Veracity === === Veracity ===
Line 21: Line 19:
 === Velocity === === Velocity ===
  
-Data velocity characterises the data bound to the time and its importance during a specific period or at a particular time instant. A good example might be any real-time system like an industrial process control system, where reactions or decisions must be made during a fixed period of time, requiring data at particular time instants. In this case, data has a flow nature of a particular density. +Data velocity characterises the data bound to the time and its importance during a specific period or at a particular time instant. A good example might be any real-time system like an industrial process control system, where reactions or decisions must be made during a fixed period, requiring data at particular time instants. In this case, data has a flow nature of a specific density. 
  
 === Value === === Value ===
  
-Since the IoT systems and their data analysis subsystems are built to add value to their owner, the costs of the development and ownership should exceed the returned value. The system is of low or no value if it does not apply. +Since IoT systems and their data analysis subsystems are built to add value to their owners, the development and ownership costs should not exceed the returned value. A system is of low or no value if it does not apply. 
  
 ====== ====== ====== ======
-Dealing with big data requires specific hardware and software infrastructure. While there is a certain number of typical solutions and a lot more customise, some of the most popular are explained here:+Dealing with Big Data requires specific hardware and software infrastructure. While there is a certain number of typical solutions and a lot more customised, some of the most popular are explained here:
  
 === Relational DB-based systems === === Relational DB-based systems ===
  
 Those systems are based on well-known relational data models and appropriate database management systems like MS SQL Server, Oracle Server, MySQL, etc. There are some advantageous features of those systems, for instance: Those systems are based on well-known relational data models and appropriate database management systems like MS SQL Server, Oracle Server, MySQL, etc. There are some advantageous features of those systems, for instance:
-  * Advantages of SQL (Structured Querying Language): enabling easy manipulation of the data while maintaining a relatively good expressiveness of the data model; +  * Advantages of SQL (Structured Querying Language): enabling easy data manipulation while maintaining a relatively good expressiveness of the data model. 
-  * A well-designed set of software tools and interfaces enabling integration with a large number of different systems;+  * A well-designed set of software tools and interfaces enabling integration with many different systems.
   * A lot of built-in data processing routines (stored procedures) provide higher development productivity.   * A lot of built-in data processing routines (stored procedures) provide higher development productivity.
   * Enables asynchronous reactions to events by triggering internal events.    * Enables asynchronous reactions to events by triggering internal events. 
   * Data reading might be scaled out using multiple entities, while writing might be scaled up using more productive servers.    * Data reading might be scaled out using multiple entities, while writing might be scaled up using more productive servers. 
-Unfortunately, scaling out data writing is not always possible and is usually supported at a high cost for software products. +Unfortunately, scaling out data writing is not always possible and is usually supported at a high cost for software products (figure 1). 
  
 <figure RelationalDBMS> <figure RelationalDBMS>
-{{ :en:iot-reloaded:dbs2.png?400 | Relational DBMS scaling options}} +{{ :en:iot-reloaded:dbs2.png?400 | Relational DBMS Scaling Options}} 
-<caption>Relational DBMS scaling options</caption>+<caption>Relational DBMS Scaling Options</caption>
 </figure> </figure>
  
 === Complex Event Processing (CEP) systems === === Complex Event Processing (CEP) systems ===
  
-CEP systems are very application-tailored, enabling significant productivity at a reasonable cost. High productivity is usually needed for processing data streams, such as voice or video. Maintaining a limited time window for data processing is possible, which is relevant for systems that are close to real-time. +CEP systems are very application-tailored, enabling significant productivity at a reasonable cost. High productivity is usually needed for processing data streams, such as voice or video. Maintaining a limited time window for data processing is possible, which is relevant for systems close to real-time (figure {{ref>CEP_systems}}). 
 Some of the most common drawbacks to be considered are: Some of the most common drawbacks to be considered are:
   * It might be scaled up only by introducing higher productivity hardware, which is limited by the application-specific design. To some extent, the design might be more flexible if microservices and containerisation are applied.    * It might be scaled up only by introducing higher productivity hardware, which is limited by the application-specific design. To some extent, the design might be more flexible if microservices and containerisation are applied. 
-  * Due to the factors mentioned above and the complexity, the maintenance costs are usually higher than a universal design.+  * Due to the factors mentioned above and the complexity, the maintenance costs are usually higher than a universal design (figure 2). 
 + 
 +<figure CEP_systems> 
 +{{ :en:iot-reloaded:ceps.png?400 | CEP Systems}} 
 +<caption>CEP Systems</caption> 
 +</figure>
  
 === NoSQL systems === === NoSQL systems ===
  
-As the name suggests, the main characteristic is higher flexibility in data models, which overcomes the limitations of highly structured relational data models. NoSQL systems are usually distributed, where the distribution is the primary tool to enable supreme flexibility. In IoT systems, software typically gets older faster than hardware, which requires the maintenance of many versions of communication protocols and data formats to ensure back compatibility. Another reason is the variety of hardware suppliers, where some protocols or data formats are specific to the given vendor.  +As the name suggests, the main characteristic is higher flexibility in data models, which overcomes the limitations of highly structured relational data models (figure {{ref>NoSQL_systems}}). NoSQL systems are usually distributed, where the distribution is the primary tool to enable supreme flexibility. In IoT systems, software typically gets older faster than hardware, which requires the maintenance of many versions of communication protocols and data formats to ensure back compatibility. Another reason is the variety of hardware suppliers, where some protocols or data formats are specific to the given vendor.  
-It also provides a means for scalability out and up, enabling high future tolerance and resilience. A typical approach is to use a key-value or key-document approach, where a unique key indexes incoming data blocks or documents (JSON, for instance). +It also provides a means for scalability out and up, enabling high future tolerance and resilience. A typical approach uses a key-value or key-document approach, where a unique key indexes incoming data blocks or documents (JSON, for instance). 
-Some other designs might extend the SQL data models by others – object models, graph models, or the mentioned key-value models, providing highly purpose-driven and, therefore, productive designs. However, the complexity of the design raises problems of data integrity as well as the complexity of maintenance. +Some other designs might extend the SQL data models by others – object models, graph models, or the mentioned key-value models, providing highly purpose-driven and, therefore, productive designs. However, the complexity of the design raises problems of data integrity as well as the complexity of maintenance (figure 3).  
 + 
 +<figure NoSQL_systems> 
 +{{ :en:iot-reloaded:keyvalue.png?400 | NoSQL Systems}} 
 +<caption>NoSQL Systems</caption> 
 +</figure>
  
 === In-memory data grids === === In-memory data grids ===
  
 This is probably the most productive type of system, providing high flexibility, productivity and scalability. Because these systems are designed to operate in servers RAM, the in-memory data grids are the best choice for data preprocessing in IoT systems due to their high productivity and ability to scale dynamically depending on actual workloads. They provide all the benefits of the CEP and Relational systems, adding a scale-out functionality for data writing. There are only two major drawbacks – limited RAM and high development costs. Some examples of available solutions: This is probably the most productive type of system, providing high flexibility, productivity and scalability. Because these systems are designed to operate in servers RAM, the in-memory data grids are the best choice for data preprocessing in IoT systems due to their high productivity and ability to scale dynamically depending on actual workloads. They provide all the benefits of the CEP and Relational systems, adding a scale-out functionality for data writing. There are only two major drawbacks – limited RAM and high development costs. Some examples of available solutions:
-  * Hazelcast    http://hazelcast.org (Uses in-memory NoSQL DB) +  * Hazelcast ((http://hazelcast.org)) (Uses in-memory NoSQL DB) 
-  * JBOSS Infinispan www.infinispan.org (Appache based key-value store) +  * JBOSS Infinispan ((https://infinispan.org/)) (Apache-based key-value store) 
-  * IBM eXtreme Scale ibm.com/software/products/en/websphere-extreme-scale  +  * IBM eXtreme Scale ((https://www.ibm.com/products/ibm-websphere-extreme-scale))  
-  * Gigaspace XAP Elastic caching edition www.gigaspaces.com/xap-in-memory-caching-scaling/datagrid (Transactions-based microservices) +  * Gigaspace XAP Elastic caching edition ((www.gigaspaces.com/xap-in-memory-caching-scaling/datagrid)) (Transactions-based microservices) 
-  * Oracle Coherence  www.oracle.com/technetwork/middleware/coherence +  * Oracle Coherence  ((www.oracle.com/technetwork/middleware/coherence)) 
-  * Terracotta enterprise suite www.terracotta.org/products/enterprise-suite  +  * Terracotta enterprise suite ((www.terracotta.org/products/enterprise-suite))  
-  * Pivotal Gemfire    pivotal.io/big-data/pivotal-gemfire  +  * Pivotal Gemfire    ((https://pivotal.io/big-data/pivotal-gemfire))
  
-======  ====== 
  
-This chapter is devoted to the main groups of algorithms for numerical data analysis and interpretation, covering both mathematical foundations and application specifics in the context of IoT. The chapter is split into the following subchapter:+<WRAP excludefrompdf> 
 +This chapter is devoted to the main groups of algorithms for numerical data analysis and interpretation, covering both mathematical foundations and application specifics in the context of IoT. The chapter is split into the following subchapters:
  
   * [[en:iot-reloaded:Data_products_development]],   * [[en:iot-reloaded:Data_products_development]],
Line 80: Line 88:
   * [[en:iot-reloaded:Introduction_to_time_series_analysis]],   * [[en:iot-reloaded:Introduction_to_time_series_analysis]],
   * [[en:iot-reloaded:Hints_for_further_readings_on_AI]].   * [[en:iot-reloaded:Hints_for_further_readings_on_AI]].
 +</WRAP>
en/iot-reloaded/iot_data_analysis.1721405116.txt.gz · Last modified: 2024/07/19 16:05 by agrisnik
CC Attribution-Share Alike 4.0 International
www.chimeric.de Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0