Dr.Sum Data Funnel
Dr.Sum Data Funnel is a tool for processing continuously occurring time-series data in real time. The collected time-series data can be processed into a format suitable for aggregation and analysis in BI tools such as Dr.Sum Datalizer or MotionBoard. Collected and processed data can be stored in Dr.Sum Server as a standard output destination. In addition, since time-series data is generated continuously, the amount of data increases over time. Therefore, Dr.Sum enables large-volume data storage and high-speed aggregation processing.
Dr.Sum Data Funnel can be applied to the following tasks
- Real-time aggregation and analysis of POS data and inventory data acquired from sensors to plan purchasing and analyze sales of hot sellers.
- Compiling and analyzing data on defective machined products to control quality
Typical processes that can be performed under "Data Processing and Storage" are as follows
- Data in microseconds, which is the frequency of data generation, is summarized in arbitrary units and sent to subsequent processing.
- Remove values that are far out of line with other values or assign flag values that indicate anomalies.
- Process data by comparing with previous values.
- Execute user-created processes as well as processes that are pre-built into the product
Flow of Dr.Sum Data Funnel
Dr.Sum Data Funnel Settings Screen
(A) Pipeline definition (B) Built-in action (C) Custom action
The program will be used to support batch processing and data processing operations performed in data preparation work. It will streamline the overall data preparation process that is performed before importing data into Dr.Sum's database. For instance, DS Script can be used for the following data preparation tasks
- When data is imported from multiple core databases into a database for aggregation, it is processed with SQL statements and imported while changing the structure of the data.
- Data is imported while going through intermediate tables for work, and the intermediate tables are deleted after the work is done.
The data preparation work shown in the first example requires changing the data structure from the core database data structure to the data structure for aggregation.
For instance, items that are not needed for tabulation can be deleted, items that are needed for tabulation can be renamed for analysis, or extracted items can be combined and imported into the database for tabulation. In other words, before data analysis, various pre-processing steps are required depending on the operation and purpose, and DS Script can make the work more efficient.
Python integration function
Dr.Sum provides the ability to work with Python programs. You can execute a Python program from an SQL statement and receive the results of the execution as tabular data. By integrating the various libraries supported by Python, analysis by methods other than crosstabulation becomes possible. For instance, a library for machine learning can be used to analyze the inference data returned by machine learning models.
Dr.Sum has various data integration features to increase the utility and value of data. The more data that can be aggregated within an organization, the easier it will be for field users to use, the more analysis can be democratized, and the more useful results can be obtained for the business. The system can also be automated when data is integrated and managed. Create an environment where anyone can easily handle data with Dr.Sum, create an organization that is connected to data, and realize DX!