Competitors Data Collector - Revision history

Falkner at 21:39, 30 December 2015

2015-12-30T21:39:52Z

Falkner at 21:58, 20 December 2015

2015-12-20T21:58:28Z

Falkner at 21:12, 20 December 2015

2015-12-20T21:12:33Z

Falkner at 20:00, 20 December 2015

2015-12-20T20:00:59Z

Falkner at 20:00, 20 December 2015

2015-12-20T20:00:13Z

Tobias: /* Overview */

2015-12-09T13:53:44Z

‎Overview

Tobias: /* Overview */

2015-12-09T13:51:26Z

‎Overview

Tobias at 13:25, 9 December 2015

2015-12-09T13:25:26Z

Tobias at 13:24, 9 December 2015

2015-12-09T13:24:26Z

Tobias: /* VPN Service */

2015-12-09T13:23:16Z

‎VPN Service

@@ Line 231: / Line 231: @@
 ; With which parameters does the service scale?
 {{comment|How many concurrent E-Shops, how many concurrent products and how many users/E-Shop customers are possible without causing loss in quality/speed for the hardware described above?}}
-: The parameter most relevant for the scaling of this module is the number of scraping jobs, i.e. the number of products monitored multiplied with the number of competitors for which these products are monitored or expressed mathematically: Σ <sub>E-Shops</sub> ( Σ <sub>Products</sub> (Number of Competitors ) )
+: The parameter most relevant for the scaling of this module is the number of scraping jobs, i.e. the number of products monitored multiplied with the number of competitors for which these products are monitored or expressed mathematically: Σ <sub>E-Shops</sub> ( Σ <sub>Products</sub> (Number of Competitors ) ). The user testing during the project came near the limits of the CPU capacity for VM2 with 10 competitors monitored while only 25% of the RAM was in use. The storage is not a limit as the scraping data does not need to be kept for more than 2 days and the database limits are not in sight with the current usage.
 ; If higher scaling was wanted, which of the hardware parameters would need to be increased?
-: CPU, RAM, Storage
+: CPU, RAM
 ; What else would be adjusted for higher scalability?
-: If more CPUs are available the scaling works within the VM. If the scheduling should be run on several VMs further parallelization would be necessary. The data extraction is already parallelized, however this is not speed-critical and therefore not relevant for scaling.
+: If more CPUs/Cores are available further parallelization of the program code is necessary. The data extraction is already parallelized, however this is not speed-critical and therefore not relevant for scaling.
 ; Which further configuration would be necessary?
 : …

@@ Line 229: / Line 229: @@
 === Limitations of the service ===
-; How many concurrent E-Shops with how many concurrent products are possible without causing loss in quality/speed for the hardware described above?
+; With which parameters does the service scale?
-: …
+{{comment|How many concurrent E-Shops, how many concurrent products and how many users/E-Shop customers are possible without causing loss in quality/speed for the hardware described above?}}
 ; If higher scaling was wanted, which of the hardware parameters would need to be increased?
-: …
+: CPU, RAM, Storage
 ; What else would be adjusted for higher scalability?
-: …
+: If more CPUs are available the scaling works within the VM. If the scheduling should be run on several VMs further parallelization would be necessary. The data extraction is already parallelized, however this is not speed-critical and therefore not relevant for scaling.
 ; Which further configuration would be necessary?
 : …

@@ Line 42: / Line 42: @@
 {{VM | text = As mentioned in the previous sections the API of the service is separated from the data collection modules, thus the both parts are running on two separate VMs. The VMs have the following specification:
 | os = Windows 8 Enterprise, 64 Bit
-| cpu = 2 Processors, 2.27GHz
+| cpu = 2 Processors, 2.27GHz, 4 cores (VM1) + 8 cores (VM2)
-| ram = 32
+| ram = 8 GB (VM 1) + 32 GB (VM2)
 | hdd = 50 GB (VM1) + 500 GB (VM2)
-| network = 1 Gbit/s
+| network = 10 Gbit/s
-| network required =
+| network required = no info available
 }}
 VM1 needs to be available from external networks (internet). VM2 should not be available from outside.

@@ Line 45: / Line 45: @@
 | ram = 32
 | hdd = 50 GB (VM1) + 500 GB (VM2)
-| network = 1Gbit
+| network = 1 Gbit/s
 | network required =
 }}

@@ Line 31: / Line 31: @@
 {{Hardware | text =
-| model =
+| model = Cisco UCS B200 M4 und M2 / Cisco UCS B230 M2
-| cpu =
+| cpu = 2 socket CPUs with 12, 20 or 24 cores
-| ram =
+| ram = 256 GB
-| hdd =
+| hdd = SAN Storage mirrored HP 3par System 7400c with 100 TB each
-| network =
+| network = between VMs: 10 Gbit, SAN Storage connection: 8 Gbit FibreChannel at minimum, outward: 10 Gbit Ethernet
-| hypervisor = VMware
+| hypervisor = VMware ESX 6 with vSphere Center 6 in the Cluster
 | balancing = none
 }}

@@ Line 1: / Line 1: @@
 === Overview ===
+[[File:Competitors Data Collector.png|thumbnail|Overview of the Competitors' Data Collector]]
 The Competitors’ Data Collector includes the four main classes
 # Scheduler,
@@ Line 7: / Line 7: @@
 # ProductResolver.
 It provides the ECC function ≫Price Monitor≪ for collecting and comparing product prices of the own e-shop with those of the competitors. The main service and its API are distributed over two separate virtual machines (VMs) as shown in Figure 2. The reason for this distribution is the need to run a VPN service which changes the IP address of the VM running the scraping service (VM2) in a regular cycle as some e-shop websites would not accept too many requests coming from one single IP address. But the VM running the API (VM1) needs to be available under a certain address as other service modules need to send and receive data from by querying the API.
 The components on the VMs are communicating over the REST API of the service module, thus the scraper fetches and writes data from and into the database running on VM1 through a request to the service API. The database storing the clean data includes the data provided by the user, e.g. URL of the own e-shop and the e-shops of the competitors or the definition of the data collection tasks (price data which shall be collected for which product and which competitors in which time period). Additionally, the database includes the collected product and price data for the defined products and the specified competitors which is displayed in the Price Monitor of the ECC. The database running on VM2 stores the data gathered from the e-shop website which needs to be assigned to the product requests of the user in order to be able to find the right data sets of the data collection jobs defined by the users within the collected data of the whole e-shop websites. Additionally, the database stores information about running and finished data collection jobs which is required by the scheduler.

@@ Line 8: / Line 8: @@
 It provides the ECC function ≫Price Monitor≪ for collecting and comparing product prices of the own e-shop with those of the competitors. The main service and its API are distributed over two separate virtual machines (VMs) as shown in Figure 2. The reason for this distribution is the need to run a VPN service which changes the IP address of the VM running the scraping service (VM2) in a regular cycle as some e-shop websites would not accept too many requests coming from one single IP address. But the VM running the API (VM1) needs to be available under a certain address as other service modules need to send and receive data from by querying the API.
-[[File:Competitors Data Collector|thumbnail|Overview of the Competitors' Data Collector]]
+[[File:Competitors_Data_Collector|Overview of the Competitors' Data Collector]]
 The components on the VMs are communicating over the REST API of the service module, thus the scraper fetches and writes data from and into the database running on VM1 through a request to the service API. The database storing the clean data includes the data provided by the user, e.g. URL of the own e-shop and the e-shops of the competitors or the definition of the data collection tasks (price data which shall be collected for which product and which competitors in which time period). Additionally, the database includes the collected product and price data for the defined products and the specified competitors which is displayed in the Price Monitor of the ECC. The database running on VM2 stores the data gathered from the e-shop website which needs to be assigned to the product requests of the user in order to be able to find the right data sets of the data collection jobs defined by the users within the collected data of the whole e-shop websites. Additionally, the database stores information about running and finished data collection jobs which is required by the scheduler.

← Older revision		Revision as of 13:25, 9 December 2015
Line 1:		Line 1:
−	== Overview ==	+	=== Overview ===

	The Competitors’ Data Collector includes the four main classes		The Competitors’ Data Collector includes the four main classes

@@ Line 230: / Line 230: @@
 # https://www.hidemyass.com/
 === Limitations of the service ===
 ; How many concurrent E-Shops with how many concurrent products are possible without causing loss in quality/speed for the hardware described above?
@@ Line 241: / Line 240: @@
 : …
-.3.9	Contact Information Competitors’ Data Collector Service
+=== Contact Information Competitors’ Data Collector Service ===
 Andrea Horch, [mailto:andrea.horch@iao.fraunhofer.de andrea.horch@iao.fraunhofer.de], +49 711 970-2349

@@ Line 226: / Line 226: @@
 === VPN Service ===
- In order to hide the IP address of VM2 to the e-shops where data is collected as some e-shops will break the connection to an external machine which has too many hits on the website the installation of a VPN service hiding and changing the own IP address is recommended. Thus, a service as one of the following should be installed:
+In order to hide the IP address of VM2 to the e-shops where data is collected as some e-shops will break the connection to an external machine which has too many hits on the website the installation of a VPN service hiding and changing the own IP address is recommended. Thus, a service as one of the following should be installed:
 # https://www.perfect-privacy.com/
 # https://www.hidemyass.com/