Cost estimate - Revision history

Falkner at 20:43, 28 January 2016

2016-01-28T20:43:58Z

Falkner at 20:40, 28 January 2016

2016-01-28T20:40:41Z

Falkner at 20:30, 28 January 2016

2016-01-28T20:30:34Z

Falkner at 20:28, 28 January 2016

2016-01-28T20:28:25Z

Falkner at 19:30, 28 January 2016

2016-01-28T19:30:27Z

Falkner at 11:56, 20 January 2016

2016-01-20T11:56:07Z

Falkner at 11:01, 20 January 2016

2016-01-20T11:01:34Z

Falkner at 11:00, 20 January 2016

2016-01-20T11:00:14Z

Falkner at 10:54, 20 January 2016

2016-01-20T10:54:21Z

Falkner: Created page with "== Cost estimate == For the future owner of the E-Compass Data Mining Services it is essential to be aware of the operational as well as the migration cost. As the migration..."

2016-01-20T10:32:03Z

Created page with "== Cost estimate == For the future owner of the E-Compass Data Mining Services it is essential to be aware of the operational as well as the migration cost. As the migration..."

New page

== Cost estimate ==

For the future owner of the E-Compass Data Mining Services it is essential to be aware of the operational as well as the migration cost.
As the migration of the services to a new data center essentially requires the deployment of a couple of virtual machines images which are available in the OVA (open virtualization archive) format and only minor adjustments by editing the service location URIs and running the available service configuation scripts the migrational cost can be neglected in comparison to the operational cost - as well in terms of finances as in terms of the time required. The complete migration of the services should be done within one or two days.

This leaves the us with the estimation of the operational cost. In order to estimate this we need the following:

'''Virtual Machine Specification for all service modules'''
* provision cost in data center (the cost estimate should include energy consumption and hardware maintenance, redundancy and backup as needed)
* by checking all inclusive public cloud offerings we can get a fairly good estimate of the provision cost, given that we know the requirements in terms of CPUs/Cores, RAM, Storage, Network capacities needed.

'''Limitations'''
In order to get the dimension of a productive environment correctly - and therefore also the cost estimate - we need to derive the limitations of the current implementation of the different services. This includes the knowledge on all parameters that have influence on the scaling of the different services modules. These include:
* current load on VM
* number of E-Shop users
* number of products monitored for each user

This information the allows to derive
* scaling limits for these VM specifications
* VM cost depending on the parameters relevant for scaling
* general provision cost per user, per product or other parameters like e.g. the number of rules defined for the notification & actions engine

'''Maintenance efforts for each service module'''
* estimate of the overall effort to maintain the system
* staff required for service maintenance

'''Helpdesk / user support costs'''
* Experience from user testing: how much effort is needed to get the services set up for each E-Shop?

← Older revision		Revision as of 20:43, 28 January 2016
Line 97:		Line 97:
	In conclusion this would add up to a total cost of approximately 220€ per E-Shop and month including the complete infrastructure (CPU+RAM+storage+network throughput+energy+operations), infrastructure maintenance, software maintenance and user (i.e. E-Shop owner) support.		In conclusion this would add up to a total cost of approximately 220€ per E-Shop and month including the complete infrastructure (CPU+RAM+storage+network throughput+energy+operations), infrastructure maintenance, software maintenance and user (i.e. E-Shop owner) support.

−	As mentioned in the beginning the migration cost would be neglectable compared to the cost of operation as the migrational effort will be very small. Even if we consider that for higher scaling in the range of 100 concurrent E-Shops further parallelisation of program code would be required the development cost ~~would~~ be below the cost for one month of operations. Therefore the figure of 220€ per E-Shop and month should describe the maximum of cost which one has to consider for a future productive operation.	+	As mentioned in the beginning the migration cost would be neglectable compared to the cost of operation as the migrational effort will be very small. Even if we consider that for higher scaling in the range of 100 concurrent E-Shops further parallelisation of program code would be required (that is for the Competitors Data Collector and the Notification & Actions Engine) the development cost should be below the cost for one month of operations. Therefore the figure of 220€ per E-Shop and month should describe the maximum of cost which one has to consider for a future productive operation.

← Older revision		Revision as of 20:40, 28 January 2016
Line 95:		Line 95:
	'''Conclusion'''		'''Conclusion'''

−	In conclusion this would add up to a total cost of approximately 220€ per E-Shop and month including the complete infrastructure, infrastructure maintenance, software maintenance and user (i.e. E-Shop owner) support.	+	In conclusion this would add up to a total cost of approximately 220€ per E-Shop and month including the complete infrastructure (CPU+RAM+storage+network throughput+energy+operations), infrastructure maintenance, software maintenance and user (i.e. E-Shop owner) support.
		+
		+	As mentioned in the beginning the migration cost would be neglectable compared to the cost of operation as the migrational effort will be very small. Even if we consider that for higher scaling in the range of 100 concurrent E-Shops further parallelisation of program code would be required the development cost would be below the cost for one month of operations. Therefore the figure of 220€ per E-Shop and month should describe the maximum of cost which one has to consider for a future productive operation.

@@ Line 7: / Line 7: @@
 '''Virtual Machine Specification for all service modules'''
 * provision cost in data center (the cost estimate should include energy consumption and hardware maintenance, redundancy and backup as needed)
 * by checking all inclusive public cloud offerings we can get a fairly good estimate of the provision cost, given that we know the requirements in terms of CPUs/Cores, RAM, Storage, Network capacities needed.
 '''Maintenance efforts for each service module'''
 * estimate of the overall effort to maintain the system
 * staff required for service maintenance
 '''Helpdesk / user support costs'''
 * Experience from user testing: how much effort is needed to get the services set up for each E-Shop?
 '''Limitations'''
 In order to get the dimension of a productive environment correctly - and therefore also the cost estimate - we need to derive the limitations of the current implementation of the different services. This includes the knowledge on all parameters that have influence on the scaling of the different services modules. These include:
 * current load on VM
@@ Line 29: / Line 33: @@
 '''General Scaling Limits'''
 For a proper operation of the E-Compass Data Mining Services it is also helpful to be aware of some general scaling limits. Questions here are how far scaling can be reached by using more hardware capacity. Depending on the software implementation of the services larger scaling may also require the implementation of further parallelization of the software source code if higher scaling is needed. All these issues need to be considered in order to get an appropriate cost estimation for the productive operation of the services.
 '''Hardware Characteristics in Test-Phase'''
 The following table gives an overview over the hardware characteristics used in the current implementations of the services and therefore over the virtual hardware used in total for all E-Compass Data Mining Services:
 {| class="wikitable"
@@ Line 49: / Line 55: @@
 '''Scaling Limits during Test-Phase'''
 The parameters relevant for scaling are described in the following table below. The term "not in sight" means, that during the test-phase the available hardware was not even close to their limits and therefore this parameter should not be critical for scaling.
@@ Line 69: / Line 76: @@
 '''Infrastructure Cost Estimate'''
 During the test phase the total required storage was below 1 TB. The cost for 1 TB of storage in Amazons S3 storage would be around 30 € per month. As we will see in the following the storage cost is therefore neglectable in comparison to the required server infrastructure.
@@ Line 78: / Line 86: @@
 '''Helpdesk and Maintenance Cost Estimate'''
 In order to come to the full cost of operation the maintenance cost on the software / service side as well as the cost for helpdesk operation (in order to help E-Shop owners with configuration and bugfixing) additional personal costs have to be considered. In order to keep these costs in a reasonable range a 9 to 5 support is proposed with 2 employees (redundancy for illness and vacation) with an average monthly income of 3.000€. The cost for the employer are estimated with 4.500€ per month for each employee which makes a cost of 9.000€ per month. Depending on the wage level in different countries the actual costs may vary significantly!
 '''Relation between Infrastructure and Helpdesk/Maintenance Cost'''
 Making this estimate the costs for support and software maintenance are in a completely different dimension than the infrastructure cost (including infrastructure maintenance). In order to get them to a comparable level the infrastructure could be scaled up to cope with 100 E-Shops instead of 10 E-Shops. From the experience of the testing phase the software maintenance effort and user support should allow for the support of 100 E-Shops with the support personell in the above estimate.
 '''Conclusion'''
 In conclusion this would add up to a total cost of approximately 220€ per E-Shop and month including the complete infrastructure, infrastructure maintenance, software maintenance and user (i.e. E-Shop owner) support.

@@ Line 23: / Line 23: @@
 * number of products monitored for each user
-This information the allows to derive
+This information then allows to derive
 * scaling limits for these VM specifications
 * VM cost depending on the parameters relevant for scaling
 * general provision cost per user, per product or other parameters like e.g. the number of rules defined for the notification & actions engine
 For a proper operation of the E-Compass Data Mining Services it is also helpful to be aware of some general scaling limits. Questions here are how far scaling can be reached by using more hardware capacity. Depending on the software implementation of the services larger scaling may also require the implementation of further parallelization of the software source code if higher scaling is needed. All these issues need to be considered in order to get an appropriate cost estimation for the productive operation of the services.
-The following table gives an overview over the hardware characteristics used in the current implementations of the services and therefor over the virtual hardware used in total for all E-Compass Data Mining Services:
+'''Hardware Characteristics in Test-Phase'''
 {| class="wikitable"
 |-
@@ Line 46: / Line 48: @@
 |}
-The parameters relevant for scaling are described in the following table:
+'''Scaling Limits during Test-Phase'''
 {| class="wikitable"
@@ Line 64: / Line 67: @@
 | Necessary Actions for Scaling || eventually changing the communication between ECC and service other modules || more Storage || more Storage || none || more CPU, RAM & Parallelization || more CPU, RAM & Parallelization
 |}
 Looking at the [https://aws.amazon.com/ec2/details/?nc1=h_ls infrastructure as a service offerings and pricing of Amazon Web Services] as an example of a possible provider of data center infrastructure the optimal server package for the current dimensioning of the E-Compass Data Mining Services (including some scaling reserves) would be an M4 instance with a sufficient number of CPU/Cores and fairly large RAM. The m4.10xlarge instance with 160 GB of memory, 40 vCPUs on a 64-bit platform would arise [https://aws.amazon.com/ec2/pricing/?nc1=h_ls costs of approximately 1.000€ per month] inclduding the complete data center operation with all additional cost, such as e.g. energy consumption and maintenance.
@@ Line 69: / Line 75: @@
 Given that the system dealt with 10 concurrent E-Shops over the testing period without reaching the scaling limits this would make a maximum of 100€ per E-Shop per month on the infrastructure side.
 In order to come to the full cost of operation the maintenance cost on the software / service side as well as the cost for helpdesk operation (in order to help E-Shop owners with configuration and bugfixing) additional personal costs have to be considered. In order to keep these costs in a reasonable range a 9 to 5 support is proposed with 2 employees (redundancy for illness and vacation) with an average monthly income of 3.000€. The cost for the employer are estimated with 4.500€ per month for each employee which makes a cost of 9.000€ per month. Depending on the wage level in different countries the actual costs may vary significantly!
-Making this estimate the costs for support and software maintenance are in a completele different dimension than the infrastructure cost (including infrastructure maintenance). In order to get them to a comparable level the infrastructure could be scaled up to cope with 100 E-Shops instead of 10 E-Shops. From the experience of the testing phase the software maintenance effort and user support should allow for the support of 100 E-Shops with the support personell in the above estimate.
+'''Relation between Infrastructure and Helpdesk/Maintenance Cost'''
-In conclusion this would add up to a total cost of approximately 200€ per E-Shop and month including the complete infrastructure, infrastructure maintenance, software maintenance and user (i.e. E-Shop owner) support.
+'''Conclusion'''

← Older revision		Revision as of 11:56, 20 January 2016
Line 64:		Line 64:
	\| Necessary Actions for Scaling \|\| eventually changing the communication between ECC and service other modules \|\| more Storage \|\| more Storage \|\| none \|\| more CPU, RAM & Parallelization \|\| more CPU, RAM & Parallelization		\| Necessary Actions for Scaling \|\| eventually changing the communication between ECC and service other modules \|\| more Storage \|\| more Storage \|\| none \|\| more CPU, RAM & Parallelization \|\| more CPU, RAM & Parallelization
	\|}		\|}
		+
		+	Looking at the [https://aws.amazon.com/ec2/details/?nc1=h_ls infrastructure as a service offerings and pricing of Amazon Web Services] as an example of a possible provider of data center infrastructure the optimal server package for the current dimensioning of the E-Compass Data Mining Services (including some scaling reserves) would be an M4 instance with a sufficient number of CPU/Cores and fairly large RAM. The m4.10xlarge instance with 160 GB of memory, 40 vCPUs on a 64-bit platform would arise [https://aws.amazon.com/ec2/pricing/?nc1=h_ls costs of approximately 1.000€ per month] inclduding the complete data center operation with all additional cost, such as e.g. energy consumption and maintenance.
		+
		+	Given that the system dealt with 10 concurrent E-Shops over the testing period without reaching the scaling limits this would make a maximum of 100€ per E-Shop per month on the infrastructure side.
		+
		+	In order to come to the full cost of operation the maintenance cost on the software / service side as well as the cost for helpdesk operation (in order to help E-Shop owners with configuration and bugfixing) additional personal costs have to be considered. In order to keep these costs in a reasonable range a 9 to 5 support is proposed with 2 employees (redundancy for illness and vacation) with an average monthly income of 3.000€. The cost for the employer are estimated with 4.500€ per month for each employee which makes a cost of 9.000€ per month. Depending on the wage level in different countries the actual costs may vary significantly!
		+
		+	Making this estimate the costs for support and software maintenance are in a completele different dimension than the infrastructure cost (including infrastructure maintenance). In order to get them to a comparable level the infrastructure could be scaled up to cope with 100 E-Shops instead of 10 E-Shops. From the experience of the testing phase the software maintenance effort and user support should allow for the support of 100 E-Shops with the support personell in the above estimate.
		+
		+	In conclusion this would add up to a total cost of approximately 200€ per E-Shop and month including the complete infrastructure, infrastructure maintenance, software maintenance and user (i.e. E-Shop owner) support.

@@ Line 2: / Line 2: @@
 For the future owner of the E-Compass Data Mining Services it is essential to be aware of the operational as well as the migration cost.
-As the migration of the services to a new data center essentially requires the deployment of a couple of virtual machines images which are available in the OVA (open virtualization archive) format and only minor adjustments by editing the service location URIs and running the available service configuation scripts the migrational cost can be neglected in comparison to the operational cost - as well in terms of finances as in terms of the time required. The complete migration of the services should be done within one or two days.
+As the migration of the services to a new data center essentially requires the deployment of a couple of virtual machines images which are available in the OVA (open virtualization archive) format and only minor adjustments by editing the service location URIs and running the available service configuration scripts the migrational cost can be neglected in comparison to the operational cost - as well in terms of finances as in terms of the time required. The complete migration of the services should be done within one or two days.
-This leaves the us with the estimation of the operational cost. In order to estimate this we need the following:
+This leaves us with the estimation of the operational cost. In order to estimate this we need the following:
 '''Virtual Machine Specification for all service modules'''

@@ Line 60: / Line 60: @@
 | Limit HDD || not in sight || 60 E-Shops with data for 2 months || 100 E-Shops with data for 2 months || not in sight || not in sight || 3000 rules / limit of users depends on user behaviour
 |-
-| Limit Network	if there is a limit at al it is the network, unless the communication is redesigned and all service modules write to a distributed database instead of communicating via REST || not in sight || not in sight || not in sight || not in sight || not in sight
+| Limit Network || if there is a limit at al it is the network, unless the communication is redesigned and all service modules write to a distributed database instead of communicating via REST || not in sight || not in sight || not in sight || not in sight || not in sight
 |-
 | Necessary Actions for Scaling || eventually changing the communication between ECC and service other modules || more Storage || more Storage || none || more CPU, RAM & Parallelization || more CPU, RAM & Parallelization
-|-
+|}

@@ Line 54: / Line 54: @@
 | Scales with || E-Shops || Users || Users || Competitors monitored || Competitors monitored || Number of rules and total number of users
 |-
+| Limit Processors / Cores || not in sight || not in sight || not in sight || not in sight || 10 Competitors || 3000 rules / limit of users depends on user behaviour
 |-
+| Limit RAM || not in sight || ? || ? || not in sight || 40 Competitors || 3000 rules / limit of users depends on user behaviour
 |-
+| Limit HDD || not in sight || 60 E-Shops with data for 2 months || 100 E-Shops with data for 2 months || not in sight || not in sight || 3000 rules / limit of users depends on user behaviour
 |-
+| Limit Network	if there is a limit at al it is the network, unless the communication is redesigned and all service modules write to a distributed database instead of communicating via REST || not in sight || not in sight || not in sight || not in sight || not in sight
 |-
+| Necessary Actions for Scaling || eventually changing the communication between ECC and service other modules || more Storage || more Storage || none || more CPU, RAM & Parallelization || more CPU, RAM & Parallelization
 |-
+|
 |-
+|