Rapid data Warehouse design
Data warehouses and even data marts can be expensive, complex projects. They are not projects to start lightly, and they are not projects that you want to launch without doing some solid planning.
But there is a way to get a handle on the tricky parts of your data warehouse scope, and to reduce your projects overall cost.
The major cost component of any data warehouse project is the Extract Transform and Load (ETL) development. Obviously every project is slightly different, but in my experience ETL will often make up in the order of 70% of the development cost. One of the drivers of this cost is the relatively high priced ETL development resources required. In the markets where I've hired resources, an ETL developer will often demand a 30-40% higher hourly rate than a business intelligence report writer, for example.
Making ETL prototypes will give you insights that can reduce cost by shortening the ETL development process and making the optimum use of those highly talented and expensive ETL resources.
What affects the cost and complexity of ETL jobs?
For any given scope, the following will have a large impact on the number and complexity of ETL jobs and therefore their cost.
-
The number of different data sources involved.
The consistency in terms of master data definitions between systems.
The level of data quality in the systems.
Ideally, you want to get a good handle on these three things before you hire all the ETL developers, and be confident that you are going to satisfy the users needs before millions of dollars are spent on Extract Transform and Load (ETL) jobs and business intelligence reports.
One part of the preparation needed to do this can be the creation of a proof of concept or mockup of key parts of the data warehouse ETL deliverable.
Now, there are mockups, there are prototypes, and there are "first versions". The the most effective approach is to create a mockup or prototype that;
Goes just deep enough into the data to: Establish all data sources that will be required Gives a high level audit of their master data and data quality Provides enough output that: End users can be supplied with example reports or cubes to get hands on The functional scope can be locked down with confidence on all sides.You might also like
|
Anker® 40W 5-Port Family-Sized Desktop USB Charger with PowerIQ™ Technology for iPhone 5s 5c 5; iPad Air mini; Galaxy S5 S4; Note 3 2; the new HTC One (M8); Nexus and More (White) Wireless (Fantasia Trading LLC)
|
The Guinness World Record for the Largest Data Warehouse: A Q&A with Tom .. — B-EYE-Network
Business unIntelligence—Insight and Innovation Beyond Analytics and Big Data Summary Is there still a need for the data warehouse? In this excerpt from his new book, Barry Devlin looks at why the data warehouse can no longer retain its old role of ..
|
Anker® 2nd Gen Astro Mini 3200mAh Ultra-Compact Portable Charger Lipstick-Sized External Battery Power Bank with PowerIQ™ Technology for iPhone 6 5s 5c 5 4S, Galaxy S5 S4 S3 Note 3, 4, Nexus 4, HTC One M8, Nokia Lumia 520, 1020 and Other Smartphones (Black) Wireless (Anker)
|
|
Optoma EH500 1080p 4700 Lumen Full 3D DLP Network Projector with HDMI CE (OPTOMA TECHNOLOGY)
|
|
ROBERTSON 2P20067 PSP242TRMVW/S Quik-Pak of 10 eBallast, Program Start, 1 or 2 Lamp, 4 Pin CFL (CFTR42W/GX24q), HPF, 120-277Vac. Home Improvement (Robertson Worldwide)
|
|
Anker® 2nd Gen Astro 6000mAh External Battery Pack with PowerIQ™ Technology 2A Output Portable USB Charger Power Bank for Smart Devices (Green) CE (Anker)
|
Q&A
Data mart usually can be constructed more rapidly and at lower cost than a data warehouse because
a data mart typically focuses on a single subject area or line of business
Is OLTP database design optimal for Data Warehouse?
Online Transaction Processing (OLTP) is exactly the opposite of Data Warehousing. The first is heavily read-write based, the second is primarily large-query based.
