Rapid data Warehouse design

ETL-Cost-vs-Other-Data-Warehouse-CostData warehouses and even data marts can be expensive, complex projects. They are not projects to start lightly, and they are not projects that you want to launch without doing some solid planning.

But there is a way to get a handle on the tricky parts of your data warehouse scope, and to reduce your projects overall cost.

The major cost component of any data warehouse project is the Extract Transform and Load (ETL) development. Obviously every project is slightly different, but in my experience ETL will often make up in the order of 70% of the development cost. One of the drivers of this cost is the relatively high priced ETL development resources required. In the markets where I've hired resources, an ETL developer will often demand a 30-40% higher hourly rate than a business intelligence report writer, for example.

Making ETL prototypes will give you insights that can reduce cost by shortening the ETL development process and making the optimum use of those highly talented and expensive ETL resources.

What affects the cost and complexity of ETL jobs?

For any given scope, the following will have a large impact on the number and complexity of ETL jobs and therefore their cost.

    The number of different data sources involved. The consistency in terms of master data definitions between systems. The level of data quality in the systems.

Ideally, you want to get a good handle on these three things before you hire all the ETL developers, and be confident that you are going to satisfy the users needs before millions of dollars are spent on Extract Transform and Load (ETL) jobs and business intelligence reports.

One part of the preparation needed to do this can be the creation of a proof of concept or mockup of key parts of the data warehouse ETL deliverable.

Now, there are mockups, there are prototypes, and there are "first versions". The the most effective approach is to create a mockup or prototype that;

Goes just deep enough into the data to: Establish all data sources that will be required Gives a high level audit of their master data and data quality Provides enough output that: End users can be supplied with example reports or cubes to get hands on The functional scope can be locked down with confidence on all sides.

You might also like

SQL Free Training - Designing a Data Warehouse
SQL Free Training - Designing a Data Warehouse
Data Warehouse Design - Healthcare
Data Warehouse Design - Healthcare
Fantasia Trading LLC Anker® 40W 5-Port Family-Sized Desktop USB Charger with PowerIQ™ Technology for iPhone 5s 5c 5; iPad Air mini; Galaxy S5 S4; Note 3 2; the new HTC One (M8); Nexus and More (White)
Wireless (Fantasia Trading LLC)
  • PowerIQ technology maximizes compatibility with Android, Apple and other devices by enabling charging speeds of up to 8 amps among 5 ports or 2.4 amps per port.
  • The size of a deck of playing cards (3.6 × 2.3 × 1.0in / 9.1 × 5.8 × 2.5cm) and unibody design for added portability.
  • Industrial-grade materials and premium circuitry ensure lifetime reliability.
  • 100-240 volt input ideal for international travel.
  • Package contents: Anker® 40W 5-Port Wall Charger, 5ft / 150cm detachable power cord, welcome guide, 18-month hassle-free warranty.

The Guinness World Record for the Largest Data Warehouse: A Q&A with Tom ..  — B-EYE-Network
Business unIntelligence—Insight and Innovation Beyond Analytics and Big Data Summary Is there still a need for the data warehouse? In this excerpt from his new book, Barry Devlin looks at why the data warehouse can no longer retain its old role of ..

Anker Anker® 2nd Gen Astro Mini 3200mAh Ultra-Compact Portable Charger Lipstick-Sized External Battery Power Bank with PowerIQ™ Technology for iPhone 6 5s 5c 5 4S, Galaxy S5 S4 S3 Note 3, 4, Nexus 4, HTC One M8, Nokia Lumia 520, 1020 and Other Smartphones (Black)
Wireless (Anker)
  • The Anker® Advantage: Join the 6 million+ powered by Amazon s #1 charging brand.
  • Exclusive PowerIQ™ Technology: Detects your device to deliver its fastest possible charge speed up to 1 amp.
  • High Capacity: Add one and a half charges (over 7 hours talktime) to an iPhone 5s, almost one full charge to a Galaxy S5 or one full charge (70+ hours of audio playback)...
  • Ultra Compact: Our original lipstick-shaped aluminum design (3.5 × 0.9 × 0.9in, 2.7oz). Recharges in 3-4 hours with a 0.8 amp adapter (not included) and the included...
  • What You Get: Anker® 2nd Generation Astro Mini 3200mAh External Battery, Micro USB cable, travel pouch, welcome guide, our fan-favorite 18-month warranty and friendly...
OPTOMA TECHNOLOGY Optoma EH500 1080p 4700 Lumen Full 3D DLP Network Projector with HDMI
  • Native 1080p high-definition resolution for crystal clear presentations
  • 4700 lumens bright output makes it ideal for use in large screen applications
  • Latest image controller technology and Full 3D compatibility for the most amazing immersive experiencer
  • Over the network management and control capability makes it the perfect projector for large installations
Robertson Worldwide ROBERTSON 2P20067 PSP242TRMVW/S Quik-Pak of 10 eBallast, Program Start, 1 or 2 Lamp, 4 Pin CFL (CFTR42W/GX24q), HPF, 120-277Vac.
Home Improvement (Robertson Worldwide)
  • Equivalent to: Fulham RH4-UNV-242-K, RH4-UNV-242-C, Keystone KTEB-242T-UV-TP-PSD-NL
  • Program Start, 1 or 2 Lamp, 4 Pin CFL (CFQ42W/GX24q), HPF,120-277Vac., -30C Min Start Temp
  • Also offered as a single quantity by ordering the ROBERTSON 3P20067 PSP242TRMVW/S
  • Take advantage of ROBERTSON s 60 years of electronic design and quality manufacturing experience.
  • Commerical grade. FIVE year warranty. Go to ROBERTSONDirect.com for complete technical data.
Anker Anker® 2nd Gen Astro 6000mAh External Battery Pack with PowerIQ™ Technology 2A Output Portable USB Charger Power Bank for Smart Devices (Green)
CE (Anker)
  • - PowerIQ™ technology maximizes compatibility with Android, Apple or other devices by enabling charge speeds of up to 2 amps.
  • - 6 mAh capacity and an efficiency rate 10% above previous generation Astros and standard alternatives charges the iPhone 3 times and the Galaxy S4 almost twice.
  • - Shake to start charge or display remaining power level on the ultra-low consumption 10-LED power meter (1 shakes consume less than 1%).
  • Sturdy construction and matte finish withstand bumps and drops. Grade A cells and premium chips ensure reliability. Input: 5V / 1.2A; Use a 1.2A adapter (not included)...
  • - Package contents: Anker® 2nd Generation Astro External Battery, micro USB cable, welcome guide.


Data mart usually can be constructed more rapidly and at lower cost than a data warehouse because

a data mart typically focuses on a single subject area or line of business

Is OLTP database design optimal for Data Warehouse?

Online Transaction Processing (OLTP) is exactly the opposite of Data Warehousing. The first is heavily read-write based, the second is primarily large-query based.