||For most companies, the challenge is getting relevant, timely customer information to employee desktops. For instance, a company might want its salespeople to have access to the latest profitability and product usage statistics about its customers.
The sources of this info are transactional applications (think check-out registers at the supermarket), which are hosted on bulletproof platforms like IBM mainframes. High security, reliability, and uptime.
However, the data is formatted for performance and reliability, not for querying. It's extremely difficult to read and process the large number of transaction records necessary to do the calculations to come up with, say, the total revenue for a store in a month. It's also very slow.
In contrast, relational databases are built for querying (they're not so good for transactional systems). Hence, ETL.
So for most companies, ETL is the process of extracting transaction data from a platform like an IBM mainframe, reformatting it into a query-friendly relational data model, and loading it into a data warehouse to support reporting and business intelligence.
Seems simple enough. The biggest challenge for companies is performance. I used to work for a large financial institution that wanted near-real time customer statistics for its sales force. It wasn't feasible to tie the reporting directly into the transaction platform, so we had to resort to daily updates. End of business was 6:00 PM and beginning of business was 8:00 PM. However, the entire ETL cycle took 18 hours. There wasn't enough time in a day to do the ETL.
Enter the ETL expert. With Ab Initio, we cut the load time to 4 hours. Tools like Ab Initio give you a great amount of control over exactly how records are read and written. Anyone who has worked with SQL knows that SQL doesn't give you that control. The goal is to read every input record only once, so you don't have to do multiple passes over the same data to do all of your calculations. (I may be oversimplifying; I'm an application owner and analyst, not an engineer.)