15 Sep 2015 Oracle BI Cloud Service Part II: Loading Data

In the first post of the Oracle BI Cloud Service (BICS) series we reviewed some of its key features. In the following posts, we will explain how the service actually works, starting with the data provisioning. All the data used in the BI Cloud Service has to be stored in the Database Schema Service or Database as a Service, so uploading your data to the cloud is the first step towards a cloud OBI. There are several ways to do it, depending on your specific requirements:

Data Loader

The default method of uploading your data to the cloud is by using the built-in Data Loader accessible from the BICS homepage.

Figure 1: Accessing the Data Loader tool (from Oracle BI Cloud Service – A First look )

Data Loader is an easy-to-use browser-based application that allows you to upload data from files (txt, csv, xsl, xlsx) with a maximum of 500,000 rows and formatted as number, date, or character (for the moment only UTF-8 encoding is supported). There is also the possibility to perform simple transformations such as Uppercase, Lowercase, Trim Spaces, and Format as a Number. You can use Data Loader to perform full manual refresh, incremental data loadings, upsert (update/insert), add data to the model, view the load history and correct errors.

SQL Developer

Those who have worked with other Oracle Database products will already know SQL Developer. Now the application features a new cloud connection that you can configure to load data from files (with no row limit) and relational sources to your cloud service. It runs on your local machine, so you need to download it from the OTN downloads webpage and install it locally. Once configured, SQL Developer connects to the cloud service through a set of RESTful web service calls and can be used to load data using SFTP (SQL*Loader utility). You can also create “carts” with multiple objects to be uploaded, and schedule incremental loads.

Figure 2: Loading carts (from Loading Relational Tables Using SQL Developer)

BI Cloud Service (BICS) Data Sync

Data Sync is another application for loading your data to the cloud service and available for download on the OTN downloads webpage. It is a wizard-driven tool supporting the load of data from CSV files and relational sources into the Database Schema Service. It also supports the load and merge of data from different sources: DB2, SQL Server, MySQL, Teradata, and TimesTen, apart from Oracle relational sources. Use BICS Data Sync to perform incremental data loads, rolling deletes, and loads according to appended load strategies. You can also schedule and monitor data loads (called “jobs”). Be aware however that Data Sync is not officially released and Oracle does not support it and does not guarantee that it will be supported in future releases (future upgrades are not guaranteed either), so care should be taken when using it.

Figure 3: BICS Data Sync (from Loading On-Premises Relational Sources and Files to the Cloud Using Oracle BI Cloud Service Data Sync)

BICS & Database Schema Service REST APIs

You can also use the BI Cloud Service and Database Schema Service REST APIs to define a customized API and programmatically load data into the Database Schema Service. With the BICS REST API, you can update statistics, drop or create indices on tables, insert, update, upsert, and delete records. You can also combine the API with any other application and script in any programming language, offering endless possibilities such as invoking applications and scripts with an on-premises scheduler of integrating the load process with ETL tools. Similarly, with the Database Schema REST API you can call out SQL queries to read data and return results, call PL/SQL scripts to read, write, modify or delete data, and define your own customized API to be invoked from your on-premises environment. Moreover, using the APEX_WEB_SERVICE package within a PL/SQL block, you can invoke any REST/SOAP API supported by cloud applications, and retrieve data from external systems.

PL/SQL Scripts

Finally, you can use PL/SQL scripts to load data from external, generic web services. Scripts can be created using SQL Workshop, a robust browser-based tool part of APEX that allows you to create and execute SQL queries and database procedures against objects in the Database Schema Service.   Check out part III of the BICS series in which we will cover how you can model your data in the cloud according to your business requirements.