Real-time Data Replication — Experience the power of Oracle GoldenGate
Announcing a new hands-on LiveLab to discover the features of Oracle GoldenGate 21c
Oracle LiveLabs allow you to experience the tools and technologies that Oracle offers in a very easy and simple way, using pre-configured environments that allows you to focus on the functionality, not the infrastructure.
I’m happy to introduce you to the new hands-on lab covering the Oracle GoldenGate 21c technology leveraging the microservices architecture, showing how you can integrate the data between different types of technologies. In this lab I’m showcasing the data flow from an Oracle Database 21c to an Apache Kafka stream, but Oracle GoldenGate allows to apply the same logic to a very wide range of systems.
The lab combines all the required components in a single VM, which allows you to get down to the actual interesting parts of the replication immediately, without having to worry about installing software. Using the “Green Button” to run on the LiveLabs Sandbox, your environment is automatically provisioned for you.
The basic setup of the lab is represented in the below drawing :
In this lab all the components are combined on a single machine, but in your actual setup systems can be distributed across various locations. The Oracle GoldenGate Microservices Architecture (MA) allows the use of the Distribution / Receive service to integrate systems that are physically far apart, using the same setup logic as in the lab.
Getting started
As already mentioned, you can use the green “Run on LiveLabs Sandbox” button, which will provision an environment for you. The only disadvantage is that this environment is limited in time to approximately 3 hours.
If you want to have a more permanent environment to first run through the lab and then continue to experiment, you can choose the “Run on your Tenancy” button. In this case you need to do a little bit more configuration: download a terraform script, adapt it to point to the OCI compartment of your choice, and run the script using Resource Manager.
Discovering the environment
Once your environment is provisioned, you can access it simply through a VNC viewer, so nothing more than a browser is required on your local machine.
Alternatively you can also access the different components of the environment from your local machine: a browser to access the various consoles of Oracle GoldenGate (starting with the Admin server on port 10000), and your preferred remote shell tool to open the command line: ssh for Mac, PuTTY for Windows.
With the microservices architecture of Oracle GoldenGate 21c, you can access all the different services independently. The easiest starting point is the Oracle GoldenGate Service Manager, which typically runs on port 10000. Here you can see the various deployment types that are installed in your environment, for example GG for Oracle, GG for MySql, and GG for Big Data, and for each deployment you can see 4 services: Admin, Distribution, Receiver and Performance Metrics:
The next step is to look at the Oracle Database and the Apache Kafka stream that are available. This is done through the console, and connecting to the Oracle Database with sqlplus, and listing the available Kafka topics.
Configuring the replication
Setting up data replication is straight-forward. To begin you’ll set up the connection to the source database, and add a TRANDATA command to the database to acquire the transaction records in the database.
Next you’ll dive into the Oracle GoldenGate services to configure the Extract process from the source database, where you can configure which tables are going to be extracted.
After that you’ll configure the Distribution service to send the captured transactions to a remote system. Although in this lab that remote system is actually running in the same VM, the receiving system could be on the other side of the world.
After verifying the Receive service of the Oracle GoldenGate Big Data deployment, we’ll configure the Replicat service that will insert the transactions into Apache Kafka.
And that’s all there is to it! You can read up on the documentation of the various services to better understand all the different configuration options that are possible for the extraction, the distribution and the replication.
Running a load on the database
To ensure that everything is working and to understand the logging and tracing utilities of the replication environment, we’ll run a load on the Oracle Database and check the records are indeed inserted into the Kafka stream we selected.
We’ll use the Performance Metrics service of the 2 Oracle GoldenGate deployments we used, and we’ll also use the terminal interface to read the Kafka stream and see the records flowing in.
Below you see an example of the process performance of the Kafka Replicat service:
Conclusions
You’ve just configured the real-time replication of data between two different technologies, across potentially a long distance, allowing to have your business data in sync, ready to be used by your business users when they need it. We just scratched the surface of the endless possibilities of this solution, allowing a wide range of use cases ranging from high-availability, to Data Warehouse inflow, or operational data between operational systems. And we did not yet mention the options to do data quality verification, or Stream Data Analytics …. And of course with the highest level of security that is possible to ensure data protection in all situations.
Stay tuned for more articles on this topic!
