SOLUTIONS PARTNERS BLOGS CONTACT

 Introduction
 Concepts
   Sync Agents
   Jobs Readers & Writers
   Synthetic CDC
 Security
 Connections 
     Overview
     Databases:
      SQL Server
      MySQL
      Oracle
      Teradata
      ODBC
     Drives/FTP:
      Local Path/UNC
      AWS S3 Bucket
      Azure Container/ADLS
      Google Drive
      FTP
     HTTP/S:
      HTTP/S Connection
      Rate Limiting
      OAuth Tokens
     Messaging:
      AWS SQS
      Azure Event Hub
      Azure Service Bus
      Google PubSub
      Kafka
      MSMQ
      RabbitMQ
     Big Data
      Google BigQuery
      Azure CosmosDB
 Jobs
     High Watermark Values
     Dynamic Jobs
     Job Readers 
      Database Reader
      Drive Reader
      HTTP/S Reader
      Messaging Consumer
      Big Data

     Job Writers 
      Create From Reader
      Database Target
      Drive Target
      HTTP/S Target
      Messaging Producer
      Big Data Target

     Triggers
     Data Pipelines
 DataZen Functions
 Resync & Replay
 Sync Agent API

Sync Agents

DataZen Sync Agents are designed to read and write from/to a wide range of systems, including messaging platforms, files, relational databases, and generic drivers (such as ODBC drivers). In addition, Sync Agents can read and write from/to thousands of HTTP/S REST data sources, some of which are pre-configured for quick configuration.

Source & Targets




Read/Write from any source & target
Maintains high watermarks
Handles complex data shaping operations

Job Types




Job Reader: Saves Data in Change Logs
Direct Job: Immediately Push Changes
Job Writer: Reuse Existing Change Logs

Processing Pipeline




Read, Aggregate, Enrich
Partition & Push Changes
Replay Change Logs as Needed

A Data Sync Agent is needed to read from and/or write to a system. It is best to install a Data Sync Agent close (for optimum network bandwidth reasons) to the systems that the agent will read from.

One or More Agents

For example, if you have a source system (ex: an Oracle database) in New York, and want to replicate changes to a Kafka endpoint in Atlanta, you could install two agents: one in each location. However, in most cases, you would use a single agent to perform both the read and write operation for most implementations; the agent would be installed close to the source system. If you have a large number of locations, you would likely install one agent per location. This can be useful for distributed sites that need to centralize data in the cloud.

Resilience

By design, the Sync Agent is highly resilient. In the target system is not available for a period of time, it will continue to extract changes from the source system, and as soon as connectivity is restored to the target, it will replay the change logs that were missed, in sequence. As a result, Sync Agents can be installed in remote locations with sporadic or intermitent connectivity.

Components

The Data Sync Agent is made of a few components:

  • Job Reader - a job that reads data from a source system
  • Job Writer - a job that writes data to a single source system
  • CDC Engine - internal engine that automatically detects changes (Synthetic CDC)
  • Staging Store - stores configuration settings and state information of the source data
  • HTTP/S Listener - a controller listener that DataZen Manager uses to manage the agent
  • Sync File - a Universal Change Log that can be copied and played back

Data Sync Agents require a connection string to a Staging database stored in SQL Server, as specified in the Installation instructions. Each agent is licensed separately.

Using LocalDB or SQL Server Express

You may use SQL Server Express or a LocalDB edition for the Staging Database. Here is an example of a LocalDB connection string for the Staging Database:

Server=(LocalDB)\MSSQLLocalDB;Integrated Security=true

LocalDB should not be used for production workloads. SQL Server Express edition is supported, but in some cases the use of SQL Server Standard edition may be required depending on how large the staging tables need to be to keep state information.





601 21st St Suite 300
Vero Beach, FL 32960
United States

(561) 921-8669
info@enzounified.com
terms of service
privacy policy

PRODUCTS

ENZO SERVER
ENZO DATAZEN

SOLUTIONS

SOLUTIONS OVERVIEW
LOW-CODE INTEGRATION
SHAREPOINT INTEGRATION
RABBITMQ INTEGRATION
HYBRID SHARDING
READ/WRITE PARQUET FILES
SQL SERVER HYBRID QUERIES

RESOURCES

ENZO AZURE VM
BLOGS & VIDEOS
IN THE NEWS
ENZO ADAPTERS
ONLINE DOCUMENTATION
TCO CALCULATOR

COMPANY

LEADERSHIP TEAM
PARTNERS


© 2023 - Enzo Unified