HTTP Connection: Rate Limiting

Most SaaS/Cloud services implement throttling conditions that limit your ability to query the endpoint too frequently. When creating an HTTP/S connection in DataZen, you can specify Rate Limiting options to avoid or minimize errors generated by throttling conditions.

Two rate limiting options are provided and be used together:

• Call Rate: limit the number of calls made within a time window
• KB Rate: limit the amount of data transmitted within a time window

When Allow Bursting is disabled, calls are spread out equally over the time window provided. For example, if 10 calls per minute are specified, a call to the service will be made every 6 seconds. However, if Allow Bursting is enabled, the calls are made as quickly as possible at the next time window, then paused if the rate limit is reached until the next time window starts. Using the same example, 10 calls would be made as quickly as possible, but if additional calls are required, the job will pause for up to 60 seconds before sending another 10 calls.

When a rate limit is detected for a job, a warning message will be displayed in the output window for awareness. The job will pause to remain within the specified rate limit.

Rate limiting applies to each individual jobs. In other words, if a call rate limit is specified for a Twitter connection, and two separate jobs use the same connection created in DataZen, each job will be call-limited independently; the aggregate number of calls across the two jobs will not be rate-limited.