For developers and website operators

Brokredge-Engine, explained.

Brokredge-Engine is the automated market intelligence engine of Brokredge, a product for Dubai real-estate brokerages. This page says what it is, what it reads, how it behaves, and how to have it blocked, slowed down or corrected. It respects robots.txt, a site's terms of use and any restriction an operator asks for.

Identity and purpose

A reader of developers' own pages, so a broker's answer can name its source.

Brokredge keeps a market reference of Dubai's off-plan projects for the brokerages that use the product. When a broker asks about a project, the answer names where each fact came from — the developer's brochure, the developer's page, the date it was read. The engine is the part that retrieves publicly accessible real-estate information from sources where automated access is permitted: a developer's own project pages, press releases and the documents they link to. It does not publish, resell or republish anything it reads, and it is in a pilot (October 2026): it reads only sites whose terms of use a person at Brokredge has read and recorded.

What it reads, and what it respects

robots.txt, first

Before anything else the engine reads your robots.txt and keeps to it under the token Brokredge-Engine — or under your rules for every crawler when it is not named. A page it may not read is never requested. The file is re-read every day.

Your terms of use

A person at Brokredge reads a site's terms before the site is enabled and records the verdict. A site whose terms forbid automated access is not read at all.

The sitemaps you publish

The sitemaps your robots.txt lists, or the site's sitemap.xml, tell it which project and press pages exist. It does not crawl a site looking for them.

Project and press pages

The pages that describe a project — handover, payment plan, unit types, the location you publish — and press releases naming projects. Nothing behind a login, a form or a CAPTCHA is read around.

Documents, one at a time, by a person

A brochure, floor plan or fact sheet a page links to is only queued. A person at Brokredge approves each file before it is fetched, and it is never attached to a client message without the developer's permission on record.

How it behaves

Four habits, each one built and tested.

Conservative request rates

At most one request every five seconds to a site, and more slowly where your Crawl-delay asks for it. After a 429 or 503 it backs off for a day; after three refusals in a row it stops asking that site for a day.

Openly identified

Every request carries the user agent Brokredge-Engine/0.9 (+https://www.brokredge.com/engine) and only standard request headers — no cookies, no credentials, nothing of any brokerage's.

Narrowly scoped

Only the developer's own hosts, approved by a person after reading the site's terms. At most three redirects, on the same site. Never a property portal, an aggregator or another broker's site.

On the record

Every request is logged with its purpose, the page it served, the answer and the bytes. Phone numbers and email addresses that appear on a page are removed before anything is stored. Nothing it reads is published, resold or republished.

For website operators

Block it, slow it down, or correct it.

To stop the engine without asking anyone, add this to your robots.txt; it stops at its next daily check:

User-agent: Brokredge-Engine
Disallow: /

Or use the form. A person reads every request and replies to the address you give: to block your site, to read it more slowly, to correct something Brokredge holds from it, or to answer a question. A request to stop is applied before the reply.