Opened 12 years ago

Closed 11 years ago

#38 closed enhancement (fixed)

Massive export speedup

Reported by: Kris Deugau Owned by:
Priority: major Milestone: 1.2
Version: Keywords:
Cc:

Description

Instead of regenerating the entire on-disk data file straight from the database each time, export individual zones to separate files in a workspace, then concatenate them. Add a "modified" flag to the domain or revzone so we only have to walk through the details of zones that have been modified, and update those files - should give close to an order of magnitude or better speedup.

Observed on test copies of live data:

  • 3081 local forward zones
  • 2598 imported records from flatfiles in "main" zone
  • 271 imported revzones from flatfiles
  • 315 imported forward zones from autoaxfr flatfiles

Initial local forward zones only, ~43-65s
Add all current flatfiles, ~114-160s

Change History (4)

comment:1 by Kris Deugau, 12 years ago

(In [368]) /trunk

First stage implementation of export caching for performance. See #38.
A lot of the actual performance boost comes from indexes on a couple of
columns in WHERE clauses:

  • status column on domains and revzones
  • type, domain_id and rdns_id columns on records

Net speedup on ~100K records and ~3K zones was
~120s -> 20s (no cache) -> 4s (cache active)

comment:2 by Kris Deugau, 12 years ago

Milestone: 1.41.2
Summary: Possible massive export speedupMassive export speedup

comment:3 by Kris Deugau, 11 years ago

(In [521]) /trunk

Finally commit a fix for the caching system so that it really
exports things properly. See #38.

comment:4 by Kris Deugau, 11 years ago

Resolution: fixed
Status: newclosed

Resolving; numerous tweaks, enhancements, and "OMG-what-was-I-smoking" block rewrites have brought export time down to ~10s even if we set the flag to always refresh the cache.

Note: See TracTickets for help on using tickets.