Very good article, Laurel.

To me, as one that has many years of experience with software development, it is obvious.

However, I also learned in my career that a good & precise design of your data model, using correct (and compact) storage of values, building the relationships correctly (in case of OO approach), using proper indices & fine tuning queries (in case of SQL approach) can make a huge difference between a "normal" and "high-end" systems.
All those become a major concern when we are talking about a very "high volume" systems that needs to process hundreds of millions of transaction per day.

Going to "compact" data model with a reliable code will save you also a lot in your capacity planning both in a cloud hosting or on premise.

DB tuning has many aspects:
1. When the DB is a SQL one (persistent classes) than as some said a "tune table might help, but this will help mostly with SQL queries, by optimizing the use of (correct) indices.
There are other SQL tools on the portal (query plan, sql index usage etc.) that might help to see if the indices are correct.

2. Using the correct data types for your data (e.g. using %Date data type to store dates in $horolog format is much more effective than a %String (YYYY-MM-DD) on large scaled databases, especially with indices where cache memory can not hold most of the DB or with systems with huge number of transactions).
BTW, this is true to both persistent classes (that you access qwith SQL) and "raw" globals that you access with COS (or other language).

3. In a large scald DBs if some of your queries are "heavy" - code optimization might also be a consideration (e.g. replacing SQL with COS that does direct $Order on the index globals).

Usually you do not mix between your "own" persistent (classes) data and Interoperability (aka Ensemble) request & respond messages (which are also persistent).

When you purge Ensemble data. the request/response messages are also being purged !

A best practice is to have for each component its own request/response messages. e.g.
PROD.BO.ComponentName.Msg.Request & PROD.BO.ComponentName.Msg.Response
where BO = business operation (so you could have BS for business service and BP for business process)
and ComponentName is the name of your components.
(sometimes few components, can share the same request & response messages, which is totally fine !)

Hello Subramaniyan,

If you can have a downtime for the DB, than you could write a script that dismount the DB, FTP it to another server and mount it again. This of course depends on the DB size and your network speed.

If a downtime is not possible, I would recommend doing a "hot backup" then copy (or FTP) it to another server and restore it. Another option is to use "external backup" with using a "freeze" & "Thaw" to ensure data integrity.

Further information:

https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls...

Hello,

I assume you are looking for something like CDC (capture data changes) for .

The basic idea is to programmatically read journal files record by record and analyze the SET/KILL ones  (according to some dictionary you build to determine which globals or classes need the CDC capability). 

I have done something similar using the ^JRNUTIL

https://docs.intersystems.com/irislatest/csp/docbook/Doc.View.cls?KEY=GC...

There are few considerations here:

1. If you are using a failover (sync) mirror, you will need to that Ens.* data on the backup so a switch can be done seamlessly, and Ensemble production can start from the exact same point when failure occurred.

2. mapping any globals into a non-journal database will still have them on the journal when the process is "in transaction". Ensemble production is writing everything "in-transaction"

3. Mapping Ens.* to another DB is good idea if you have several disks, and you want to improve I/O on your system (specifically within cloud VMs that usually have a bit slower disks than on-premise). Another option is to let the system purge Ensemble data regularly and maybe keep less days of history.

4. Mirror journals are regularly being purged after all sync/a-sync mirror members got them.

In my opinion, it is much better & faster to store binary files out of the database. I have an application with hundreds of thousands of images. To get a faster access on a Windows O/S they are stored in a YYMM folders (to prevent having too many files in 1 folder that might slow the access) while the file path & file name are stored of course inside the database for quick access (using indices). As those images are being read a lot of times, I did not want to "waste" the "cache buffers" on those readings, hence storing them outside the database was the perfect solution.

Thank you Robert,

Excellent article.
In fact we use this approach when we need to gain speed for queries that run on hundreds of millions records indexes and need to check few items only, so we save checking the "base" class by using [ DATA ... ]  

In addition, to gain more speed, which is so essential in huge queries, we use the (good, old) $order on the index global itself to gain more speed. This is much faster than normal SQL. 

Hello,

I recommend either ways:

1. copy the *.DAT file to new server: for this you need to dismount the DB on source before copy (so you have downtime), mount it and do a merge between 2 DB/Namespaces.

2. Do a (hot) backup for the DB that has the table  = NO downtime (!) - then copy the *.BCK file to other server, restore to a NEW DB and merge.
This is slower - but downtime is 0