Early Access Program for new Table Partitioning feature

InterSystems Official

#Early Access Program (EAP) #SQL #InterSystems IRIS #InterSystems Official

Hi,

We’re launching an Early Access Program for an upcoming Table Partitioning feature that will help IRIS customers manage very large tables, and distribute row data and associated indices across databases and storage tiers. Table Partitioning cuts deep into the core of IRIS relational data management, so we want to make sure we get things right through working with a few engaged customers who can provide feedback on the initial deliverables, and fine-tune as needed.

If you are working with very large relational datasets, looking for more operational efficiency, and willing to roll up your sleeves to test the new capability, please register at https://www.intersystems.com/early-access-program/. You will receive a welcome email that directs them to the evaluation portal, where you'll find a temporary development license, recent kits and container images that contain the new capability, and a tutorial to get you started.

We intend to reach out to registered participants periodically, when we have relevant updates to share, and of course are available to assist your explorations directly, as needed.

Thanks,
benjamin

Stephen Canzano · Mar 12

will this implementation be employed for the HealthShare ATNA tables as these have billions of rows at customer sites

1 0

Yes, the ATNA tables are definitely on our joint HS/IRIS radar and have been a motivating use case. Right now we're making the basic functionality available on IRIS through this EAP, and are simultaneously working with the HS team to look at the best way to adopt this. Migrating existing data is a non-trivial scenario though.

0 0

Herman Slagman · Mar 15

Is this only for very large tables?
Because we have logging that nobody looks at after a month of three, but we have to keep them for five years.

Dmitry Maslennikov · Mar 16

While we moved our application to AWS, and we have some data, which we need to keep for a while. With this feature, we can move old data to a cheaper storage.

I believe the ability to move to a cheaper storage is mostly the case. Another option is that some table is too big, and someone would like to split it to be stored in multiple different databases, together with the indexes.

Yuri Marx · Mar 16

If this functionality allows partitioning by business criteria, such as date of sale, or store that sold it, it is possible to obtain greater performance in queries that use the same criteria.

The value should be static, at the time of creating record on table

For dates it uses range feature, so, you can split it by months, year (I suppose)

and move the whole bunch of records including indexes to another databse

Benjamin De Boe · Mar 17

yes, when your queries include criteria that correspond to the partition key, we'll ignore partitions we know cannot contain any data satisfying those criteria. e.g. if you partition by year and your query is looking at the current month, we can safely skip any partitions for 2024 or before.

Herman Slagman · Mar 17

Does it work in a mirror constellation?

yes, though the work to support applying updated mappings (as part of a MOVE PARTITION command) to all mirror members is still ongoing, so that's not something you can validate with the software currently published on the EAP portal.

Harshitha Balak... · Mar 17

Looking forward to exploring this! Are there plans to integrate partitioning features with existing IRIS analytics tools?

Benjamin De Boe · Mar 18

Any tools that use SQL to access partitioned tables will just work, as from the SQL query perspective there is no change. This includes Adaptive Analytics, InterSystems Reports, and any third-party BI tools. Also, IRIS BI cubes can use partitioned tables as their source class.

We currently have no plans to support partitioning of IRIS BI cubes themselves, as they have their own bucketing structure and less commonly have both hot and cold data, so some of the motivations for table partitioning don't apply.