New post
AllAccepted answers
Murray Oldfield · Jul 26, 2019

On Linux use Asynchronous IO (asyncio) for RANREAD testing.
asyncio  enables direct IO for database reads and writes which bypasses file cache at the OS and LVM layers 

NOTE: Because direct IO bypasses filesystem cache, OS file copy operations including Caché Online Backup will be VERY slow when direct IO is configured.


Add the following to [config] section of the cache.cpf and restart Caché/HealthShare/IRIS: 

wduseasyncio=1
 

It might be helpful if you just have a 15-minute pButtons to run while RANREAD runs to see operating system io stats, eg iostat.

From zn "%SYS"

%SYS>set rc=$$addprofile^pButtons("15_minute","15 minute only", "1", "900")

%SYS>d ^pButtons
Current log directory: /trak/backup/benchout/pButtonsOut/
Available profiles:
     1  12hours     - 12 hour run sampling every 10 seconds
     2  15_minute   - 15 minute only
     3  24hours     - 24 hour run sampling every 10 seconds
     4  30mins      - 30 minute run sampling every 1 second
     5  4hours      - 4 hour run sampling every 5 seconds
     6  8hours      - 8 hour run sampling every 10 seconds
     7  test        - A 5 minute TEST run sampling every 30 seconds

select profile number to run:

Murray Oldfield · Jun 12, 2019

Hi Rich, the short story is that I have fixed this an pushed to GitHub. A new container version will appear very soon.

The problem is caused by an unexpected date format in the "Profile" section of the pButtons HTML file.  Year format yy instead of expected yyyy. 

As a workaround, you can open the HTML file in a text editor and edit the date manually to be the expected format. eg

Profile run "24hours_5" started at 00:00:00 on Jun 08 19.

change to:

Profile run "24hours_5" started at 00:00:00 on Jun 08 2019.

Thanks for bringing this to my attention.!

Murray Oldfield · May 15, 2018

For newer storage, especially all flash, 10,000 iterations will be too quick, change this to 100,000 for sustained read activity -- it should be less than a minute on SSD storage for each step. For example using the above example;

for i in `seq 2 2 30`; do echo "do ##class(PerfTools.RanRead).Run(\"/db/RANREAD\",${i},100000)" | csession CACHEINSTNAME -U "%SYS"; done
Murray Oldfield · Mar 15, 2018

hmmm... it worked for me just now.... I did notice when I exported from the markdown editor I used it came across as "\<!--break--\> escaping the slash, but edit to "<!--break-->"  worked OK. on the post about minimum monitoring and alerting solution. To be honest I had not tried until you mentioned it. haha yes switch MD to WYSWIG is a big mistake :(

Murray Oldfield · Nov 8, 2017

Hi, I think you have got it. Mark and I are saying the same thing. Sometimes a picture helps. The following is based on some presentations I have seen from Intel and VMware. It shows what happens when multiple VMs are sharing a host, which is why we recommend a reserved instance. But this also illustrates hyper-threading in general.

So to recap; A hyper-thread is not a complete core. A core is a core is a core, hyper-threading does not create more physical cores. The following picture shows a workload going through 4 processing units... micro execution units in a processor. 

One VM is green a second is purple, white is idle. A lot of time is spent idle waiting for memory cache, IO, etc.

As you can see with hyper-threading on the right  you don’t suddenly get two processors, not 2x, and expectation is  ~10-30% improvement in processing time overall. 

The schedular can schedule processing when there idle time, but remember on a system under heavy load CPU utilisation will be saturated anyway so there are less white idle time.

Murray Oldfield · Oct 26, 2017

I also received feedback on this from a coworker who has been running workloads on AWS which I think is useful; 

 "I selected the large EC2 instances which used all the available vCPU on underlying processors.   ( AWS would reserve  a few vCPUs  for management functions.  )    Thus,  a AWS 40vCPU system is roughly  equivalent to 20 core with HT  bare metal system of the same clock speed. So right,  If you want the equivalent of 4 VMware cpus , allocate at least 8 vCPUs. Also note the  clock speeds of AWS are slower than what you are typically using.  [ie the servers I use in my on premises benchmarks]. You can  use the specint rate as the comparison." 

For more on the last comment and how to compare SPECint metrics see:

https://community.intersystems.com/post/intersystems-data-platforms-and-...

Murray Oldfield · Oct 25, 2017

Good question. InterSystems uses AWS for many different workloads and scenarios. There are so many types of server offered, it really is a case where you must test yourself.

As a starting point:

You really need understand just what AWS is giving you. For a list of instance types see the link below.  And note the small print; Each vCPU is a hyperthread of an Intel Xeon core except for T2 and m3.medium.

If you know your sizing on VMware or bare metal with hyperthreading enabled and you usually need 4 cores (with hyperthreading) - I would start with sizing for 8 EC2 vCPUs. Of course you will have to test this before going into production. It would be great if you came back to the community if you came back and commented after your testing.

AWS:

https://aws.amazon.com/ec2/instance-types/

For more general information I recommend Marks post on EC2 reference architecture.

https://community.intersystems.com/post/intersystems-technologies-amazon...

Mark also noted there is a difference between reserved instances and on-demand.  Reserved is just that – dedicated resources, so you won’t be contending with other VMs. If your requirement is consistent, and you want consistent performance look at the reserved instances.

Murray Oldfield · Mar 14, 2017

Hi, It would have been better to say "Database" could be different to "Journal".   SPBM can be different for all VMDKs (disks), but that doesn't mean it should be. As an example on a four node all flash cluster;

I am using just two storage policies for a production VM disks.

- Policy: VSAN default: for OS and backup/scratch. Failures to tolerate (FTT)=1. Disk stripes=1. Object space reservation=0.

- Policy: Production: for database and journals. FTT=1. Disk stripes=1. Object space reservation=100.

For performance on each production VM use separate PVSCI adapters for OS, journal, database, and backup/scratch.

 

For non-production VMs I have a policy that makes better use of available capacity, there is still HA in the storage:

- Policy: Non-Production: for all data disks. FTT method=RAID 5/6 (Erasure coding). FTT=1. Object space reservation=0.

Note. These values are not written in stone and will depend on your requirements. While you need to think about performance it should be great out of the box. What you must also consider is availability and capacity.

Murray Oldfield · Mar 5, 2017

To clarify and answer a question asked offline with an example; 

"Alternatively, you can enable OS-level authentication and create a Caché account for the OS user running the script."

Create a user for backup functions in the operating system named backup or similar, and add a user with same name in Caché.

Assign an appropriate role to the new Caché user based on your security requirements (for example you can test with %All role).

Then enable the system parameter to allow OS authentication.  Follow the steps in Configuring for Operating-System–based Authentication. (%Service_Terminal on Unix or %Service_Console for Windows).

The advantage of using a standard user name is you have a consistent approach for all instances.