New post
Article
· Feb 3, 2016 2m read

Linux TZ environment variable not being set and the impact to Caché

#Best Practices #InterSystems Business Solutions and Architectures #System Administration #Caché

During recent large scale benchmarking activities, we were seeing excessive %sys CPU time that negatively impacted scaling of the application.

Problem

We have found that a lot of the time was spent in the localtime() system call due to the TZ environment variable not being set.  A simple test routine was created to confirm the observation, and the elapse time differences and CPU resources needed with TZ set versus TZ not set were astonishing.  It was discovered that the inherit use of stat() system calls to /etc/local_time from localtime() are very expensive when TZ is not set.

Recommendation

InterSystems highly recommends confirming in any Linux installation whether x86 or Linux on Power that the TZ environment variable is set appropriately for optimal performance.  For more details see: "man tzset".

The current Caché 2016.1 Field Test contains optimizations in regards to date and time functions, and initial testing indicates a profound improvement.  Below are sample outputs from testing the new internal function calls in a loop with Linux on Power in both cases with TZ *not* being set:

Before Caché 2016.1 FT:

real	0m22.60s


user	0m1.64s


sys	0m20.89s

With Caché 2016.1 FT:

real	0m0.40s
user	0m0.37s
sys	0m0.00s

Please post comments with any experiences you may have had with your application regarding TZ environment variable and also the impact of Caché  2016.1 Field Test has with or without TZ being set.

Discussion (5)2
Log in or sign up to continue
Alexey Maslov · Jun 17, 2016

Hello Mark,

As I had no idea which date and time functions can be affected with TZ setting, I'd tested some of them using a codelet:

s top=1000000 f fun="$h","$ztz","$zts","$zh" {s ts=$zh f i=1:1:top { s @("d="_fun) } w fun,?8,$fn($zh-ts/top,"",10),! } d $zf(-1,"echo $TZ")

My testing environment was:

%SYS>w $zv
Cache for UNIX (Red Hat Enterprise Linux for x86-64) 2015.1.2 (Build 607_0_15223) Thu Jul 16 2015 17:33:31 EDT
%SYS>!cat /etc/centos-release
CentOS release 6.6 (Final)

The results w/o TZ:

$h      0.0000023653
$ztz    0.0000009993
$zts    0.0000004639
$zh     0.0000003155

and with TZ:

$h      0.0000011856
$ztz    0.0000001379
$zts    0.0000004690
$zh     0.0000003189
Europe/Moscow
1 0
Alexey Maslov · Jun 17, 2016

It's easy to set TZ for a Linux/UNIX tty user, but what about an app that is running some flavor of client/server mode? In this case Cache process inherits its environment from a special kind of parent, usually from SuperServer^%SYS.SERVER. 
At the moment I have no idea how to set the environment for SuperServer. I've tried:
1) setting system-wide using /etc/profile.d/*.sh
2) setting system-wide using /etc/environment
3) setting for cacheusr user using his .bash_profile.
Running a sample below, I'm getting a nice picture on /dev/pts and opposite one on |TCP|1972. The results are added in comments. I used CacheActiveX.dll (%Service_Bindings) for client/server connection.

tz(fun)
 if $zversion["UNIX" {
   set f="echo $TZ" open f:("QR"):1 if '$test { write 0 quit "" } use f read tz close f
   set f="id" open f:("QR"):1 if '$test { write 0 quit "" } use f read id close f
 } else {
   set tz="", id=""
 }
 set fun=$get(fun,"$ztz") ;"$h"
 set top=1000000 set ts=$zhorolog for i=1:1:top { set @("d="_fun) } set res=$zutil(110)_" "_$principal_" "_id_" tz="_tz_" "_fun_" "_$fnumber($zhorolog-ts/top,"",10) ;d $zf(-1,"echo $TZ") 
 quit res
 ;
 ; tap01.sparm.com |TCP|1972|1311 uid=502(cacheusr) gid=503(cacheusr) groups=503(cacheusr) tz= $ztz 0.0000011239
 ; tap01.sparm.com /dev/pts/0 uid=504(alex) gid=503(cacheusr) groups=503(cacheusr),10(wheel),505(alex) tz=Europe/Moscow $ztz 0.0000001212
 ;
0 0
Alexey Maslov Jun 17, 2016 to Alexey Maslov

After 
# su cacheusr -
$ csession cache
QMS> w $$tz^ztest()
tap01.sparm.com /dev/pts/2 uid=502(cacheusr) gid=503(cacheusr) groups=503(cacheusr) tz= $ztz 0.0000008660

So the user type (=cacheusr) rather than process type seems to be a special case. After setting TZ manually I'm getting an expected responce:
tap01.sparm.com /dev/pts/2 uid=502(cacheusr) gid=503(cacheusr) groups=503(cacheusr) tz=Europe/Moscow $ztz 0.0000001271

but why cacheusr is not getting an environment from any standard place?

0 0
Mark Bolinsky Jun 17, 2016 to Alexey Maslov

setting the TZ environment variable needs to be done in the system-wide profile such as /etc/profile.  This should define it properly for you.  I would recommend a restart of Caché after setting it /etc/profile.  

Also the impact of the TZ environment variable not being set should be reduced (eliminated) with the current 2016.1+ releases where we have changed the way this operates.

Kind regards,
Mark B-

0 0
Alexey Maslov · Jun 21, 2016

It seems that the source of the problem is a method of [re]starting Cache.

When it is started from shell using `ccontrol start ` command, SuperServer (as well as its childs) recognizes TZ that's actual system-wide.
But when it is started using a service script `service ca_cache start`, TZ is not recognized. There is nothing special in my script, its start() function is implemented as a wrapper for just the same `ccontrol start ` command as in the first case. It seems that service scripts are started from some special environment where some environment variables are deliberately unset.

I fixed the issue by setting the correct TZ in /etc/environment file and including one line of code into the service script:

start() {
        echo "Starting ca_$prog:"

        [ -f /etc/environment ] && . /etc/environment && export TZ

        ccontrol start $prog quietly
 
0 0