New post
AllAccepted answers
Luca Ravazzolo · Jan 19, 2022

Thanks @Mark Bolinsky to highlight those important KPI and links

@David Foard you can automate and have it all displayed and be free to improve the dashboards with System Alerting and Monitoring or SAM that it's free in its community edition.

https://github.com/intersystems-community/Sam

https://docs.intersystems.com/components/csp/docbook/DocBook.UI.Page.cls...
 

Hope this helps

Luca Ravazzolo · Nov 22, 2021

Hi @Trevor Strong 

The container image simply installs the standard Apache package in the container and adds the CSP add-on.

For any update on the Apache web server we should all keep an eye on https://www.cvedetails.com/product/66/Apache-Http-Server.html?vendor_id=45 and consider patching/upgrading/re-building as necessary and according to the security policies and best practices of the organizations we work for.

HTH

Luca Ravazzolo · Oct 12, 2021

Hi Ben,

The short answer is yes, you are correct.

The longer one :-)
Stating the obvious, from a tool point of view, to be able to rollback operations means understanding the present state of an instance and of course have a record of all the previous states. In order to be able to do that one needs the concept of a "release state". As soon as you get into maintaining state you quickly escalate the complexity of a solution. See Terraform for example and ICM itself that supports the replication of its state via Consul.

There are tools like Helm, ArgoCd, etc. that help in that, however that is left to the user. Enhancing InterSystems IRIS is an option but that is not available now. At present we rely on a GitOps approach. 

GitOps is a paradigm that incorporates best practices applied to the application development workflow all the way to the operating infrastructure of a system.

Embracing GitOps give us some benefits like:

  • Deploying faster and more often (with a DB we could argue those adjectives, nevertheless we can still appreciate the benefits)
  • Easier and quicker error handling and recovery
  • Self documenting deployments
  • Increased developer productivity and an enhanced experience for teams
  • Greater visibility on the lifecycle of developed features

However, GitOps itself is not the delivery & deployment panacea of this complex area. GitOps has issues too. There are shortcomings when auto-scaling and dynamic resources are implemented; there is no standard for managing secrets; observability is immature; rollbacks don't have a standard practice, etc.

The powerful CRUD operations that we can run with the CPF merge feature adds to the complexity. A solution needs to be implemented that may leverage one or more tools that organizations use in their automated provisioning pipeline, just like you would do when embracing the GitOps paradigm.

I think there are two ways to solve our rollback issue, at present.
The first one would be a programmatic approach, maybe a diff operation on the git hash declarations of (last_op_def) vs (last_op_def  -  1)
If last_op_def contains a Create-resource I then need to rollback that with a Delete-resource or Modify-resource. And even in this simple case how do you determine that? Human intervention is probably needed.

The second option, simpler and safer, would be to simply re-run the container, the base state we know, and apply configuration settings #1 and #2 only.

There are probably other options involving verifying the CPF file. However, the present CPF file does not hold all of an instance settings.
There are also other issues to these type of automations, like: what if you want to rollback after the creation of a database and data was written to it?

It's complex.

Luca Ravazzolo · Oct 11, 2021

Hi Ben,

Thanks for the pertinent question.

First of all the CPF merge feature is available for containerized and traditional tarball-based installations and configuration. I want to call that out as at times that is unclear in people's mind.

The CPF merge feature allows you to define and apply a delta to your present instance configuration (initial from an install or otherwise from day2). As we know, an instance configuration is described in the Configuration Parameter File or CPF that show us (output mainly) the parameters with which the instance is running.

The CPF merge feature supports CRUD operations and is idempotent. That is useful for three reasons IMO

  1. I can Delete or Update resource settings from the outset (day1). There might be settings that are not desired in a new installation
  2. I can Update a single property from day2. All I would need to do is simply add the modification to the same declaration and stamping it as v2.0
  3. It maps directly to InterSystems IRIS internal operational classes so we know the API is robust

Therefore, if a specific property or resource is not desired anymore, your #3 in your example, by modifying the Web Application Security with an Update/Modify operation or deleting it completely, you would return to your latest state of #2. IOW other settings are not affected.

Your single-instance, single source of truth is still the CPF file.

For an initial list of the CRUD operations see the [Actions] page. Please note how the official list will grow as we will be able to run more tests. 

Let me know if this is helpful.

Luca Ravazzolo · Jul 13, 2021

@Lorenzo Scalese great way of exposing the IRIS internal API! I like it and I like the way the community brings innovation and supports the needs of users. Great effort, Lorenzo!

I also wanted to draw attention to a utlity that InterSystems has been supporting for several versions. We call this feature the CPF merge feature.

Q: What is the CPF merge feature?

A: It's the capability to configure an instance dynamically from the outside. It can be used with any configuration management tool like Chef, Puppet, Ansible, Salt or simpler bash or any cloud service provider provisioning tool like AWS CloudFormation, Terraform or orchestrator like Kubernetes. A user can define the ultima state of an IRIS instance. The operation is executed idempotently and all you need is an environment variable called ISC_CPF_MERGE_FILE=the_file_that_holds_my_desired_config

The CPF merge file could have been JSON, YAML, TOML or whatever but we decided to go with the familiar format we know, for now. The CPF merge file provides a way to Create, Delete and Update instance resources.

The doc.

Some Examples - Note how the CPF merge feature does not only helps us in single instance configuration but also automates more complex cluster configurations like Mirror pairs and shard architecture topologies.

I hope this is useful to the reader who is seeking more elegant and easy ways to automate InterSystems IRIS clusters.

Luca Ravazzolo · Jul 2, 2021

I see both of them

 {
      "RepositoryName": "intersystems/arbiter",
      "Tags": [
        "2019.1.1.615.1",
        "2020.1.0.215.0",
        "2020.1.1.408.0",
        "2020.2.0.211.0",
        "2020.3.0.221.0",
        "2020.4.0.547.0",
        "2021.1.0.215.0"
      ]
    },
    {
      "RepositoryName": "intersystems/arbiter-arm64",
      "Tags": [
        "2020.4.0.547.0"
      ]
    },

--

Command used

docker run --rm carinadigital/docker-ls \
  docker-ls \
    -u luxabc \
    -p abcdefghijklmnopqrstuvxyz0987654321 \
    --registry https://containers.intersystems.com \
      repositories \
        --level 2 \
        --json

--

Luca Ravazzolo · Apr 19, 2021

The adoption of containers, just like the adoption of a new UI paradigm from CHUI to GUI (think Visual Basic and similar 1990s client-server UI tools) to web-based design (formatting, graphical display, estate utilisation and URL links to other resources), forces us to think differently and adopt new "modus operandi".

When I first started with containers the first two thing I had to figure out were:

  • how do I keep this thing alive? It's not an OS (that's were ccontainermain came from) and
  • how do I securely connect to this thing? :) and so off I was installing SSHd

Those were Docker early days 2014/15

There was a Spanish company called Tutum, subsequently acquired by Docker that had a CentOS-based base-OS-layer with an SSH daemon already installed and so life was good back then.

Pointless to say that as one grows into understanding the technology, the new work-paradigm & the user-methodologies start to grow as the ecosystem of tools (consider docker exec, Docker Swarm, Kubernetes, Nomad, AWS ECS/Fargate, etc. but also fully automated CI/CD pipelines with Dev-QA-Sec-Ops), one starts to appreciate that things are done differently.

Since then, I've had a many a conversation with people trying to install SSH. It's not secret. We like what we know :-)

I've also had conversations with customers that want to move to a more modern provisioning pipeline, adopt containers and have a portable and more homogeneous  solution for their app. At times there are business constraints... I get that, however, when you start analysing the possible implementation solutions (you have CHUI-based solution that you want to port, how do you handle individual users? .profile at the OS level? You might as well have no containers. You cannot adopt that approach with Kubernetes anyway... do you handle it all in the containers? It means you must make /etc/passwd durable and all the $HOME directories, etc. etc. it becomes super complicated straight away... at that point your container-based provisioning becomes and hinderance vs an enabler.

Bottom line: "The times are a changing" as Bob Dylan used to sing and if you are interested in this new "cloud-native" way of working you're better off leaving things behind and adopt a new way of working that has many benefits... even if that means rewriting the CHUI interface.

Ultimately, just because it is doable it does not mean it is the right thing to do. Personally I don't feel I need ssh into containers when I develop nor do I see developers needing it. It is easy enough to jump into containers in any environments. OTOH I do understand the need to have better loggings and stats (think utilities/side-cars like cAdvisor). I think those type of sidecars should be like leaches and attach themselves as soon as they see an interesting container starting... but that is another story for automation, monitoring and for the next story :-)