Converting iris stream to python bytes is very slow

Question

Question

Jonathan Perry · Dec 17, 2024

I'm trying to use embedded python code that receives an Iris %Stream.GlobalBinary and uses image manipulation library PIL.

Because PIL can't work with IRIS %Stream, I need to convert the image to python bytes.

This process seems to have very bad performance compared to writing to a file and then loading it from a file in python.

Is there a better way to send a stream into python? The conversion code I'm using is below.

Thanks.

def iris_stream_to_bytes(stream):
	stream.Rewind()
	s = ""
	while not stream.AtEnd:
		r = stream.Read(1024)
		s += r
	b = bytearray()
	b.extend(map(ord, s))
	return bPython
Python

Product version: IRIS 2024.2

$ZV: IRIS for Windows (x86-64) 2024.2 (Build 247U) Tue Jul 16 2024 09:57:03 EDT

Discussion (5)1

Log in or sign up to continue

score 0 · Answer 1 · 2024-12-18T05:00:07-05:00

I'm using it all the time for string and i haven't seen any performance issue so far.

May be try to increase the buffer :

def stream_to_string(stream,buffer=1000000)-> str:
    string = ""
    stream.Rewind()
    while not stream.AtEnd:
        string += stream.Read(buffer)
    return stringPython
Python

score 0 · Answer 2 · 2024-12-20T22:10:02-05:00

Jonathan Perry Dec 20, 2024 to Guillaume Rongier

Thanks, increasing the buffer to 1000000 reduced the processing of a 18MB image from 20 seconds to 2-3 seconds.

0 0

score 0 · Answer 3 · 2024-12-18T05:38:06-05:00

ClassMethod Stream2BytesArray(stream As %Stream.Object) As %SYS.Python
{
set builtin = $SYSTEM.Python.Builtins()

set b = builtin.bytearray()

  do stream.Rewind()
  while 'stream.AtEnd {
    set r = $system.Python.Bytes(stream.Read($$$MaxLocalLength))
    do b.extend(r)
  }
  quit b
}

score 0 · Answer 4 · 2024-12-20T22:11:15-05:00

Haven't had the chance to compare this method to converting it through python.

Do you think that doing the conversion on the cache side will be faster?

score 1 · Answer 5 · 2024-12-23T02:18:15-05:00

Simple test

Class dc.test [ Abstract ]
{

ClassMethod Stream2BytesArray1(stream As %Stream.Object) As %SYS.Python [ Language = python ]
{
  stream.Rewind()
  s = ""
  while not stream.AtEnd:
    r = stream.Read(1000000)
    s += r
  b = bytearray()
  b.extend(map(ord, s))
  return b
}

ClassMethod Stream2BytesArray2(stream As %Stream.Object) As %SYS.Python [ Language = python ]
{
  import iris

  maxlen = iris.system.SYS.MaxLocalLength()
  b = bytearray()

  stream.Rewind()
  while not stream.AtEnd:
    b.extend(iris._SYS.Python.Bytes(stream.Read(maxlen)))
  return b
}

ClassMethod Stream2BytesArray3(stream As %Stream.Object) As %SYS.Python
{
  set maxlen = $$$MaxLocalLength,
      b = $SYSTEM.Python.Builtins().bytearray()

  do stream.Rewind()
  while 'stream.AtEnd {do b.extend($system.Python.Bytes(stream.Read(maxlen)))}
  return b
}

/// d ##class(dc.test).Test()
ClassMethod Test()
{
  s builtin=$system.Python.Builtins(),
    maxstr=$tr($j("",$$$MaxStringLength)," ","0"),
    stream=##class(%GlobalBinaryStream).%New()
  f i=1:1:7 d stream.Write(maxstr)
  w "Stream.Size=",stream.Size,!
  
  f n=1:1:3 {
    s fn="Stream2BytesArray"_n,t=$zh,r=$CLASSMETHOD($this,fn,stream),t=($zh-t)_" s."
    w $$$FormatText("%1 len(r)=%2 time=%3",fn,builtin.len(r),t),!
  }
}

}

USER>d ##class(dc.test).Test()
Stream.Size=25488008
Stream2BytesArray1 len(r)=25488008 time=.752708 s.
Stream2BytesArray2 len(r)=25488008 time=.14895 s.
Stream2BytesArray3 len(r)=25488008 time=.080003 s.

The numbers speak for themselves.