very-big-strings digested

for those who have no time or other problems to read the http://uniface.communityzero.com/uniface?go=2146041 properly (the formatting was lost because of an accident in uniface.info), I trimmed it, snipped it, highlited it and put a current-status info in front Success, Uli

On the very same day (080903) I heard about the outstanding performance of my proposal,
but as well that the filedump is a nogo because the XML-string should be used internally,
I replied with a variation of the democode using a buffer-string instead of the filedump.

Running the example code for 100 outer loops on my machine took some awful 124 secs.
Introducing one string offload per outer loop reduced it to some 2 secs (> 98 % reduction).

Using XMLWRITER you get the generated XML in tranches and accumulate them in a string.
The longer the XMLstring the slower it goes.

But think first (successful programming is all about choosing the best-fit algorithm):
– We know a very long XML string inside the XMLWRITER will degrade performance
– We know that writing an XML is a sequential process
– We know that we can get the current XML substring anytime we want
Why not flush the generated XML from time to time and accumulate them in our string?
It’s the same logic as in my introduction of the buffer which caused a > 98 % reduction of processing time.

Success, Uli

BTW: due to my initiative, Mike Zille presented this as a booster on his
XMLWRITER workshop at the CBG-meeting in Stuttgart last year. 

********************************** now it’s history *****************************************
Posted: Aug 29, 2008  11:34 AM by -GHAN-
i’m using the uxmlwriter to create XML-streams to transport occurences from my Entity from one component to another.
Works well so far, but as the number of records grow, uniface goes slow-motion!
Retrieving a 1000 Records of an Entity containing 180 fields takes (on this machine) 1.5 minutes!
The Code does nearly the same as the xmlwriter does, so you can test it yourselves by doing:
-[CODE follows]—–
putmess “start : %%$clock”
$1=1
while($1 < 1000)
$2 = 1
while ($2 < 180)
vString = “%%vString%%%XMLTEEEEEEEEEEXT”; ### simulating a XML-Tag
$2 = $2 + 1
endwhile
$1 = $1 + 1
endwhile
putmess “End : %%$clock”
Posted: Sep 2, 2008  5:26 PM by ulrich-merkel
The “trick” is not to let the string grow too much:
just use a filedump/append now and then
to offload a big chunk of the string  and start with a “clean” string again.
At the end, just file_load the longstring.
Posted: Sep 3, 2008  8:10 AM by -GHAN-
We gave the filedump-thing a try lately and tested the same functionality with and selfmade-3gl which handled the data. Outstanding performance
Posted: Sep 3, 2008  2:08 PM by ulrich-merkel
Here is a modification of your example with a nearly linear performance
Triggerfrom Form: A_LONGTEXT
****** Y4::
****** entry LONG1
1 variables
2 string vString
3 endvariables
4 X_TEXT = “%%X_TEXT%%%Long1 start : %%$clock”
5 $1=1
6 while($1 < 1000)
7-1 $2 = 1
8-1 while ($2 < 180)
9-2 vString = “%%vString%%%XMLTEEEEEEEEEEXT”; ### simulating a XML-Tag
10-2 $2 = $2 + 1
11-2 endwhile
12-1 $1 = $1 + 1
13-1 message/hint “%%$clock%%% %%$1%%%”
14-1 endwhile
15 X_TEXT = “%%X_TEXT%%% End : %%$clock%%^”
16 end ; long1
****** entry LONG2
1 variables
2 string vString, vBuffer
3 endvariables
4 X_TEXT = “%%X_TEXT%%%Long2 start : %%$clock”
5 $1=1
6 while($1 < 1000)
7-1 $2 = 1
8-1 vString = “”
9-1 while ($2 < 180)
10-2 vString = “%%vString%%%XMLTEEEEEEEEEEXT”; ### simulating a XML-Tag
11-2 $2 = $2 + 1
12-2 endwhile
13-1 vBuffer = “%%vBuffer%%%%%vString%%%”
14-1 $1 = $1 + 1
15-1 message/hint “%%$clock%%% %%$1%%%”
16-1 endwhile
17 X_TEXT = “%%X_TEXT%%% End : %%$clock%%^”
18 end ; long2

Leave a Reply