Baanboard.com

Baanboard.com (http://www.baanboard.com/baanboard/index.php)
-   Performance & Tuning (http://www.baanboard.com/baanboard/forumdisplay.php?f=61)
-   -   bdbpost performance (http://www.baanboard.com/baanboard/showthread.php?t=58561)

makiju 5th June 2010 13:52

bdbpost performance
 
I've always wondered why bdbpost always gets slower during import. No matter what db or hardware or baan version...
For best performance every table should be imported with new bdbpost -process basically...
I think it does not matter what options to use either. What do you think? What is the root cause of this?

dave_23 7th June 2010 03:36

So you mean that the first table loads fast, and then subsequent tables load slower?

i've always assumed fragmented memory for the bdbpost process.

Dave

günther 7th June 2010 11:18

I've never checked that. But I could imagine that it has to do with transactions; in some programming situations I've seen such a behaviour whe there was one big transaction - after using commit.transaction() after 1000 records the performance was better. That would declare why it's faster to load each table by a single bdb.post.

Günther

dave_23 7th June 2010 18:09

bdbpost does regularly commit though. there's even a setting that lets you choose the commit frequency.

I'll take it a step further - within a single large table, the first tx's are fast, and the final tx's are usually very slow.

There's very little in the RDBMS that would account for this, especially since baan doesn't have FK constraints. Inserts into a table should maintain a fairly consistent speed regardless of the table size (especially if you have "rows before indexes" turned on, which you definitely should)

So my guess is basically that once bdbpost hits close to the 2GB limit of a 32bit process (or the max size of your data segment at the OS level) it starts swapping. And the reason it happens on a large number of smaller tables is because the cleanup between table loads isn't as good as it could be.

Dave

Dikkie Dik 2nd August 2010 15:06

My 2 cents (I know I am late):
- As mentioned before the inserts are pretty easy if you use the rows-before-indexes option
- Most of the time you will see that the disk and database buffer gets full over time. These buffers can handle a certain load, but most of the time the disks are the bottleneck at a certain time.

to check if I am correct you do a "ps -ef >> ps.log"
then you grep on the bdbpost process, the driver (if you are still on an ancient portingset) and the used database server process and see how many seconds they increased during these 60 seconds. If the total amount (of the 2 or 3 processes) is close to 60 seconds, you only burn CPU cycles, so no disk issue. If the number is less than 50 seconds, you already have a disk delay of more than 15%.

Hope this helps,
Dick

makiju 6th December 2010 17:21

Just amazed how well is Red Hat performing on PC hardware...also with bdbpost.
Example:
-dump size on disk = 5GB
-Oracle 11.2 via listener (no 32-bit client on server), default parameters
-baan4c4 with recent porting set

Import time with single bdbpost -process...result was 40 minutes!
RAID 1+0 (4 disks)

I didn't remember to monitor process memory consumption trend...need to do it next time.

Dikkie Dik 6th December 2010 17:48

Quote:

Originally Posted by makiju (Post 163815)
I've always wondered why bdbpost always gets slower during import. No matter what db or hardware or baan version...
For best performance every table should be imported with new bdbpost -process basically...
I think it does not matter what options to use either. What do you think? What is the root cause of this?

The root cause has been found and solved in portingset 8.7a. Hope this helps.

As a workaround you can increase the commit size. Setting this a factor 10 higher helps significant.

makiju 6th December 2010 18:27

What commit size do you mean?
Porting set note does not contain any information about bdbpost -thing...

Dikkie Dik 6th December 2010 23:06

Quote:

Originally Posted by makiju (Post 167648)
What commit size do you mean?

I mean the -r option. By default bdbpost commits per 100, now you can set it to 1000 or 10000.
Quote:

Originally Posted by makiju (Post 167648)
Porting set note does not contain any information about bdbpost -thing...

You are right I see it is in 8.7b

Hope this helps,
Dick


All times are GMT +2. The time now is 11:12.


vB.Sponsors
©2001-2017 - Baanboard.com - Baanforums.com