This website is not affiliated with, sponsored by, or approved by SAP AG.

>=NW640, 5 Steps to run BDLS in parallel and record time

Moderators: Snowy, thx4allthefish

>=NW640, 5 Steps to run BDLS in parallel and record time

Postby thx4allthefish » Fri Feb 13, 2009 9:54 am

Introduction

as some of you may have experienced, BDLS has changed in releases from NW640 upwards. the 'new' BDLS cannot be run in parallel any longer, every attempt of doing so leads to the jobs aborting or getting 'stuck'. apart from this news the absurdly long runtimes of BDLS are grating on the nerves of every basis person who has to do a lot of system copies.

i met my challenge last week and i think i have found a way out of this dead end.

Step 1. Since the new BDLS refuses parallel runs, don't use it

the 'old' BDLS is still around and - SAP reassured me - can be used as of old without any damage being done. so i did it. in order to get a report generated which you can use for your parallelization, rund RBDLS2LS using tx. SE38 in background for a couple of minutes, then abort it. by then it should have generated a report with the name RBDLSXXX (where XXX = the number of the client). that's the one you want.

Step 2. Help the database work, create temporary indexes

after the report RBDLSXXX has been generated, have a look at the coding and search - for example - for table GLPCA. you will recognize that this report will scan GLPCA twice, first for field LOGSYS, second time for field AWSYS. other tables contain up to 3 fields that match the scan criteria and each scan run will only operate on one of the fields. i'm not perfectly sure why, but i think all of those fields may (in certain setups) contain different information and not necessarily the same information in all of the fields, so that might be the reason for several scans on the same table.

identify all of the tables that need scanning and conversion. the most easy way to do so is to run the new BDLS in test mode and have a look at the tables BDLSHDR and BDLSPOS afterwards. you need not do this every time, but every now and then - just to be sure you have even the newly created (custom or otherwise) tables.

create an index for every field that identifies for the scan. in our example, create:

Index Z01 on GLPCA for field LOGSYS / if operating on a MaxDB, avoid fields indicating the client, in this case RSCLT
Index Z02 on GLPCA for field AWSYS / ...

continue to do so for the largest tables that will be converted, like CE1ZZZZ, VBAP, GLPCA, VBUK, COEP ... and several others. if using an IS or add-on there might be several more ...

some databases do not take kindly to the attempt of indexes created in parallel, like in our example: creating Z01 and Z02 at the same time might not work - switch to sequential processing in such a case.

Step 3. Create Variants

create variants for your report RBDLSXXX and try to make packages that make sense regarding the size of the tables per package. select the tables to be converted like: A* to D* (caution: if on MaxDB this will proceed all tables the names of which begin with A, B and C - it will not proceed on tables starting with a D, so in this case, make your variants either overlapping, like A* to D* and the next from D* to G* or simply add single values for the selection). for tables the names of which start with '/' and such ... there is note telling to create a variant with the selection: exclude A* to Z* ... again: for MaxDB this does not work. MaxDB interprets such a statement as: proceed all tables the names of which begin with a Z and then it stops. i ended up, naming the tables in such namespaces singularly.

try to exclude as many of the biggest tables as possible by naming them in your variants. for example: if table CE1ZZZZ does not have any values in the fields LOGSYS, there is no need to scan/convert the table. excluding them will shorten the runtime. you can check on such tables using tx. TAANA, grouping by the fields to be scanned.

with the 'old' BDLS there is no obligation to do a testrun before doing a conversion. so you might want to de-flag the testrun option.

de-flag the 'check for every entry' also.

SAP sometimes advises to enlarge the commit-counter up to 10.000.000 - i was none to happy with that and after several experiments i let it be on the default value of 100.000

Step 4. Adjust Operation Modes and/or Process Setup

depending on the number of variants you will want for as many batch-processes as there are variants. your hardware setup allowing for such, try to create operation modes (RZ12) that will switch dialogue-processes to batch during runtime. make sure, you leave some dialogue-processes for logins etc.

Step 5. Run your jobs and Clean Up

run as many jobs in parallel as you are able to, split if necessary. do not forget to drop the indexes you created in step 2. as they might badly influence the overall system performance for "normal" tasks and are no longer needed. frees the space used, also.

Statistics

all the figures given now were taken on a NW640/22 ERP 5.0 / MaxDB 7.6 / Linux 2.6 / size of DB: 1 TB / mode: conversion run / DB-cache empty (directly after startup).

GLPCA without any indexes: 32 hours
GLPCA with one index on both fields LOGSYS and AWSYS: 15 hours
GLPCA with two indexes, one for each of the fields LOGSYS and AWSYS: 7 seconds
complete run of new BDLS without variants, exclusion of tables, and indexes: 108 hours
complete run of old BDLS with improvements of step 1 - 5: 3 hours

hope this helps :wink:

ETA 2010-09-17: If you ever have to reset a crashed BDLS, follow note 962674.
curiousorange wrote:I give up. Humanity isn't worth saving. Why is there never a Vogon Constructor Fleet around when you really need one?
thx4allthefish
 
Posts: 5694
Joined: Sat Oct 26, 2002 6:18 pm
Location: barolo barrel

Re: >=NW640, 5 Steps to run BDLS in parallel and record time

Postby Snowy » Fri Feb 13, 2009 2:42 pm

Wow!

great info!

I prefer not to run BDLS and just adapting SM59 hostnames for these connections

if people can handle understanding that PBW400 in the QAS system really means QBW400, than BDLS is not needed.

But still, your information is greatly appreciated because I understand that some people MUST see the right description in SM59/RSA1 to function properly.
SapFans Moderator

Search: http://www.sapfans.com/forums/search.php
Notes: http://service.sap.com/notes
Help: http://help.sap.com
Rules: http://www.sapfans.com/forums/viewtopic.php?t=344127
Snowy
 
Posts: 28768
Joined: Mon Oct 21, 2002 2:33 pm
Location: 3.1415926535

Re: >=NW640, 5 Steps to run BDLS in parallel and record time

Postby thx4allthefish » Mon Feb 16, 2009 3:33 am

thanks!

if it were only for the one system-copy and a connect to BI i wouldn't have gone to all that trouble. in my case we copied a whole system landscape to be shipped to China and re-built there ... so there's no avoiding BDLS ...
curiousorange wrote:I give up. Humanity isn't worth saving. Why is there never a Vogon Constructor Fleet around when you really need one?
thx4allthefish
 
Posts: 5694
Joined: Sat Oct 26, 2002 6:18 pm
Location: barolo barrel

Re: >=NW640, 5 Steps to run BDLS in parallel and record time

Postby Snowy » Thu Dec 10, 2009 7:36 am

I would like to share indexes that helped me.

I use ECC 6.0, we mainly use it for Retail stuff and we also use HR with Payroll
my DB is about 1TB and BDLS runs within 1 hour now.

create tablespace PSAPBDLSI

CREATE INDEX SAPR3."BKPF~ZB" ON SAPR3.BKPF (MANDT, AWSYS) NOLOGGING TABLESPACE PSAPBDLSI PARALLEL;
CREATE INDEX SAPR3."CE11000~ZB" ON SAPR3.CE11000 (MANDT, COPA_AWSYS) NOLOGGING TABLESPACE PSAPBDLSI PARALLEL;
CREATE INDEX SAPR3."COBK~ZB" ON SAPR3.COBK (MANDT, LOGSYSTEM, AWSYS) NOLOGGING TABLESPACE PSAPBDLSI PARALLEL;
CREATE INDEX SAPR3."COEP~ZB" ON SAPR3.COEP (MANDT, LOGSYSO, LOGSYSP) NOLOGGING TABLESPACE PSAPBDLSI PARALLEL;
CREATE INDEX SAPR3."COES~ZB" ON SAPR3.COES (MANDT, AWSYS) NOLOGGING TABLESPACE PSAPBDLSI PARALLEL;
CREATE INDEX SAPR3."COFIS~ZB" ON SAPR3.COFIS (RCLNT, LOGSYS, RLOGSYS, SLOGSYS) NOLOGGING TABLESPACE PSAPBDLSI PARALLEL;
CREATE INDEX SAPR3."EKPO~ZB" ON SAPR3.EKPO (MANDT, EXT_RFX_SYSTEM) NOLOGGING TABLESPACE PSAPBDLSI PARALLEL;
CREATE INDEX SAPR3."GLPCA~ZB" ON SAPR3.GLPCA (RCLNT, AWSYS, LOGSYS) NOLOGGING TABLESPACE PSAPBDLSI PARALLEL;
CREATE INDEX SAPR3."GLPCP~ZB" ON SAPR3.GLPCP (RCLNT, AWSYS, LOGSYS) NOLOGGING TABLESPACE PSAPBDLSI PARALLEL;
CREATE INDEX SAPR3."GLPCT~ZB" ON SAPR3.GLPCT (RCLNT, LOGSYS) NOLOGGING TABLESPACE PSAPBDLSI PARALLEL;
CREATE INDEX SAPR3."LIKP~ZB" ON SAPR3.LIKP (MANDT, VERURSYS) NOLOGGING TABLESPACE PSAPBDLSI PARALLEL;
CREATE INDEX SAPR3."LIPS~ZB" ON SAPR3.LIPS (MANDT, VGSYS) NOLOGGING TABLESPACE PSAPBDLSI PARALLEL;
CREATE INDEX SAPR3."MKPF~ZB" ON SAPR3.MKPF (MANDT, SPE_LOGSYS, AWSYS) NOLOGGING TABLESPACE PSAPBDLSI PARALLEL;
CREATE INDEX SAPR3."PA2002~ZB" ON SAPR3.PA2002 (MANDT, LOGSYS) NOLOGGING TABLESPACE PSAPBDLSI PARALLEL;
CREATE INDEX SAPR3."SRRELROLES~ZB" ON SAPR3.SRRELROLES (CLIENT, LOGSYS) NOLOGGING TABLESPACE PSAPBDLSI PARALLEL;
CREATE INDEX SAPR3."STXH~ZB" ON SAPR3.STXH (MANDT, LOGSYS) NOLOGGING TABLESPACE PSAPBDLSI PARALLEL;
CREATE INDEX SAPR3."VBFA~ZB" ON SAPR3.VBFA (MANDT, LOGSYS) NOLOGGING TABLESPACE PSAPBDLSI PARALLEL;

ALTER INDEX SAPR3."BKPF~ZB" NOPARALLEL;
ALTER INDEX SAPR3."CE11000~ZB" NOPARALLEL;
ALTER INDEX SAPR3."COBK~ZB" NOPARALLEL;
ALTER INDEX SAPR3."COEP~ZB" NOPARALLEL;
ALTER INDEX SAPR3."COES~ZB" NOPARALLEL;
ALTER INDEX SAPR3."COFIS~ZB" NOPARALLEL;
ALTER INDEX SAPR3."EKPO~ZB" NOPARALLEL;
ALTER INDEX SAPR3."GLPCA~ZB" NOPARALLEL;
ALTER INDEX SAPR3."GLPCP~ZB" NOPARALLEL;
ALTER INDEX SAPR3."GLPCT~ZB" NOPARALLEL;
ALTER INDEX SAPR3."LIKP~ZB" NOPARALLEL;
ALTER INDEX SAPR3."LIPS~ZB" NOPARALLEL;
ALTER INDEX SAPR3."MKPF~ZB" NOPARALLEL;
ALTER INDEX SAPR3."PA2002~ZB" NOPARALLEL;
ALTER INDEX SAPR3."SRRELROLES~ZB" NOPARALLEL;
ALTER INDEX SAPR3."STXH~ZB" NOPARALLEL;
ALTER INDEX SAPR3."VBFA~ZB" NOPARALLEL;

ANALYZE INDEX SAPR3."BKPF~ZB" ESTIMATE STATISTICS SAMPLE 2 PERCENT;
ANALYZE INDEX SAPR3."CE11000~ZB" ESTIMATE STATISTICS SAMPLE 2 PERCENT;
ANALYZE INDEX SAPR3."COBK~ZB" ESTIMATE STATISTICS SAMPLE 2 PERCENT;
ANALYZE INDEX SAPR3."COEP~ZB" ESTIMATE STATISTICS SAMPLE 2 PERCENT;
ANALYZE INDEX SAPR3."COES~ZB" ESTIMATE STATISTICS SAMPLE 2 PERCENT;
ANALYZE INDEX SAPR3."COFIS~ZB" ESTIMATE STATISTICS SAMPLE 2 PERCENT;
ANALYZE INDEX SAPR3."EKPO~ZB" ESTIMATE STATISTICS SAMPLE 2 PERCENT;
ANALYZE INDEX SAPR3."GLPCA~ZB" ESTIMATE STATISTICS SAMPLE 2 PERCENT;
ANALYZE INDEX SAPR3."GLPCP~ZB" ESTIMATE STATISTICS SAMPLE 2 PERCENT;
ANALYZE INDEX SAPR3."GLPCT~ZB" ESTIMATE STATISTICS SAMPLE 2 PERCENT;
ANALYZE INDEX SAPR3."LIKP~ZB" ESTIMATE STATISTICS SAMPLE 2 PERCENT;
ANALYZE INDEX SAPR3."LIPS~ZB" ESTIMATE STATISTICS SAMPLE 2 PERCENT;
ANALYZE INDEX SAPR3."MKPF~ZB" ESTIMATE STATISTICS SAMPLE 2 PERCENT;
ANALYZE INDEX SAPR3."PA2002~ZB" ESTIMATE STATISTICS SAMPLE 2 PERCENT;
ANALYZE INDEX SAPR3."SRRELROLES~ZB" ESTIMATE STATISTICS SAMPLE 2 PERCENT;
ANALYZE INDEX SAPR3."STXH~ZB" ESTIMATE STATISTICS SAMPLE 2 PERCENT;
ANALYZE INDEX SAPR3."VBFA~ZB" ESTIMATE STATISTICS SAMPLE 2 PERCENT;


run BDLS here.
when BDLS has run, drop temporary indexes:

DROP INDEX SAPR3."BKPF~ZB";
DROP INDEX SAPR3."CE11000~ZB";
DROP INDEX SAPR3."COBK~ZB";
DROP INDEX SAPR3."COEP~ZB";
DROP INDEX SAPR3."COES~ZB";
DROP INDEX SAPR3."COFIS~ZB";
DROP INDEX SAPR3."EKPO~ZB";
DROP INDEX SAPR3."GLPCA~ZB";
DROP INDEX SAPR3."GLPCP~ZB";
DROP INDEX SAPR3."GLPCT~ZB";
DROP INDEX SAPR3."LIKP~ZB";
DROP INDEX SAPR3."LIPS~ZB";
DROP INDEX SAPR3."MKPF~ZB";
DROP INDEX SAPR3."PA2002~ZB";
DROP INDEX SAPR3."SRRELROLES~ZB";
DROP INDEX SAPR3."STXH~ZB";
DROP INDEX SAPR3."VBFA~ZB";

you can now drop PSAPBDLSI
SapFans Moderator

Search: http://www.sapfans.com/forums/search.php
Notes: http://service.sap.com/notes
Help: http://help.sap.com
Rules: http://www.sapfans.com/forums/viewtopic.php?t=344127
Snowy
 
Posts: 28768
Joined: Mon Oct 21, 2002 2:33 pm
Location: 3.1415926535

Re: >=NW640, 5 Steps to run BDLS in parallel and record time

Postby RichB » Thu Dec 10, 2009 11:49 am

Thanks Snowy,

We've avoided running BDLS for years by adjusting the RFC connection details. This has worked just fine in our Staging/QA landscape... especially when we largely an R/3 shop. However, there may be a need for this soon as we may have multiple test systems connected to the same env. Logical system management is critical to connectivity with our SCM/BI/XI systems so we may need to do this. Our large systems are in the 8-15 TB range so performance is key.

On the Oracle front...

If you're using Oracle 10g, There's no need to estimate statistics after an Index build(create). Oracle creates the COMPLETE index stats during the create. The process of gathering index stats after a create is a holdover from Oracle 9i. The estimation is unnessary and actually provides 'less accurate' stats than the ones Oracle just created a step earlier. Also the estimation can go into the decimal range... so for 'uber' large indexes, you can use a '.1' or '.01' sample size.

Of course, you do have to be aware of tables that have locked stats. Those need a 'FORCE' index stats creation. Otherwise, the index will be created with no stats.

HTH,
Rich
RichB
 
Posts: 2
Joined: Thu Dec 10, 2009 11:35 am
Location: Concord, CA

Re: >=NW640, 5 Steps to run BDLS in parallel and record time

Postby thx4allthefish » Wed Dec 16, 2009 12:41 am

aaaawwww, Snowy, you lucky guy - obviously Oracle is capable of using one index with two of the target fields performantly *sigh*, no such luck with MaxDB here.

but now for something different: i noticed you created an index for SRRELROLES. if that table has grown so much that you have to include it in your BDLS-index-setup, you should probably take steps to reduce the size of this table. look at note 505608 for background information on the 'why' of the growth of that table and -if safe- run RSRLDREL to decrease table size.
curiousorange wrote:I give up. Humanity isn't worth saving. Why is there never a Vogon Constructor Fleet around when you really need one?
thx4allthefish
 
Posts: 5694
Joined: Sat Oct 26, 2002 6:18 pm
Location: barolo barrel

Re: >=NW640, 5 Steps to run BDLS in parallel and record time

Postby Snowy » Wed Dec 16, 2009 6:07 am

i took a list of tables from an SDN blog and added a few tables myself. I will check your note.

thanks!
SapFans Moderator

Search: http://www.sapfans.com/forums/search.php
Notes: http://service.sap.com/notes
Help: http://help.sap.com
Rules: http://www.sapfans.com/forums/viewtopic.php?t=344127
Snowy
 
Posts: 28768
Joined: Mon Oct 21, 2002 2:33 pm
Location: 3.1415926535

Re: >=NW640, 5 Steps to run BDLS in parallel and record time

Postby Snowy » Fri Sep 03, 2010 8:21 am

SapFans Moderator

Search: http://www.sapfans.com/forums/search.php
Notes: http://service.sap.com/notes
Help: http://help.sap.com
Rules: http://www.sapfans.com/forums/viewtopic.php?t=344127
Snowy
 
Posts: 28768
Joined: Mon Oct 21, 2002 2:33 pm
Location: 3.1415926535

Re: >=NW640, 5 Steps to run BDLS in parallel and record time

Postby thx4allthefish » Wed Dec 07, 2011 1:12 am

adding to Snowy's advice:

  1. Please note that the blog is very old by now.
  2. Please accept that the advise given might not work in every combination of OS/DB/SAP (ERP/NW/BI ...) release: you still have to make the effort of investigating in your environment.
  3. Please note that a new transaction - substituting BDLS - is available for complex system landscapes
  4. Always make sure you read the latest OSS-notes on the subject.


Last bumped by thx4allthefish on Wed Dec 07, 2011 1:12 am.
curiousorange wrote:I give up. Humanity isn't worth saving. Why is there never a Vogon Constructor Fleet around when you really need one?
thx4allthefish
 
Posts: 5694
Joined: Sat Oct 26, 2002 6:18 pm
Location: barolo barrel


Return to Technical

Who is online

Users browsing this forum: No registered users and 1 guest





loading...


This website is not affiliated with, sponsored by, or approved by SAP AG.